Think twice before patching something that you can nail down. Patching might improve short-term velocity but nailing it will definitely improve the long-term one. At the end of the day, patching will ensure data consistency and quality over time.
Have you ever heard about the "quick fix"?
Dave: « Knock, Knock. »
You, in the data team: « Who's there? »
Dave: « It's Dave »
You: « Dave who? »
Dave: « (Dave-loper) Dave from the Mobile Team. We have a small issue in the production app. "Sign up" events don't contain user info anymore. Can you patch it on Data Engineering side? You just have to join with the production data. Otherwise, all acquisition dashboards will be broken. It will be a mess 🥺»
Dave knows you. They know you don't like broken dashboard. And as you might be new around, you don't want to make a mess.
So here you go, you fix it. JOIN blah blah user blah COALESCE bla blah blah email.
🥳 TADA. It's (fixed) patched. You're the nice folk on whom we can rely!
However, 2 things happened:
Receive similar content twice a month, along with the latest data news, tips and attractive European job offers.
You know they know you. And you know the "broken dashboard" thing was to nicely try to coax you. But you've got experience.
You know that the potential fix is not that hard but maintaining it will be costly. So you ask for it to be fixed. You stay strong.
🥳 TADA. You're not the nice folk anymore but the issue was (magically) rolled back, fixed and deployed again!
Now is the time for you to bring back some positivity. You can suggest to run a one time single query to fix the events generated while the broken version was up. It should be a one-off script/query that you won't maintain.
Not only you'll help them, but you'll help yourself. It will ensure data consistency and quality over time. Without this query/script (in my first job we called them "Moulinette"), you'll have to maintain a transformation layer "coalescing" some values for some time.
For further information on data quality, read Ensuring Data Quality Within Your Organisation.
💡Tip: now you can add a test at ingestion or at transformation to make sure that you detect the same issue faster next time! It's the first step towards actual data contracts etc...
You might not be the nice folk anymore but you're the one silently building a stronger and more robust system, pipeline and team.
Life is not always that easy and the solution might not be black or white. Often there are trade-offs to make but saying "No" often forces to find actual solutions.
Over time, you'll be the owner of your entire stack, its pros and its cons. The more patches, the more difficult it gets to maintain and to add new capabilities.
As a Data Engineer, an additional transformation layer containing a patch is harder to maintain than a software fix + a one-off script.
As a Data Analyst, an additional dashboard is harder to maintain than a duplicable notebook.
It's your job to find the right balance between short-term velocity and long-term one. Patching improves the first, nailing the later. We've talked about a team velocity before. For further reading, I encourage to take a look at Speed and Velocity in the Data Team.