Quick fix is rarely quick

Have you ever heard about the "quick fix"?

Dave: « Knock, Knock. »

You, in the data team:  « Who's there? »

Dave: « It's Dave »

You: « Dave who? »

Dave: « (Dave-loper) Dave from the Mobile Team. We have a small issue in the production app. "Sign up" events don't contain user info anymore. Can you patch it on Data Engineering side? You just have to join with the production data. Otherwise, all acquisition dashboards will be broken. It will be a mess 🥺»

The quick fix — You patch It

Dave knows you. They know you don't like broken dashboard. And as you might be new around, you don't want to make a mess.

So here you go, you fix it. JOIN blah blah user blah COALESCE bla blah blah email.

🥳 TADA. It's (fixed) patched. You're the nice folk on whom we can rely!

However, 2 things happened:

• You're now the go-to person when there is a bug. As soon as something will break they will knock on your shoulder. Welcome to the world of low impact, time consuming, bug fixing tasks.
• A patch will always peel off. You don't know when but it will. And very often faster than you think. It will most probably be way harder to fix later.

The hard reasonable choice - You nail It

You know they know you. And you know the "broken dashboard" thing was to nicely try to coax you. But you've got experience.

You know that the potential fix is not that hard but maintaining it will be costly. So you ask for it to be fixed. You stay strong.

🥳 TADA. You're not the nice folk anymore but the issue was (magically) rolled back, fixed and deployed again!

Now is the time for you to bring back some positivity. You can suggest to run a one time single query to fix the events generated while the broken version was up. It should be a one-off script/query that you won't maintain.

Not only you'll help them, but you'll help yourself. It will ensure data consistency and quality over time. Without this query/script (in my first job we called them "Moulinette"), you'll have to maintain a transformation layer "coalescing" some values for some time.
For further information on data quality, read Ensuring Data Quality Within Your Organisation.

💡Tip: now you can add a test at ingestion or at transformation to make sure that you detect the same issue faster next time! It's the first step towards actual data contracts etc...

You might not be the nice folk anymore but you're the one silently building a stronger and more robust system, pipeline and team.

---

Life is not always that easy and the solution might not be black or white. Often there are trade-offs to make but saying "No" often forces to find actual solutions.

Over time, you'll be the owner of your entire stack, its pros and its cons. The more patches, the more difficult it gets to maintain and to add new capabilities.

As a Data Engineer, an additional transformation layer containing a patch is harder to maintain than a software fix + a one-off script.

As a Data Analyst, an additional dashboard is harder to maintain than a duplicable notebook.

It's your job to find the right balance between short-term velocity and long-term one. Patching improves the first, nailing the later. We've talked about a team velocity before. For further reading, I encourage to take a look at Speed and Velocity in the Data Team.