If you’ve worked on a data team, you may recognise this. A stakeholder pings you for a quick analysis. A dashboard suddenly breaks. A pipeline fails and it’s all hands on deck. By Friday, you’ve solved a dozen problems, buta few weeks later, the same issues resurface. That analysis you pulled together? Buried in a notebook no one will open again. The clever SQL trick you figured out after three cups of coffee? Forgotten by next month. That nasty bug your teammate patched? Only he remembers how, and he left two years ago. This is the hamster wheel most data teams live on. You move faster, but you don’t really get smarter. Here’s the good news: you don’t need a big budget or complex tools to change this. With a few lightweight practices, you can make your team’s knowledge stick, cut rework, and onboard new people faster.
This image was generated by AI. You can see that because the guy is totally missing the fire.
1. Create a Lasting Place for Knowledge
If you want lasting knowledge, you need a lasting home. Start with an internal wiki. It doesn’t really matter if you use Confluence, Microsoft DevOps Wiki, or open-source wiki.js. What matters is that you make sure that:
-
It’s painless to create entries
-
Content is searchable
-
You can export your entries, so that you can actually own your data and be able to migrate at a later point, if needed
Don’t overcomplicate it. Pick a tool, make it easy, and use it.
2. Store Analyses in One Place
Most “one-off” analyses aren’t really one-off . They come back around in some shape or form. The problem is, by the time the same question pops up again, the original work nowhere to be found. Keeping a central place to store analyses, however simple, means the next person doesn’t have to start from scratch. It’s less about perfect polish and more about leaving breadcrumbs that others (including future you) can actually follow.
Create an analyses repository. The key is to make it easy to use. In this case, you shouldn’t care about branch protection rules or coding standards, just offer a go-to no fuss place where people can store their analyses. For example, we created a repo with only three ground rules:
-
An analysis should be stored in a folder named <date>_<descriptive_analysis_name>, this eliminates conflicts, so everyone can just work off and push to the main branch.
-
The folder should have a data/ subfolder, where all input and output data is located, these folders is ignored by git.
-
Each analysis should have a README that explains the context of the analysis, why we’re doing it, what we’re doing, and instructions on how to reproduce the analysis.
Bonus tip: Create a tools/ folder, with helpful tools people can use to easily and securely connect with sources / source data, transform or export data, etc. It may be wise to introduce more ground rules for that specific section, though.
3. Write Post-Mortems That Don’t Suck
If you’ve ever sat through a “post-mortem” meeting that turned into finger-pointing, you know why teams avoid them. But that misses the point. A good post-mortem isn’t necessarily a meeting. It’s definitely not a blame game. I’ve adopted post-mortems as just a short note that captures what happened and what I learned, keeping it short and structured:
-
Title: Emoji + date + description (✅ resolved, ❌ won’t fix, 🟠 pending).
-
**Problem:**What was the actual problem you encountered? Include error trace and link if available.
-
**Reason:**What caused the problem? The answer to this is often the result of a (short) investigation.
-
**Fix:**What were the actions taken to fix the problem?
-
**Preventive measures:**Where additional steps taken to prevent the problem to happening again?
-
**Impact (optional):**Were there notable dependencies / consequences to the problem others need to be aware of?
Below is a post-mortem I actually wrote (albeit censored), so you can see how short it can be:
✅ 2023–10–21 — load wait_for_ingest failed
-
Problem: The wait_for_ingest Airflow task failed, halting downstream processes.
-
Reason: The referred ingest took longer than expected.
-
Fix: Retried the task.
-
Preventive measures: Delayed DAG by 15 minutes to 06:15 LT. Increase task retries from 1 to 2.
No pages of jargon, and just enough context so the next time something similar happens, you know what to do, or what was already done.
Over time, if these notes stack up, you can start to see patterns and act on them. You stop treating incidents as isolated fires and start spotting the weak points in your system.
4. Keep an Open “How To” Section
Internal wikis are great, but over time they tend to evolve into tightly organised collections of documentation. The more your team writes, and the bigger it gets, the more structure you need to keep things manageable.
That’s natural. But it also raises the barrier to adding new docs.
One way to lower that barrier: create the least-governed space possible. A simple section, with only one rule: every title starts with “How to …”.
In my experience, this has been great. It’s a place for documentation that doesn’t quite fit anywhere else (yet), and it encourages everyone to contribute. Over the years, my “How To” section has filled up with things like:
How to create a Lambda function
How to ingest a new table into the data lake
How to add dbt snapshot history
How to map Parquet columns to Redshift
How to get an RDS backup in the data warehouse
How to create a new Redshift read_only user
How to connect to the Data Warehouse from Sagemaker Studio
How to encrypt data using Redshift
How to optimise dbt models
How to fix the CloudFormation UPDATE_ROLLBACK_COMPLETE stack state
How to work with AWS AppFlow
How to view a dashboard
How to minimise Lambda function startup latency
How to manage MLOps pipelines
How to load test an API endpoint
How to upgrade dbt and Airflow
How to solve an Negsignal.SIGKILL error within Airflow
As you can imagine, this simple documentation creates a wealth of otherwise undocumented processes and knowledge over time. You might notice some entries are more relevant and viewed than others, but that is okay. The goal of this section is to easily capture and share knowledge, not to optimise it.
5. Record Decisions
Have you ever wondered: “Why are we using x, while y exists?” And when you start asking around no one seems yo know and you’re left with questions. What are the benefits of the x approach? Was y even considered? This happened to me more than I like to admit, and it’s exactly why data teams need decision records.
I first came across Architecural Decision Records (ADRs) a few years ago, when a colleague pointed me to a great repository that explains what they are (“An architecture decision record (ADR) is a document that captures an important architectural decision made along with its context and consequences.”) and includes some formats to get started with them. I’ve since learned that these documents do not need to be limited to architectural decisions only, but can be integrated into the more general decision making process as well. Hence, Decision Records. In it’s simplest form, Decision Records are organised documents, that track decisions that are made along with their context.
My tip if you’re new to Decision Records, is to Keep It Stupidly Simple. The most popular ADR format contains just four sections: The status, the decision, the context and the consequences. Similar to Post-Mortems, this can be a great place to start.
Decision Records form a very powerful tool you can expand on, that is why I dedicated a full post to this that goes into more detail and the implementation if you want to know more.
6. Make It Stick
The hardest part isn’t setting up the wikis, repos, or templates. It’s making people actually use them. It’s not a tooling problem. It’s a habits problem.
Here’s how you embed these practices into your daily work:
-
When someone asks you how to do something, don’t type the answer in chat. Write a quick how-to, drop the link, and ask if it’s clear.
-
After an incident, jot down a post-mortem and bring it to your next team sync.
-
When you adopt new tech, create a Decision Record and share it for review.
It comes down to this: lead by example, stay consistent, and show how these practices save time. People will follow once they see the payoff.
Data teams don’t fail because they lack skills or tools. They fail because knowledge slips through the cracks. Analyses vanish, fixes get forgotten, and decisions lose their context. To fix this, start small. Write down the next incident. Save the next analysis. Capture the next decision.
Each small piece of documentation may feel trivial in the moment. But together, they compound into a system of lasting knowledge. A system that helps your team stop firefighting and start learning.
Hi, I’m Bastiaan 👋🏼 Founder of datalyft, a small Dutch data agency helping companies transform their raw data into real value. I write about the Modern Data Workflow, where I explore tools & processes to supercharge your data capabilities. Follow me for more!