Photo by Desola Lanre-Ologun on Unsplash
Have you ever wondered: “Why are we using x, while y exists?” And when you start asking around it becomes clear that it is because someone introduced it ages ago. Well, that person does not work at the company anymore. So now you’re left with questions. What are the benefits of the x approach? Was y even considered? This happened to me more than I like to admit, and it’s exactly why data teams need decision records.
The missing piece
I might be w̶e̶i̶r̶d̶ different, but I’m passionate about documentation. To me, it’s the ultimate test. If I can clearly explain my solution to someone else, it’s probably a fitting solution. Alternatively, if I can’t explain it, it needs more work.
Documentation can take on many forms. Inline comments for very specific and detailed information, READMEs to document relevant repository information, and often some kind of internal wiki to store information about overarching topics such as coding standards, setups, architectures, et cetera.
People tend to use these mediums to either document what they’ve done or how to do something. “Setup your dev environment with Docker by running docker compose up”, “We use leading commas in our SQL queries”. All helpful, but lacking a crucial component. The why.
This is often left undocumented because when documentation is written, the whyalready**exists within the context of the writer. “We use docker compose to make sure everyone’s environment is similar, and we can control the container dependencies”, or “We use leading commas in our SQL queries, to makes commenting out lines easier, and PRs cleaner” This phenomenon is called the “Curse of Knowledge” — and it occurs when someone unconsciously assumes people know as much about a topic as they do.
Documentation lacking context causes a problem, because context changes. If we look at the introductory situation: “Why are we using x for this, while y exists?” — Well, solution y may not have existed when the decision was made, or may not have been considered in the decision making process. To prevent problems like these, we can use Decision Records.
Decision Records
I first came across Architecural Decision Records (ADRs) a few years ago, when a colleague pointed me to a great repository that explains what they are (“An architecture decision record (ADR) is a document that captures an important architectural decision made along with its context and consequences.”) and includes some formats to get started with them. I’ve since learned that these documents do not need to be limited to architectural decisions only, but can be integrated into the more general decision making process as well. Hence, Decision Records.
In it’s simplest form, Decision Records are organised documents, that track decisions that are made along with their context.
Implementation
Setting up Decision Records for your team is pretty straightforward. The challenge, however, lies within the fact that it is not so much a technical implementation, but a cultural one. It’s change management. Because, having Decision Records is useless when they’re not embedded into your team’s processes. That’s why I’d like to walk you through implementation alongside three phases: the development, the deployment, and the evaluation.
1. Development
Developing a Decsion Record system that works for your team is about collaborating on picking a fitting template and structure. There are a few notes I can share on this.
Keep It Stupidly Simple. The most popular ADR format contains four components: The status, the decision, the context and the consequences. Perfectly fine to start with. This goes for structure as well. When I started out, I used the various data platform components as the structure for my Decision Records. To give a simplified example:
-
DR000: We need a Data Platform
-
DR004: We need orchestration manage our ELT processes
-
DR004.1: We use Airflow for our workflow management
-
DR004.1.1: We use Patterns within Airflow through TaskGroups
This shows how you can build on previous decision records, from high-level decisions and capabilities, to your desired granularity.
Another key in the development of your system is to optimise for engagement. Decision Records are a team effort. This is not only how they’ll gain support and traction, it also invites for deeper collaboration and shared learning experiences. One way of achieving this, is to encourage interaction by implementing a minimum reviewers & approver threshold, and allowing for discussions close to where the records are stored. A good candidate would be a git repository.
2. Deployment
Now we get to the exciting stage, the deployment. How can you embed Decision Records in your teams processes, and actually work with them?
The value of DRs is driven by the network effect. The more people use it, the higher the value. In practice, this means that a successful deployment happens only when there is at least one advocate: A person that is determined to make Decision Records work, and motivates and involves the team to make it happen.
To clearify, someone must take the lead to maintain momentum and give direction, ensuring that the team understands the importance of thorough documentation and commits to the practice over time. Without this guidance, it’s easy for the deployment to stall or become disorganised, where the full benefits of DRs remain untapped. The team role of the advocate is irrelevant, when healthy team dynamics are in place. And since you made it this far, it could be you!
One way of embedding DRs into your process, is to employ a technique called Habit Stacking*.*Habit Stacking means that you use current habits to build new habits. In our context, this means that you can decide to process new and changed Decision Records at your (bi-)weekly team meeting. The structure is already in place, and just requires you to extend the fixed agenda items.
3. Evaluation
Evaluation is a crucial aspect of (cultural) changes. This is the stage where you ask questions. Are we motivated? Are we consistent? Are we getting as much value out of DRs as we expected? What can we improve? In my experience, every team has their own way of evaluating. It is highly recommended to evaluate DRs and their implementation, too.
Hi, I’m Bastiaan 👋🏼 I write about the Modern Data Workflow, where I explore tools & processes to supercharge your data capabilities. Follow me for more!