In today’s business landscape, data has emerged as a pivotal driver of success. As a start-up or scale-up, you understand the crucial role that data plays in gaining a competitive edge and accelerating growth.
However, as your company grows, so do your data needs, and keeping up with the ever-evolving requirements becomes a significant challenge. With a sea of data tools and technologies available, it’s easy to feel overwhelmed when it comes to selecting the right ones to build a robust data platform.
In this article, we will delve into the process of selecting the right data tools. We’ll start by examining the required individual platform components, ensuring a comprehensive understanding of your organisation’s specific needs. Then, we’ll shift our focus to the characteristics that align with your organisation, and finally — armed with these insights — you’ll be well-prepared to make informed decision on which data tools to select.
Example data platform with components
The Components
To select the right tools, you must first understand your needs. A data platform is essentially a collection of individual components, falling under three distinct component types:
-
Data Stores
-
Capabilities
-
Products
1. Data Stores
Data stores are the physical locations where your data is stored. Two common types of data stores are data lakes (unstructured data) and data warehouses (structured data).
When evaluating the data stores you need, consider the nature of data that will populate your platform: Identify the file types you need to support and the expected volumes of incoming data, as well as how the data will be ingested. Also think about how data will be accessed within your organisation: will data stores be distributed across different departments, or will they be centralised in one or a few locations?
Take a moment to write down the data stores you need.
2. Capabilities
Capabilities are the tools and processes used to interact with the data within the data stores. Key capabilities are:
-
Integration — moving data from outside the platform into a data store, or between data stores,
-
Transformation — applying business logic to data for analytical purposes,
-
Orchestration — scheduling processes and managing their dependencies,
-
Machine Learning — experimenting and developing ML models,
-
Governance — managing the relationships between data and the people/processes,
-
etc.
Of course, there are a lot more capabilities you can think of, a few examples are displayed in the figure above. Create a list of which capabilities are needed within your organisation.
3. Products
Ultimately, data platforms serve the primary purpose of creating and delivering business value through data. This value delivery is facilitated by products, with popular types being Dashboards (BI) and APIs.
Consider which products you intend to support with your data platform, both now and in the future. Identifying these products will help shape the direction of your platform, ensuring it aligns perfectly with your organisation’s objectives and growth aspirations. Take a moment to list the products you want to support.
By breaking down your platform into components, you gain a clear and comprehensive overview of your current requirements while also creating a guideline for future changes. Once you’ve figured out the components you need, it’s time to move on to the next step.
The Characteristics
Choosing data platform tools is always about trade-offs, this makes it — in my opinion — less about the features of every tool, and more about its characteristicsthat matter to the organisation. Finding the right tools is about finding the optimal balance between the speed of delivery, scalability, and the associated costs.
Venn diagram of speed, scalability and costs
-
Speed — speed of delivery is the time required to spend before you can use the component within the data platform. You can also opt to include the required maintenance time here.
-
Scalability — with the scalability of the component, you should try to assess the component’s long-term performance within the organisation, i.e. its ability to withstand future changes.
-
Costs — costs are pretty straightforward. How much bang do you get for your buck, i.e. how cost-effective is your option?
For every characteristic, figure out its importance to the organisation. Collaborate with your stakeholders to distribute 10 points across these three characteristics according to their importance, so that the most important characteristic gets the most points. e.g. Speed: 4, Scalability: 4, Costs: 2.
The Process
Now that you know which components you need, and the characteristics that matter, it’s time for the decision process. During this exercise, you try to pick the best solution for every platform component in three steps:
-
Researching the available solutions for every component
-
Ranking the solutions per characteristic
-
Calculating the final scores
1. Researching the options
Work your way through your component list, and research available solutions for every component. Try to find at least 3.
💡 Be creative! Not all ideal solutions will be readily available. Sometimes it’s necessary to think beyond the standard options out there to find a better match. To help you with this, write down the most expensive option, the quickest and dirtiest option, and the most embedded option.
2. Ranking
If you have all available options, rank them based on the characteristics together with your stakeholders. How does each option score against their alternatives, and why? As an example, let’s look at the data warehouse (all numbers are fictional):
The higher the ranking, the more points the solutions should get. To accomplish this, we can invert the ranking to get the points.
3. Calculating the final score
The final score per solution is the sum of the points of the ranking, multiplied by their importance, respectively.
Final score calculation
In this case, Redshift would be the preferred choice as the tool for the Data Warehouse, though Snowflake is very close.
It’s crucial to document why the ranking — and subsequently the points — is like it is. Document the calculations, coupled with all the considerations that were decisive. This is critical input down the road, where you may want to reconsider tooling due to changing business conditions or characteristic weights.
Some final words
Remember, building a robust data platform isn’t a one-and-done task — it’s an ongoing process. Regularly reassess your data needs and evaluate the effectiveness of the tools you’ve chosen to ensure they continue to meet your requirements. As your company scales and evolves, be prepared to adapt and modify your data infrastructure accordingly, to stay ahead of the curve.
In conclusion, the road to a successful data platform may seem complex, but by being proactive, flexible, and well-informed, you can effectively navigate the challenges and make the right decisions that will propel your start-up or scale-up toward success.
Hi, I’m Bastiaan 👋🏼 Data Lead at a scale-up. I write about the Modern Data Workflow, where I explore tools & processes to supercharge your data capabilities. Follow me for more!