What You Need to Know Before Going on a Data-Driven Journey? 6 Key Areas

In the past X years, during which we’ve been assisting companies in transitioning from analog to data-driven entities, we’ve encountered many challenges and roadblocks. In this article, I’ve distilled them into 6 crucial areas. If shaping your company around data interests you, even a basic understanding of these will save you plenty of time and money.  Not to mention saving your nerves and eliminating uncertainty because, without grasping the broader picture, it’s easy to get lost in a sea of data-related terms like AI, digital transformation, big data, data warehouses, data engineering, machine learning, etc.

So, what are the top challenges you should know if you want to embark on the data path?

1. Business Goals and Company Strategy

The first and foremost step in becoming “data-driven” is to clearly define why you want to collect data in the first place, meaning what your business objective is. Only a clear definition of the objective allows you to precisely identify what data you want to collect and what tools, skills, and processes you’ll need.

Collecting data detached from business objectives is bound to fail because, sooner or later, a misalignment will occur between what you’re collecting and analyzing and what is genuinely needed by the staff – management, managers, and specialists.

The purpose of data collection must stem from the strategy of the company or a particular department. It should not be ad hoc, created for the “data-driven project.” Although, of course, if such a project provokes discussions about strategic objectives, that’s great – but objectives come first nonetheless, and everything else comes after. One of the questions I like to ask clients in this context is what they want to do with the data once they have it. It often turns out that different people from the Management Board answer entirely differently to this same question. Most often, it means that the business objective needs to be defined, specified, or agreed upon at the level of all departments.

So, you must determine the company’s development direction and strategic objectives before building a data-based organization. Only then will you match the appropriate data collection solutions to them.

2. Management Awareness and Organizational Culture

The term “data-driven” brings about certain misunderstandings, the biggest of which is the belief that working with data is a challenge strictly for IT teams or analysts. And although they are a vital and integral part of the process, working with data is, first and foremost, the responsibility of the management. 

If the management does not feel and understand the importance of data and doesn’t cascade this mission throughout the entire company and ensure its execution, the various divisions within the organization will, sooner or later, approach data collection casually and imprecisely. This, in turn—adhering to the principle of “rubbish in, rubbish out”—will translate into low-quality conclusions and analyses. The system will collect data and generate reports, only for the management to distrust them and revert to making decisions based on gut feeling or ego. And that’s precisely what you sought to avoid by creating a data-driven organization. Think of it this way: data system is like the nervous system of an organization—it touches every aspect of the company, from product development to customer acquisition. It allows for making better decisions faster, based on previous experiences. Therefore, you don’t want to treat it lightly as a “nice to have” or a side project someone else will take care of. The culture of using data and ensuring its quality represents a significant shift that needs to originate from the leadership.

3. Current Company Data

In numerous projects, I’ve encountered the sentiment that “We collect data and want to start doing something with it.” But is that really the case? Are you sure that the data (both in type and quality) your company is currently collecting holds value in what you want to achieve business-wise?

As I mentioned earlier, your business goal determines what data you need. Only after defining, for example, “we want to predict the behaviors of app users to increase their retention” can you determine whether the current data is valid. Whether you’ve been collecting it long enough, and whether it’s precise and consistent. Of course, asserting “we don’t have data” isn’t a catastrophe — it’s merely the starting point and a signal to begin collecting data in a manner that ensures its utility. Becoming a data-driven company isn’t an overnight success; it’s a journey that requires time and preparation. It’s essential to be aware of this so you neither expect transformation to occur overnight on the one hand nor give up if you’re starting from point zero on the other.

4. Tools for Data Collection and Analysis

When you visit any product page, like CRM or ERP, you’ll likely read that it will solve all your business problems. And while it might solve some, it typically does so within a specific domain, such as marketing or logistics. Rest assured, there’s no one-size-fits-all tool that will magically make your company data-driven. Achieving that status requires a suite of various tools, creating a complete architecture. I’ll delve into the architecture itself in the next step, but it’s crucial to know the tools that compose it.

Excel. Its low entry barrier and relatively extensive analytical capabilities make it a fundamental tool for most companies. However, as a company grows, Excel becomes insufficient: numerous tables emerge (i.e., many sources of truth), and smoothly transferring data between tables becomes a nightmare. This process can’t be efficiently automated, especially on a large scale. That’s why companies subsequently turn to domain-specific systems, such as ERP, CRM, SCM, CMS, etc. These systems structure data and automate its flow. It’s a substantial step forward but has its limits since each system has a different purpose and focuses on its domain.  This is fine if you want to confine data collection to one department, like sales. But if you aim to be data-driven and draw cross-sectional conclusions — you must go a step further.

Data Warehouse (data lake, big data platforms). A crucial point on the map — if you want to be data-driven, you must have a singular place where data from multiple sources converges.  Here, data is appropriately processed, i.e., prepared for “consumption”, either for direct business use (e.g., making a decision) or further processing. Notably, thanks to the development of cloud technologies, such systems are now reserved for more than just the largest players. For a Sales Director to utilize data produced by a data warehouse, Business Intelligence tools are needed. These solutions enable data visualization and reporting, allowing the sharing of analysis results so that non-programming-skilled individuals can easily work with data. Note that BI systems aren’t for connecting many data from various sources — that significantly lowers their efficiency. That’s where the aforementioned data warehouses come in.

Data Exploration Tools and Data Mining Systems elevate data analysis, enabling a profound understanding of gathered information and extracting valuable knowledge, which can drive strategic decisions. With them, you can catch business trends and patterns that are hard to spot when data is analyzed in isolation.

Meanwhile, AI and Machine Learning Tools assist in responding to current business events and predicting future trends and behaviors. Using data stored in systems like Data Warehouses, AI can analyze collected information, foresee future events, and automate specific decision processes. At the same time, ML learns to use available data, continually refining its forecasting and decision models.

5. Architecture

An architecture emerges when you align all the IT tools mentioned above into a coherent whole. This, not the tools themselves, is pivotal in a data-driven organization, forming the bedrock of a single source of truth. In the simplest terms, architecture aims to merge data from various sources into a single spot and process it into a form that allows it to be consumed—meaning put to business use.

In this entire process, 3 types of tools are employed:

1. Tools with Ready-to-Use Interfaces—like CRM or ERP systems. They don’t demand programming skills, so they’re the easiest to use, but of course, they’re not entirely hands-off. For instance, a salesperson must deeply understand the CRM they use; otherwise, they’ll input incomplete data, leading to incorrect analytical conclusions. These tools have the lowest entry barrier and can be used by many users, making them ideal for processes that demand flexibility. If you want to change a product’s price in CRM, you don’t want to involve a developer or analyst—your salesperson will suffice. But, for the same reason, they offer limited possibilities; they are confined to a particular business domain and the tool’s capabilities.

2. Data Analysis Tools—databases, data warehouses, and big data platforms based on SQL. This is the language of analysts; it is fairly simple and does not demand programming skills but requires syntax knowledge. The entry barrier is naturally higher than with interface systems, so managing these tools and implementing changes require specialists. SQL, on the other hand, offers massive capabilities: you can accomplish 90% of all analytical tasks using SQL and tools based on it.

3. Developer Tools—which require programming skills in languages like Python, R, Scala, or Java. Possibilities? Endless. To put it bluntly, you can do anything. Where SQL falls short, Python will surely suffice. At the same time, it is the least flexible tool as it demands the highest competencies—every change is associated with a new line of code.

And now, a question: why is it worthwhile to understand these categories?

Primarily because the architecture of a data-driven company encompasses all these tools, they are utilized for different purposes and in other contexts. Visual tools work best where many variables are in play, and processes can change on the fly. Developer tools are on the opposite side, fitting wherever the process is relatively static and repetitive. Sandwiched in the middle are SQL tools—deployed wherever data is being prepped to be in the correct form for users. From the system architecture perspective, none are less or more important – all have their role in the appropriate places within a data-driven company.

6. Skills

The last area, which hasn’t always been clear to our clients but is crucial when building a data-driven company, involves essential competencies. You need to know that working with data requires specialists with different, often conflicting skills and mindsets. This is most easily seen in the example of a programmer and an analyst, who both may be proficient in Python and SQL, but…

• A data engineer focuses on efficiently getting things done and needs to understand data sources, transformations, and how to deliver proper information for further use. For them, the process’s efficiency, consistency, and logic are paramount.

• Conversely, an analyst doesn’t prioritize efficiency as much but cares more about the outcome and the insights derived from the data. They aren’t concerned with the pace of the process but with the quality of conclusions; they need to fundamentally understand the data and its meaning, know how to interpret it and build models and dashboards so that the data is credible for users.

At Alterdata, we simplify this division into two main sections:

• Technology specialists who know how a system should be constructed to run smoothly,

• Business specialists who work with data and its resulting insights.

However, this is only a simplification because even within these two groups, you will find even more specific skill sets.

The challenge here is that you can only combine some of these skills in one person and must choose between hiring them all separately or outsourcing some competencies. Both approaches have pros and cons, but the most challenging part of hiring an exclusive team (besides the costs) is that you need only certain roles for specific periods. And this, by the way, is one of the reasons we created Alterdata — we cover the whole process of data analytics, and as we combine different skill sets within one company, we can engage in specific processes at specific times and return only when required.

Summary: 6 Key Takeaways Before You Begin Your Data-Driven Journey

1. Start by determining the company’s strategic direction and business objectives; they will define the next steps.
2. Building a data-driven company is not a weekend getaway; it’s a change in the entire organization’s culture, and a good example must come from management.
3. The data you collect and how it’s collected depends on the business goal of the project, not the other way around.
4. There isn’t one perfect tool that will transition you to a data-driven organization; you must use various tools and connect them.
5. Architecture, or the proper arrangement of tools, is the way to go.
6. You need various competencies that may mutually exclude each other, making it impossible to find them all in a single individual.
   

But if you do manage to find such a person… please, introduce them to us!