Data Management: Building a Data-Centric Future

Data is eating the world. It’s the new beating heart of business, from using artificial intelligence to predicting the optimal number of items on a lunch menu to addressing real-time healthcare diagnostic challenges. But all that data is powerless without data management.

What is Data Management?

Data management is the practice of collecting, storing, and using data securely, efficiently, and cost-effectively. More than that, it’s the activities and mindset to manage data as a valuable resource. The methodical collection and analysis of data are at the core of today’s technology capabilities. And organizations that don’t practice data management sacrifice the valuable business insights it provides.

For context, let’s look at some recent data on data:

• 92.1% of companies who’ve invested in AI and data management see a return on that investment (NewVantage)
• 91.7% are increasing that investment further (NewVantage)
• By 2025, global data creation will be 180 zettabytes, nearly double the 64 zettabytes created in 2020 (Statista)
• Data management spending in healthcare will reach $274.3 billion by the end of 2022 (Business Wire)

Why is data management important for digital transformation?

Data is at the core of digital transformation. Much of the data produced by businesses is seen as a byproduct, the outer hull we need to break through to get to the valuable seed inside. Digitally mature organizations have found ways to use their data to improve operations, understand their customers better and even transform their businesses. There are tons of real-world examples of this. 

Netflix began as a simple mail-order movie rental company and eventually innovated into the streaming space. Over time, it outcompeted other media delivery companies by using data to predict exactly what their subscribers wanted to watch next. While other companies have caught up to Netflix in terms of market share, it’s Netflix’s algorithm that’s made them so valuable. And that algorithm is entirely powered by data.

Amazon’s origins are as an online bookseller. After outcompeting others both online and in the brick-and-mortar world, the company set its sights on other retail sectors and grew to be a leader across the board. Eventually, they transformed into media, grocery, and cloud services. But it all started by selling books and collecting customer data.

One of the most essential planks in your Digital Transformation platform is an effective data management strategy. To make the best use of your data, it needs to be treated like an asset, not just a byproduct.

How do organizations treat data as an asset?

Going back to our definition of data management, the three broad activities: Collecting, Storing, and Using data all need to be done effectively and efficiently. 

Data as an Asset: Collecting Data

The data we collect needs to be clean, consistent, and trustworthy. It’s essential to understand how fresh the data needs to be for specific insights. In some cases, they need to be as near real-time as possible, like the mobile app giving you real-time directions based on traffic. In other cases, we rely on longer-term data, like using that same app to determine the fuel efficiency you get based on the season. 

Raw ingredients matter. And the reports, calculations, and models that consume the data will only provide good insights if they are fed useful, representative data. Organizations that do this best use a combination of people, processes, and technology. The right pipelines can move data around quickly and catch errors on the way, but the results are better when someone is responsible.

Data as an Asset: Storing Data

Treating data like an asset means emphasizing keeping it safe when it’s moving and at rest. The first things that come to mind are privacy and security. We are responsible for making sure only appropriate data is available to appropriate systems and people while remaining secure. The way data is stored may change if it contains personal information. Regulations like GDPR and CCPA mandate that people have the right to ask not to be tracked and even forgotten. 

Another part of keeping data safe can be summed up in Non-functional Requirements or what you may hear data architects call “the -ilities.” A few examples are reliability, availability, recoverability, maintainability, and scalability. Each of these should be considered when designing or picking a solution. It’s tempting to try and maximize for all of them, but there is no free lunch. They often run counter to each other, and the trade-offs all need to be evaluated. 

Data as an Asset: Using Data

Once data is collected and stored, we get to the good part: using it. The thread of security and privacy continues with the need to limit the use of data. Depending on your use case, there will be different methods in how you approach using your data.

Business Intelligence techniques like reporting and traditional analytics work best with highly structured data. They often use well-defined data stores like data warehouses running on SQL databases like Oracle, SQL Server, or PostgreSQL. This technique is great for getting specific information from stable data sets, but it’s a rigid model that costs a lot to change. 

Predictive analytics and data science techniques often work best with raw data from less structured sources. At MentorMate, we lean into the relatively new model, dubbed LakeHouse by Gartner. In a LakeHouse model, the data is stored in a flexible, less structured data lake. An extra layer retains most of the benefits of a structured database like data integrity, access controls, and access to data in a structured way. The model allows data scientists to feed their models and experiments with the raw data they need while giving business and data analysts the structure to build reports and dashboards effectively.

Final Thoughts

No matter what industry you’re in, there’s a ton of data surrounding your business. That’s true whether you take advantage of it or not. When managed properly, data is the key to unlocking valuable insights about your business that would otherwise go unrecognized. And the important thing to remember is that if you don’t utilize that data, a competitor probably will. And who knows, they could use it to unlock the next big disruptive idea that turns your industry on its head.