Earlier this year, NewVantage Partners published their 2022 Data and AI Executive Leadership Survey. Of the enterprise data officers, chief analytics officers, and CIOs surveyed, just 26.5% said they’ve created a data-driven organization. Over 90% of executives cited culture as the primary obstacle to becoming data driven. About 8% cited technology.
I’m with the 8%.
For the last 30 years, organizations have struggled to get to the first level of analytics maturity: timely, accurate descriptive analysis. Most are still stuck doing that same kind of analysis. Few have advanced to predictive analytics; even fewer have a mature program of machine learning and AI — the kind of analyses that would make them data driven.
The amount of technology investment has been enormous. But we’re trapped in a culture driven by the predominant technology of the last 30 years: the data warehouse and star schema. To change the culture, we have to change the technology.
An Old, Tired Story
The culture that’s grown up around the data warehouse is based on the old, tired story that the IT team doesn’t think the business knows what they want, and the business doesn’t think that IT understands their needs.
It’s true that neither side is getting their needs met and both sides are frustrated. But it’s not so much a lack of understanding of each other’s needs as it is a lack of technology to meet them.
The business wants all the data now and they want fast query performance. IT may want to give them all the data but knows the performance won’t be there unless they curate the data down into star schemas and business views. That means the business has to wait a long time for less data than they really want. Everyone is trapped in a vicious circle.
If we want to do next level analysis and create a data-driven organization, we need to change the way that data moves from source systems to the business, eliminating the need to build star schemas altogether.
No Need to Summarize
Incorta does that by giving people access to the data in what is known in the star schema world as the “raw data landing zone.” This is the data right out of the application with just a little cleanup but no aggregation or transformation.
Most people in IT today would say, “That would not be a usable format. It has to be aggregated and summarized for performance reasons.” With advances in columnar databases, cloud storage, in-memory computing, and our proprietary Direct Data Mapping technology, that is no longer true.
With Direct Data Mapping, we create a map of all of the relationships of all of your data. We do not aggregate, filter, or summarize. It’s effectively a single virtual schema that allows you to assemble business views on the fly. It has row-level security. There are no limitations to cross-querying, or performance. Business users can ask any question at any given time.
That changes the culture, and opens the door to new types of analysis.
More Questions, More Analysis
The business doesn’t have to know all the questions they want answered. They don’t have to provide their business requirements. They have access to all of the data. That changes their relationship with IT.
It also changes the way IT works with data. Data engineers and data scientists can spend far less time on transformation and more time doing data science, working on machine learning algorithms, and deploying those algorithms into dashboards.
They can go out and find more data sources to add to the data hub. That improves AI and machine learning capabilities. Even descriptive reporting can benefit from access to new information.
The Next Evolution
In the next evolution of reporting, enterprises will want real-time access to detailed data found in billions of records. They can’t be prisoners to curated business views and overnight batch refreshes. Your data scientists can write a perfect algorithm that says, “If you have ten scones on your shelf at this store, at this location, at this time, you should discount them because they will all be thrown out at the end of the day.” But if you can only get the data refreshed overnight, it’s too late.
That’s a technology problem, not a culture problem. You can’t do things differently if the technology doesn’t support it. It wasn’t until humans had fire that they could cook food, gather in the evening, and move to cooler climates, among other things. All of those things changed our culture.
Leaders that see culture as the barrier for advanced analysis should shift their gaze to their technology. Data ingestion from source systems to the star schema is one of the oldest technologies in analytics today. When companies adopt solutions that bypass the star schema, the number of queries they can execute skyrockets — from 400 a day to their data warehouse to 70,000 queries a day to their modern data hub for one of our customers. That’s how you move up the maturity curve and become a data-driven organization.
The first step? Giving it a try for yourself.