In all my years in the world of data analytics, I have never seen as much forward progress and momentum as I have since the start of the COVID-19 pandemic. Idle debates about the merits of one technology vs. another are fast-becoming a thing of the past. Today, everyone is hyper focused on specific business challenges and how to solve them – and that’s exactly how it should be.
Back when I got my start at Oracle in 1997, data analytics was still a relatively new field. You couldn’t buy a system off the shelf – you had to pick out components, figure out how to make them work together, and then keep the system running. Nobody was betting their business on data. All that mattered was getting business systems like ERP and SCM to work in the first place. This of course requires very special expertise, and so virtually all data analytics projects and technologies at the time were firmly under the control of a company’s IT professionals.
Now that’s all totally normal – every successful enterprise technology goes through an early adopter phase, when the people who love getting hands-on with technology before all the bugs are worked out are the ones who call the shots. But you know what’s not normal? Having a technology get stuck in the hands of IT specialists for 30 years.
The astronomical rise of computer processing power from the 1940s to today. No other technology in human history has improved at this rate – not even close. (Source: Nvidia for The Social Dilemma, a Netflix documentary)
It’s unbelievable to think that anything related to computers could stay more or less stagnant for so long and yet that’s exactly what happened with business analytics. The result is a surreal world where computing power has risen exponentially, yet companies spend millions of dollars on data projects that routinely fail to live up to expectations. Stranger still, until recently, there didn’t seem to be any expectation that analytics professionals should be held accountable.
We’d been living in this crazy alternate universe for so long that I was starting to think it might never change. For every one trailblazing visionary who wanted rapid, data-influenced decision making across their business, there were nine others who–for reasons that I cannot fathom–were perfectly content to carry on with business as usual.
Then COVID happened and all of a sudden, data analytics went from being a “nice-to-have” with plenty of room for tinkering, experimentation, and foot dragging – to a lifeline that desperately needs to work, right now.
It’s as though companies saw the future of their business flash before their eyes. Suddenly the absurdity of decades exorbitant budgets with little to show for the investment; horrendous timelines for delivering analytical solutions, and practically zero accountability–things they would never tolerate in other parts of the business–became intolerable. At long last, a great reckoning had finally come to the world of data analytics.
It’s been a long time in the making. If you zoom out, you will see several trends that have been on a collision course with our antiquated approach to analytics for years. Taken together, these trends underscore the importance of data analytics as this new phase gets underway:
The Way Companies Approach Data is Rapidly Evolving
Companies are rapidly shifting from using data to understand their business to data becoming central to the business model. Examples include Facebook bringing data to advertising, Netflix bringing data to entertainment, Stitchfix bringing data to retail fashion.
Across the board, we are also seeing major changes in the way companies think about, deploy, and manage data projects. Today, data is fast becoming a core driver of decisions being made at every level of the organization.
It’s no surprise then that the value of IT is now being measured by its ability to expand access to data—enriching, distributing and securing clean and accurate data sets that can be utilized by data consumers throughout the organization. Individuals and teams want to spin up experiments on the fly, testing and proving value up front and scaling in stages rather than asking for large investments based on little more than a plan.
IT Teams are Consolidating and Simplifying their Technology Stacks
IT teams were already overextended before the pandemic arrived. In recent months, the challenge has only intensified.
IT teams struggle to keep digital transformation on track with expectations—especially when it comes to data. The backlog is growing faster than IT can find solutions and they are falling further and further behind every day. While data stockpiles are growing exponentially, the number of insights discovered from one year to the next barely inches upwards.
That’s in big part because today’s analytics technology stacks require lots of experts in very specific technologies to make the tools work, preventing people with deep expertise on the business side of the equation from making an impact. Instead of addressing the root causes, however, IT teams are stuck fixing issues created by tools that never truly solved their problems in the first place.
The Cloud is Breathing New Life into Data Lakes
We are at a point now where the cloud has proven that it’s a safe environment to run the most mission-critical applications and store the most sensitive data. Banks, healthcare organizations—even the U.S. Federal government—are all embracing cloud technology with open arms.
Cloud storage options like Microsoft ADLS Gen2, Google Cloud Storage and Amazon S3 are now setting the standard for performance, security, availability, scale and cost. With the broad adoption of the Parquet columnar file format, which any query engine that’s worth its salt supports, companies now have both data durability and analytics performance without the risk of vendor lock-in.
Now, data scientists, AI practitioners and business analysts can all come together on the same data set, using the same definitions, and without making copies or changing the data. This expands and accelerates data enrichment, resulting in data that is more accurate, contextual, timely and all-around better for everyone.
As the initial shockwaves of the COVID-19 pandemic fade, more and more companies are starting to realize that even though the world is transforming in profound and unexpected ways, it’s not all doom and gloom. One of the bright spots is that we can use data to find new paths forward – charging into the future with newfound clarity of purpose, priorities, focus and drive.
Which side of history will you come out on? Will you be remembered as a leader who spent these critical months fiddling with the knobs and dials of an outdated data analytics tech stack wondering why “best of breed” isn’t working? Or will you be the one who looks up, sees the tidal wave of change that’s already happening in the world of data analytics and catches a ride?
Witness the astonishing speed of Incorta at our live product demo, which takes place every Wednesday. Learn how you can easily access and seamlessly connect all of your data, achieve lightning-fast analytics performance on your transactional data, and enable true self-service reporting with real-time data.