Executive Summary
The world is a very different place than it was just two years ago. Through the COVID-19 pandemic, economic upheaval, numerous natural disasters, political turmoil and seismic social change, many organizations have struggled to survive.
In a world full of uncertainty, organizations are turning towards data and analytics to illuminate the path forward. While the adoption of data analytics was already on the rise leading up to 2020, the global pandemic accelerated this trend while also upending how organizations are approaching the discipline.
To start, 18+ months of turmoil has rendered some historical data unusable, pushing organizations to temporarily abandon data science projects that are especially reliant on that type of data (i.e., forecasting, predictive analytics) and increase their investments in other areas, such as external data sources and descriptive analytics.
Furthermore, insight-driven organizations who had access to real-time enterprise data were able to respond to the crisis as it played out, resulting in a substantial competitive advantage. For those who still lack that ability, the pressure to expand and accelerate data access is rapidly intensifying.
Last but not least, organizations have realized that achieving true situational awareness requires the integration of multiple data, analytics and AI tools and solutions. In order to achieve this, organizations are looking to cultivate enterprise-grade analytics ecosystems - modular data systems whereby subcomponents can be selected and assembled in a multitude of ways to satisfy specific user requirements. Also referred to as “composable analytics” or “composable data fabrics,” achieving this level of integration requires, first and foremost, a holistic, enterprise data strategy.
With all of these changes and challenges afoot, we at the AI, Data & Analytics Network surveyed 400+ data leaders to find out how organizations are leveraging data, advanced analytics and Artificial Intelligence (AI) in the post-pandemic age.
Our Respondents
Our 400+ respondents come from over 100 countries and across dozens of industries. Top job titles included CIO, Chief Data Officer, VP IT and Head Data Scientist.
Breakdown By Region
Unlocking the Power of Enterprise Data
Organizations create and store almost unfathomable amounts of data every second of every day. In fact, the average human generates 1.7 MB of data per second. However, data within itself is not valuable. In order to harness the limitless power of enterprise data, it must be transformed into insights and shared with decision makers.
Organizations are realizing en masse that a siloed approach to data management - where the finance team uses financial data to enhance financial decisions, the HR team uses employee data to improve HR processes, and so on - only offers so much in terms of visibility, scalability and value. In order to truly capitalize on data assets, organizations must embrace a holistic, enterprise approach to data management.
In fact, over half (53%) of our respondents already have an enterprise data strategy in place while an additional (36%) confirmed they were planning on implementing one within the next 5 years.
Do you currently have an organizational data strategy in place?
In terms of the types of enterprise data companies are currently using, the range is quite diverse. While 66% of companies are using financial data, 58% are using customer/CRM data, 46% are using employee data and 42% are leveraging sales & marketing data. At 17% and 16% respectively, manufacturing data and IoT data came in at the bottom most likely because, as this was a cross-industry survey, these areas were only relevant to a fraction of our total respondents. Overall, these numbers suggest that most companies who implement enterprise data strategies are, at the very least, successful in breaking down data silos and facilitating data sharing across functions.
What types of internal data does your organization use for decision-making?
Reaping the Rewards of Data Analytics
Data and analytics are, and will always be, powerful enablers of process innovation and business performance. With that in mind, it is no surprise that 43% of respondents selected “increased operational efficiency and productivity” as a top benefit of their current data analytics strategy.
A far off second, 12% of respondents selected “enable digital transformation.” Furthermore, 9% chose “leveraging data to improve enterprise-wide decision making” and 6% said “improve customer experience.”
ALSO READ: AI and Government Modernization
However, when asked where they saw data and analytics delivering the most value in the future, there were some considerable shifts. This time around, 27% selected “increased operational efficiency and productivity” followed by “enabling AI, ML and/or predictive analytics” (12%) and “identifying & generating new revenue streams” (11%).
Instead of leveraging data to simply reduce costs, organizations are increasingly looking to use data to drive growth and innovation. While for some entities this might mean developing new, data-driven products and revenue streams, for others it could mean leveraging AI to deliver predictive, strategic insights. What is clear though is that companies are starting to create their own, unique vision for data science success.
“If artificial intelligence is your penthouse, then your data management strategy is your foundation. What most companies do wrong is that they try to build the ‘penthouse’ before the ‘foundation.’ You have to focus on data quality and governance processes first.”
Harjot Singh
Chief Data Officer, RAC WA.
In what area does data add the most value to your organization?
Despite the real and potential benefits of data analytics, these initiatives are not without their challenges. When asked about obstacles to adoption, 30% of respondents selected “budget limitations” while 27% cited “lack of skilled talent.” The technical talent shortage has been well documented by mainstream media outlets and research institutions. In fact, QuantHub estimates that, in 2020 alone, there was a shortage of 250,000 data science professionals. In an effort to close this gap, many organizations are not only rethinking the talent acquisition and engagement strategies, but they’re also looking to grow their own data scientists. With the emergence of low-code automation and self-service analytics, accessing data-driven insights is easier than ever. These tools combined with effective data literacy training can reduce the burden on existing data teams by empowering “citizen analysts.”
“As a business framework, data science can be applied to both drive cost effectiveness and identify levers of growth. Driving cost efficiency is the low hanging fruit - the place to start if you want to deliver quick wins.”
Shiv Kumar
Head of Analytics, Schlumberger
Data & Analytics Funding Priorities
Budgets for managing and actioning data are certainly on the rise. In fact, 56% of our respondents expect their data budgets to increase while only 3% said they anticipated their budgets to decrease.
Do you expect your final, approved budget for managing and actioning data & analytics to increase over the next year?
What percentage of your budget is allocated towards data & analytics tools & services?
In terms of where exactly this funding is going in terms of solutions, the market is surprisingly stable and diversified. However, the top area of investment, both now and down the line, cloud data warehousing solutions - a service that collects, organizes and securely stores data.
As evidenced here, organizations are abandoning their legacy on-premises data warehousing technologies, such as Hadoop, for these new cloud data platforms as they offer more agility, computing power and access. Not only are they easier to use, but they also offer more computing power for less money and are easier to scale.
When it comes to future funding priorities, though cloud data warehousing remains a top area of investment (29%), it’s clear many plans to re-allocate their budgets towards specific mature applications of data science such as predictive analytics (29%), AI/ML (26%) and workflow analytics (25%). This makes sense as, for many organizations, data warehouses serve as the foundational technology that supports all other data science initiatives and priorities.
Other areas we anticipate experiencing significant growth are:
- AI/ML Monitoring solutions (+7%)
- Deep learning frameworks (+3%)
- Enterprise data catalogue solutions (+2%)
[inlinead-1]
This points to the fact that organizations are growing increasingly committed to building out their AI/ML/deep learning capabilities. At the same time, more and more AI applications are making it out of production and into the real-world where they can be tracked and optimized.
In addition, the increased interest in enterprise data catalogue reflects a growing desire to make data analytics as accessible to as many people as possible - a trend we’ll take a closer look at later in this report.
Though companies expect to invest in some areas more than others, it should be noted that no single solutions category surpassed 30% or fell below 6%. To us this indicates that organizations are rejecting a one-size-fitsall, “follow the leader” approach to data analytics. Instead, they are all paving their own way based on their own unique objectives, limitations and data policies.
Though many organizations are still in the early stages of development when it comes to fleshing out their enterprise data strategies, going forward, we expect data analytics to continue to evolve from transactional to strategic and “intelligent.” Instead of building data systems and strategies around compliance and/or lean-inspired costcutting methodologies, organizations are beginning to build a vision for value-based analytics - data analytics that drive growth and deliver tangible ROI.
“Data analytics is the glue that pulls together technology and ideas into one cohesive vision. If you want to help the business to achieve a certain goal and reverse engineer a solution, you have to understand the business.”
Catherine Lopes
Head of Data Strategy & Analytics, ME Bank.
What data & analytics solutions have you invested in over the past twelve months? And what do you expect to invest in over the next year?
Data Democratization
Data is more than just the currency of business, it’s the lifeblood of growth, innovation & profitability. As enterprise data and analytics emerge as critical drivers of competitive advantage and strategic performance, ensuring that all employees- not just IT analysts and data scientists- have access to actionable data is more important than ever. This is where the concept of data democratization comes in.
Beyond providing non-technical users access to enterprise data systems and data lakes, the real goal of data democratization is to transform all employees into data scientists; a pivotal step towards establishing a data-centric enterprise. Accomplishing this requires both the implementation of new, user-friendly self-service analytics dashboards along with comprehensive cultural transformation and the reskilling of talent to ensure data literacy.
However, at this point in time, many companies are a long way from achieving these ambitions. In fact, 32% of those surveyed said that less than 10% of employees at their company have access to data. Only 6% said that more than 76% of their workforce had access to data and analytics.
Looking towards the future, this seems to be changing. To start, 44% of our respondents say they expect their data democratization budgets to increase over the next year. In addition, as mentioned before, organizations are currently and expected to continue making significant investments in data democratization-enabling tools such as enterprise data catalogues and cloud-based data warehouses plus another key area of investment that we have yet to get to: data visualization.
Do you expect your final, approved data democratization tools budget to increase over the next year?
What percentage of your budget is allocated towards data visualization tools & services?
Data Visualization
A powerful enabler of data democratization, data visualization (DV) refers to the translation of numerical or statistical information into a visual or pictorial format (i.e., map or graph). As images can be more intuitive than words or numbers, this process makes it easier for humans to identify and derive meaning from trends, patterns, and outliers found within the data.
For the data scientist, data visualization is a powerful tool for communicating large, complex and esoteric data sets to business leaders and other non-technical data consumers (i.e., front line staff, customers, etc.).
ALSO READ: Reimagining AI, Data & Analytics
Instead of simply displaying metrics, data visualization enables data scientists to craft a compelling narrative around the data - a process known as data storytelling.
In other words, it closes the gap between advanced analysts and a broader, general business audience.
When asked “how is your organization leveraging data visualization,” a significant majority of our respondents (69%) said it was “to create more intuitive dashboards for routine reporting.” Following that, they selected “to create better ad hoc reports for internal usage” and “to democratize data and enable self-service analytics.”
As organizations look to harness the power of enterprisewide data-driven decision making via self-service analytics and business intelligence dashboards, organizations are increasing their investments in data visualization tools. In fact, 56% of our respondents say they expect their budgets for such tools to increase over the next year.
What percentage of your budget is allocated towards data visualization tools & services?
Do you expect your final, approved budget for data visualization tools to increase over the next year?
Advanced Analytics
Advanced analytics (AA) encompasses a broad range of mature (and buzzworthy) data science techniques such as predictive modelling, machine learning and process automation. Through the use of these tools, data scientists and organizations can gain deeper insight into data, transforming it into actionable information and/or predict the likelihood of certain events.
One term often confused with advanced analytics is “big data.”
In actuality, “big data” is simply a term that is used to describe data sets that are too large or complex for ordinary computing devices to process. As such, “big data analytics” refers to the use of advanced analytic techniques to analyse “big data.”
However, “big data” is not a requirement for advanced analytics. While some companies - usually the larger, more resource and data-rich organizations - are diving headfirst into expensive, large-scale advanced analytics such as High-Performance Data Analytics (HPDA) and AI, many others are instead embracing a more calculated approach.
Instead of trying to follow in the footsteps of the Googles and Apples of the world, many organizations are embracing “smaller” analytics, mature techniques that deliver high levels of strategic value but require fewer computing resources than “big data” and large AI models.
What types of advanced analytics are you currently leveraging?
In fact, almost all of our respondents are currently leveraging advanced analytics in some capacity. At 54%, predictive analytics is the most popular AA method amongst those surveyed. Given that the use cases for predictive analytics are wide-ranging (i.e., personalized marketing campaigns, fraud detection, demand planning, etc.) this comes as no surprise.
Following predictive analytics, 45% are leveraging machine learning (ML) and data visualization. As mentioned earlier in this report, all 3 are expected to be key areas of investment in years to come. In addition, 55% of our respondents expect their advanced analytics budgets to increase over the next year.
Despite the promise of advanced analytics, one thing that might be holding the discipline back is that it continues to have a high failure rate. In fact, 53% of respondents said that less than 25% of advanced analytics models have “met or exceed business expectations” in terms of value delivery.
ALSO READ: Democratizing Data & Analytics
There are myriad potential reasons for this outcome. While it’s true that mature modelling methods are difficult to execute from a technical standpoint, the issue could also be more cultural. If the business does not understand what data science can and cannot do or how to effectively use datadriven insights to improve decision making, these efforts will inevitably fail.
Do you expect your final, approved budget for advanced analytics to change over the next year?
Do you expect your final, approved budget for advanced analytics tools to increase over the next year?
What percentage of your budget is allocated towards advanced analytics solutions and services?
Data ROI
While enterprise data will always be used to enhance decision making and monitor operational performance, leading-edge organizations want more. Increasingly, companies are looking to monetize data by using it to not only optimize business processes but drive real-tangible revenue growth.
“Data monetization” is the use of data to obtain quantifiable economic benefit.
Generally speaking, there are two ways companies typically monetize data:
- Indirect Data Monetization - using data to develop new business models, identify untapped markets and/or boosting operational performance
- Direct Data Monetization - the sale of data to 3rd parties
According to our research, 35% of respondents have already launched at least one formal data monetization initiative and an additional 29% plan to do so in the next 2 years. Only 21% said that monetizing data was not a priority. As expected, only a very small percentage (7%) are achieving data ROI directly by selling their own data, a high risk and increasingly scrutinized endeavour. The most common forms of monetization are “the use of data to optimize operational efficiency” (39%) and “the use of data to develop new products, services & customers” (35%).
Over the next couple of years, focus on data ROI is expected to intensify across the board. In fact, when asked about upcoming data monetization/ROI initiatives they plan on launching within the next year, 51% said “the use of data to optimize operational efficiency,” 49% selected “the use of data to develop new products, services & customers,” and 24% selected “data wrapping” (up from 19% in the previous section). Once again, only a very small percentage, 8%, selected “direct sale of data to third parties.”
In addition, 40% of respondents said they expected their budget allocations for data ROI initiatives to increase over the next year.
Has your organization developed and/or actioned a data ROI & monetization strategy?
If yes, what type of data monetization/ROI initiatives is your company currently running vs. plan to execute in the future?
Do you expect your final, approved budget for data ROI & monetization to increase next year?
What percentage of your budget is allocated towards data monetization tools & services?
Artificial Intelligence
In the 70 years since Alan Turing released his seminal paper, Computing Machinery and Intelligence, Artificial Intelligence (AI) has gone from a theoretical concept to an integral part of our everyday lives. From recommending what to watch next on Netflix to optimizing Google search functionality, there are few areas of our lives that haven’t been impacted by AI.
That being said, AI is still very much in its infancy and even the most advanced organizations are still figuring out how to use it safely, ethically and effectively. However, this is to be expected.
In fact, according to our survey, almost half of our respondents (48%) have yet to launch their first AI initiative and an additional 28% are still in the early, experimental stages. While 15% are already using it to help run their dayto-day business, only 5% say they are actively using AI to deliver transformational results.
Reinforcing this point is the fact that over half of respondents (53%) say that less than 10% of AI models are effectively used to enhance decision making. Similarly, 59% of respondents cited that less than 10% AI models even make it into production.
Where does your organization fall on the Artificial Intelligence Maturity Curve?
Though these numbers may seem low, they’re certainly not surprising as building, training and validating an AI model is an incredibly complex process that requires an abundance of expertise, time and computing resources. Bad or biased data, issues with interpretability / explainability and architectural defects are just some of the factors that can easily derail an AI project.
However, despite these difficulties, 54% of respondents say they expect AI budgets to increase over the next year. This is likely because, though the cost and difficulty associated with AI deployments is high, organizational leaders know that those who fail to operationalize AI within the next few years will be at high risk for obsoletion within the next decade. Furthermore, the potential of AI to deliver substantial financial benefits is simply too great to ignore.
Though at this point, 60% of respondents are allocating less than 10% of their overall budgets towards AI, we do expect this to change over the next few years. Like any other new technology, it will probably take about a decade for AI to start delivering. Not only will AI methodologies start to deliver tangible business outcomes, but it’s also likely these technologies will transform and disrupt entire sectors by 2030, if not long before.
Like with any major technological breakthrough (i.e., the internet), it will probably take most companies about a decade to build out the talent, technical infrastructure and strategic vision needed to truly capitalize on AI. In other words, the race towards AI has already begun even though the finish line may be years away.
What percentage of AI models make it into production?
What percentage of your AI models are successfully used to enhance decision making?
Do you expect your final, approved budget for AI development to change over the next year?
What percentage of your budget is allocated towards AI solutions and services?
Data Governance
Data-driven insights and innovations are only as accurate as the data going into them. As the old adage goes, “bad data in means bad data out.” Ensuring data accuracy across the data management and analytics lifecycle is not just about delivering meaningful business insights (though that is very important), but also about building trust with customers, employees and other stakeholders.
The Data Governance Institute defines data governance as “a system of decision rights and accountabilities for information-related processes, executed according to agreed-upon models which describe who can take what actions with what information, and when, under what circumstances, using what methods.”
In other words, data governance provides a logical structure for classifying, organizing, and communicating complex activities involved in making decisions about and taking action on enterprise data.
Though traditionally considered a compliance issue, as organizations look to capitalize on their data assets, data governance has evolved into a strategic imperative. This is because data governance increases the quality, usability and accessibility of data. Without some sort of “rules of engagement” most companies will find themselves struggling to manage, protect and leverage data assets.
As such, 48% of our respondents expect their data governance budgets to increase over the next year and 21% plan on investing in data governance-specific solutions over the next year. In addition, as organizations increasingly rely on data to drive decision making and innovation, these investments will increase in years to come.