Great Expectations, a leading open-source platform for data quality, has announced the results of a survey analyzing the main points of concern when it comes to the quality of their data.
The result was lackluster at best, with 77% of organizations listing quality issues and 91% saying that it’s impacting their company’s performance. This has understandably had the knock on effect of 13% having low trust in data quality.
The reasons for this were manyfold, with respondents stating that this low trust stemmed from broken apps or dashboards, decisions based on unreliable or bad data, teams having no shared understanding of metrics, and siloed or conflicting departments. Additional issues impacting data trust included alert fatigue, misalignment on certain metrics, and friction between teams.
Data confidence is critical for organizations to make informed business decisions, and data confidence today is rather poor. Another survey, by Deloitte, showed that more than two in three executives, 67%, say they are “not comfortable” accessing or using data from advanced analytic systems. In companies with strong data-driven cultures, 37% of respondents still express discomfort. Similarly, 67% of CEOs in a similar survey by KPMG indicate they often prefer to make decisions based on their own intuition and experience over insights generated through data analytics.
What both the Great Expectations and Deloitte study confirm is that many executives lack a high level of trust in their organization’s data, analytics, and AI, with uncertainty about who is accountable for errors and misuse. Data scientists and analysts also see this reluctance among executives — a recent survey by SAS finds 42% of data scientists say their results are not used by business decision makers.
As Abe Gong, CEO and Cofounder of Superconductive, the company that makes Great Expectations, explains, “Poor data quality and pipeline debt create organizational friction between stakeholders, with consequences like degraded confidence. This survey made it clear that data quality issues are prevalent and they’re harming business outcomes.”
Elsewhere in the Great Expectation survey, findings showed that data quality issues can make it difficult or impossible to see a “single view” of an end-user or service, lower productivity, obscure reliable performance metrics, and overwhelm development teams and budgets with data migration tasks. Data practitioners blamed poor data quality on lack of documentation (31%), lack of tooling (27%), and teams not understanding each other (22%).
Respondents also stated that data scientists spent too much time preparing data for analysts, while end-users complained about gaps in their data (such as lost transactions), and production teams were mired with delays.
“Data quality is critical to facilitate the making of decisions with confidence across the organization, enabling a singular understanding of what that data means and what it’s being used for. That’s why support for data quality efforts should be found at every level of an organization, from data scientists and engineers to the C-suite and board who have confidence in outcomes for decision-making,” Gong said.
However, the Great Expectation study showed that, despite everything, 89% of respondents said their leadership was supportive of data quality efforts, and that 52% believed leadership regards data quality with high trust. When asked about their company’s current approach to data quality, 75% said they validated data. Overall then, the data signals that the intention is at least there, and that it is simply dirty data that is the issue.
Fortunately, as this network highlighted last year, there are several tools at businesses disposal to help navigate this tricky situation. From data cleansing to data enrichment, to adopting better governance strategies that better reflect the data they represent, a smorgasbord of solutions are available. If your business is also lacking trust in its data, read ‘5 Data Quality Tools to Ensure Accuracy and Integrity’ now.
The Great Expectations survey was conducted in May 2022 by Pollfish, an independent research platform, leveraging responses from 500 information services and data professionals based in the United States (57% men and 43% women aged 18-54). 60% of respondents were employed at companies with 250 or more employees.