Six years ago, Geoffrey Moore, a leading management consultant and business author, said that companies lacking big data analytics capabilities are “blind and deaf, wandering out onto the Web like deer on a freeway.” Two years later, the Economist Intelligence Unit shared findings that the majority of business executives surveyed were still using “gut feeling” vs. data in their decision-making processes. Unfortunately, things haven’t improved much since.
Think about the last time you wanted to view insights from a new business data source in your company. I’d guess you had to ask for a report to be created, or even call a meeting to discuss the data. If you ended up in a meeting, I’d guess the conversations centered invariably around stale data, or missing information, raising the need for follow-up emails and meetings to allow time to search for answers somewhere deep in a hard-to-navigate spreadsheet, application or a data lake.
There is great frustration amongst a broad set of decision makers today—it’s difficult for them to get the information they want, and they certainly aren’t able to do it in a timely way. Business decision makers are conditioned to accept a dependence on analysts and IT, and at the same time, IT is overwhelmed by the growing number of systems being deployed. These systems are increasing the volume and variety of data—not to mention the ever-expanding expectation that it all be accessible in different formats, in different places, and at different times—all while maintaining high levels of security.
There are forces at work driving businesses to data dysfunction. I see four leading candidates:
I believe the biggest factor for data dysfunction is the new normal: the average company has hundreds of data-generating sources they rely on to run their business. According to the latest Netskope survey, the average Enterprise is using over 1,000 cloud application services. Individually, few of these business applications make it easy for executives to find insights, and these sources are almost always living solely in their own silos. This trend of depended fragmentation will only continue—as of early 2018, a search of ProgrammableWeb showed that there are over 19,000 public-facing data-generating systems with APIs, and that list is continuing to grow.
Few of these data sources and business applications are integrated with others, and virtually none of them make it easy for an average business user to find information. According to a Dresner Market Study from 2017, 67% of business leaders surveyed think that BI hasn’t been successful in delivering its promise. According to Forrester, 69% of unstructured data that’s been collected by organizations isn’t being used at all.
Distributed Decision Making
The reality in today’s world of business is that almost everyone is a key decision maker. The CEO and management team make strategic decisions about the business. The next layer of management makes operational decisions, and the rest of the organization are making many tactical decision daily. Outside of the business, the supply chain and partners also are decision makers, and like those internal to an organization, can improve productivity and business outcomes throughout the business ecosystem—if they have access to relevant and timely data. Decision making has largely been moved down the organization because of better access to data enabled by technology that’s making organizations run faster.
But this decision-making isn’t leveraging data insights efficiently. In a 2016 article, Fortune magazine found that even in our age of information and data, business activity and decision-making are actually slowing down, not speeding up. Data is overwhelming, siloed, and not easily accessed, which for many leads to paralysis instead of enlightenment. There are too many data sources, too many tools connecting to siloed data, and a shortage of data talent needed for decision makers to get out of a state of immobility. Under these conditions, data makes us question our own decisions, or even paralyze, instead of shining a light on the path forwards.
Greater Demands of Our Data
Regardless of where companies are in their data-driven journey, or how big or small they are, they are faced with greater demands for information from customers, prospects, partners, vendors, and employees—in real-time, and globally. Employees are demanding that their companies provide equal or greater technology as a tool to help do their job, as they expect from their own mobile applications used to run their personal life. All stakeholders inside and outside the organization want information on-demand or in ‘real time’ to optimize business outcomes without having to rely on utilizing complex software tools or human gatekeepers in order to reach relevant information.
The Legacy of Traditional Approaches
Traditional tools for collecting and sharing data—spreadsheets, PPTs, BI platforms, meetings—can’t scale or perform to meet the needs of today’s data-thirsty, highly competitive organizations. These tools were built for a world of information workers doing their work at their desk, on their desktop computer, using client-side installed applications.
Because they weren’t built from the ground up for web, cloud, or mobile, and they often don’t surface insights that are easy to understand and act upon, existing investments to capture and leverage data from systems across the business aren’t scale effectively. Put otherwise, scale was previously limited by physics—the amount of computing resources you were willing to pre-invest in to enable scale. In the past, leaders that chose to invest on the low side have been met with gripes from internal and external customers about lack of speed and capacity. Leaders that chose to invest on the high side were rewarded with institutional waste and resources sitting idly by while capital was tied up in unused capacity.
Traditional, and even more recent, approaches to solving the real-time data insights challenge are time consuming, costly, and repetitive. They require IT, data, or BI professionals to invest in a collection of third-party technologies and services to get data into a useable format, before data can even begin to be analyzed or delivered to business decision makers. None of these solutions are complete platforms or natively integrated, so they add to the burden already shouldered by IT, while negatively impacting IT’s ability to serve the business with timely data for different audiences from multiple sources across the organization. This challenge is particularly complex, given that levels of data analysis skills are highly varied across the population. In most organizations, there tends to a gap between those who need timely access to data for decision making and those who have the skills needed to access and analyze data.
This reality creates a problem for organizations: choosing which business problems to solve first, and which next, based upon limited resources. A literal line of questions waits for proper resources to become available.
There’s a Real Cost of Disconnected Data, and It’s Huge
For those looking to improve the way their business is run, timely access to actionable data is likely to be the competitive advantage they need. According to a 2015 Accenture / MIT study, the best-performing companies leverage multiple data sources to make better, more informed decisions, and to find better ROI for their efforts.
Conditions are now Right for Change
The need to evolve the way we work with data on a daily basis has been increasingly impressed upon business leaders, but what’s changed is that the conditions are finally right to spark a new move towards what’s been promised in the past.
Thanks to the proliferation of APIs, and to the growing acceptance of SaaS applications and the cloud as a platform for business, the market is ready for a new approach to solving the “data crisis.” These advances have primed the pump for change—now, we’re ready to bridge the divide between legacy systems and the more modern applications and services that are generating volumes of business data (that are being underleveraged). To serve the non-technical user and spur adoption throughout an organization, this new breed of technology behaves more like the consumerized web, cloud, and mobile applications. Service-oriented architectures have brought innovation to market much faster thanks to interchangeable technology components and easier time-to-analysis. Open, feature-rich application programming (API) interfaces and microservices that can be leveraged by a community of developers—both inside organizations and by outside consultants and ISVs—can enrich the platform far beyond the resources and imagination of the platform vendor.
The declining cost of cloud computing and storage is also a key enabler, making it possible to offer an organization access to all of its relevant data with one single cloud platform, and to take on the resource prediction problem with real-time, on-demand scaling. And smart phones are shifting expectations from internal users for a mobile-first experience—expectations legacy systems aren’t able to meet.
Where Do We Go from Here?
Bring Your Data Together
Embrace the way that businesses are managed by directly connecting to and bringing together all relevant data and social insights, which directly enable business decision makers to get timely insights they need to understand performance of their business, collaborate with their peers, and take action to improve results for their organization.
The right tech can’t be accessible by just one part of your company. Sure, your analyst might love it, but you also need tech that your C-suite wants to use, every day, to run their business. You need technology that connects with all your new data sources. We’re not living in a world where marketing, HR, and sales data need to live in different silos, behind different logins, and accessible only through manually-generated monthly reports.
Empower each person inside your organization with your most valuable asset. An effective solution will bring together a broad set of capabilities—at its foundation, extending from the ability to connect and ingest critical data sources—and allow for analysis, sharing, and collaboration on those merged data sets. You’ll want rich and interactive mobile capabilities to extend data into the field and real-time updates to increase confidence in the data. Getting data threaded through an organization isn’t a one-size-fits-all solution. It must be recognized that most companies have a distribution of needs and capabilities, and need to seek out and deploy a solution which is customizable and scalable.
…And Alter the Paradigm to Change How Business Is Managed
For generations, business has operated with a top-down hierarchical view, in a constant loop of reviewing reports. Success requires cultural change. Stop spending time reviewing data and out of date reports and start letting the data work for you. Leverage AI to drive alerts to know when key milestones are happening, and to generate intelligent recommendations on next steps. Don’t delay when there is clear impetus for action. Avoid the extra meetings and emails, and use the freed up time more productively.
For companies large and small, coping and thriving in a real-time, big data world requires change, because the traditional way of managing a business is no longer able to scale and keep pace. The industry is seeing plenty of investment, activity, and innovation, but it often doesn’t seem to be headed in the right direction to allow companies to find value and insights for business decision makers at all levels, from the C-suite to the front lines. The way we see and work with data on a daily basis must change and develop to match today’s conditions—it’s time to make waves towards what has been promised in the past.