The Future of Business Intelligence

Learn about the emergence and growth of business intelligence (BI) software technology. Discover new BI technical innovations and how BI can benefit your company.

Successful business intelligence secrets: BI front-end tools

The following is an excerpt from Successful Business Intelligence: Secrets to Making BI a Killer App, by Cindi...

Howson. It is reprinted here with permission from McGraw-Hill Professional; Copyright 2007 and The McGraw Hill Companies, Inc.

The future of business intelligence centers on making BI relevant for everyone, not only for information workers and internal employees, but also beyond corporate boundaries, to extend the reach of BI to customers and suppliers. As the Successful BI case studies have demonstrated, when best practices are applied, BI usage can explode beyond the paltry 25% of employees today to a much more prevalent business tool. It will take cultural shifts, new ways of thinking, and continued technical innovation. Business intelligence has the power to change people's way of working, to enable businesses to compete more effectively and efficiently, and to help nonprofits stretch their dollars further. All of this is possible based on insights available at the click of a mouse, push of a button, or touch of a screen.

As discussed throughout this book, much of the key to successful business intelligence has to do with the people, processes, and culture. Don't rely on technical innovation alone to solve the biggest barriers to BI success, but by all means, do get excited about the innovations that will make BI easier and more prevalent. BI as a technology has changed dramatically since its inception in the early 1990s. This chapter focuses on the most recent technical innovations with examples of how customers are taking advantage of them. In the final section, I leave you with some words of wisdom to inspire you to think about how business intelligence can become your company's next killer application.

Emerging Technologies

As part of the Successful BI Survey, respondents were asked to choose items from a list of emerging technologies that they believe will help their companies achieve greater success. Figure 14-1 shows which items are considered most important in helping companies achieve greater success. Web-based business intelligence and dashboards were rated the highest, with predictive analytics and alerting also at the top. Surprising to me, Microsoft Office Integration, BI Search, and Mobile BI were selected by only a small percentage of survey respondents.

The view according to business users, however, is slightly different, as shown in Figure 14-2. Business users account for only 10% of the survey respondents. Those who describe themselves as hybrid business-IT personnel account for 23% of respondents. I have specifically excluded IT personnel and hybrids from Figure 14-2, to show the gap in perceived importance of certain technologies. When viewing responses only for business users, the importance of Microsoft Office integration moves to the top of the list, while alerting moves down.

Some of these differences can be explained by gaps in understanding of the feature benefits, but also by a respondent's point of view. For example, IT professionals have been burned in the past by the thousands of disconnected spreadsheets and the ensuing data chaos. As Microsoft Office integration with BI has improved dramatically in 2007, IT professionals may not realize that spreadsheet-based analysis can now be "safely" enabled and can be something to be embraced for knowledge workers familiar not just with Microsoft Excel, but also with Word, Outlook, and PowerPoint. In a similar fashion, if you are a BlackBerry user, you may rate Mobile BI high. While web-based business intelligence may have been introduced in the late 1990s, these solutions only reached the rich functionality of their desktop predecessors in the 2005 time frame. A number of companies are not yet on the latest releases, though, and still use client/server BI deployments. Depending upon where a survey respondent is in their web-based BI deployment will influence how this capability was rated.

At The Data Warehousing Institute's (TDWI) Executive Summit in February 2007, I participated in a panel on the role of emerging technologies in extending the reach and impact of business intelligence. Attendance was restricted to BI directors and executive sponsors who influence their company's BI strategy. Attendees could vote on a limited number of items that they thought would have the biggest impact in the next few years. The most highly ranked item: performance management and predictive analytics. The things that got few to no votes were BI search, dashboards, and rich Internet applications, contrary to what I believe will have the biggest impact. As we delved into what these technologies mean, and in some cases, demonstrated them, the perceptions changed significantly. In this way, we sometimes don't know the impact any of these capabilities will have until the technology has become more mature and the industry understands it better. If you think about the way breakthroughs like the iPod and YouTube have revolutionized their markets, when they first were introduced, they were met with a mixture of fascination and confusion, without a clear understanding of where they would lead. Recent BI innovations must go through a similar process of the industry first understanding their potential, accepting or rejecting them, and then either embracing or adopting the innovations in a limited fashion. Another fundamental difference in evaluating impact is whether the impact is measured by the value of one user or decision or aggregated by multiple users and decisions. Recall from Chapter 12 (see Figure 12-4), thousands of individual decisions can have as big, if not bigger, an impact on a company than a single decision. There seems to be a natural tendency to rate capabilities that have a single big impact as being more important or more likely to help achieve greater BI success.

Figure 14-3 provides a framework for evaluating changes in BI technology to determine which new and emerging capabilities will prove most valuable to your company, how mature they are, and when to monitor them or when to embrace and actively deploy them (adapted from TDWI's Technology Evaluation Framework). The X axis provides an indication of how mature the technology is, and the Y axis gives an indication of which technology will make BI pervasive. Recall from the section on BI users as a percentage of employees in Chapter 4 that the average usage of BI within a company is currently at 25%, and even if budget were available and the deployment were wildly successful, survey respondents felt the use rate would extend only to 54% of employees.

The Y axis, then, indicates the degree to which an enabling technology will take BI's reach closer to 100% of employees. Business impact and BI prevalence are not linearly correlated, however. One enabling technology, such as predictive analytics, may yield a big value for a single decision, say, a $4 million savings by better marketing campaign management. Another enabling technology such as BI embedded in operational processes may affect thousands of users, each of whom makes dozens of decisions on a daily basis; the monetary value of these individual decisions may be small when measured in isolation, but enormous when taken in aggregate. The size and shading of the bubbles in Figure 14-3 give an indication of which items have a bigger single value. The bigger the bubble and darker the shading, the bigger the impact on a single decision or person. For each innovation, consider both the technical maturity and the business impact to decide how to proceed:

  • Embrace Items in the upper-right quadrant show innovations that are mature and that should be embraced as they will help speed user adoption across multiple user segments.
  • Adopt Where Appropriate Items in the lower-right quadrant show innovations that are mature but that may serve only specific segments of users. Mobile BI is an example of this; the technology is more mature than BI search, for example, but benefits only those users who have smartphones such as a BlackBerry.
  • Evaluate and Test Items in the upper-left quadrant are relatively new but will have a profound impact on user adoption. BI Search is a good example of this. The technology is very new and not well understood. A number of usability and performance issues still need to be worked out, but the potential impact on user adoption is enormous.
  • Monitor and Understand Items in the lower-left quadrant are so new that they may be riskier investments. Items here are less proven and have less market adoption.

Figure 14-3 portrays broad industry maturity of these capabilities and the degree to which most vendors offer the capabilities. For clarity, I have selected only certain innovations; it is not meant to be an exhaustive list of all things going on in the industry (for this, check the BIScorecard blog). All items are in the context of business intelligence as a technology. So while performance management is certainly a mature concept and technology, the integration of performance management with business intelligence is still a work in progress, leaving this item positioned slightly behind web-based BI and Microsoft Office integration on the maturity spectrum.

The subsequent sections describe these capabilities that have not otherwise been addressed in Chapters 2 and 3.

Predicting the Future

Data mining, statistical analysis, and predictive analytics are nothing new. These technologies are well established and are used in a number of different applications such as fraud detection, customer scoring, risk analysis, and campaign management. What's changed is how they have become integrated in the BI platform. Traditionally, predictive analytics has been a backroom task performed by a limited few statisticians who would take a snapshot of the data (either from a data warehouse or from a purpose-built extract from the source system), build a model, test a model, finalize it, and then somehow disseminate the results. While the expertise to build such models remains a unique skill set, the industry recognizes that the results of the analysis should be more broadly shared, not as a stand-alone application, but rather, as an integral part of the BI solution. This does not mean that predictive analytics software will become "mainstream," but rather that the results of such analyses can be readily incorporated into everyday reports and decision making. The analysis, then, is what needs to become mainstream.

Predictive analytic tools from different vendors do continue to differ significantly in how they work and in what information is stored in the database versus calculated and presented in a report or incorporated into an operational process.

At Corporate Express, for example, predictive analytics are being used to improve customers' online shopping experience.

Market basket analysis helps retailers understand which products sell together and provide product recommendations. In the past, Corporate Express provided these recommendations by logical product pairings. So if a customer ordered a stapler, the online store would recommend a staple remover as the marketing team had marked this as a complementary product.

In analyzing the data, though, it turned out that what was most often purchased with a stapler was not a staple remover, but rather a ruler, tape dispenser, and a wastepaper basket—items that indicate a purchase for a new employee. With the manually associated product recommendations, there was no significant impact on sales. Leveraging MicroStrategy and SPSS, Corporate Express tested a new market basket option. The model analyzes past shopping carts and produces recommendations to ensure the greatest lift. As a result, the average order size for market basket pairings doubled (versus those orders with no pairings), and the market basket application is expected to generate an incremental gross profit of more than $2 million in 2007.

Dow Chemical also has begun extending the reach of predictive analytics with SAS's JMP product (pronounced "jump"), a solution that combines visual analysis with statistics. As discussed in Chapter 12, Dow uses BusinessObjects and Cognos PowerPlay as enterprise reporting and analysis standards. Through these tools and the data in the data warehouse, Dow began looking at the high cost of railroad shipments: $400 million annually across North America.

A team of statistical experts studied the variables that most affected these costs and pulled data from the data warehouse and external data sources into SAS JMP. By benchmarking current payments versus industry norms, the analysis showed Dow was overpaying by 20%, or $80 million. In entering new contracts, the purchasing department now uses the software to predict appropriate rates, enabling them to negotiate more aggressively.

For both Corporate Express and Dow Chemical, the move to predictive analytics has been evolutionary. The underlying information architecture and a culture of fact-based decision making had to first reach a level of maturity and data quality before predictive analytics could be embraced. While both companies have been doing statistical analysis for decades, the degree to which predictive analytics has now been incorporated into daily processes (online store at Corporate Express and purchasing negotiations at Dow Chemical) reflects the degree to which predictive analytics has shifted from the backroom to the front line, with the most casual of users deriving value from such analytics.

BI Search & Text Analytics

BI Search offers a number of promising benefits to business intelligence:

  • Simple user interface.
  • A more complete set of information to support decision making, with the integration of structured (quantitative) and unstructured content (textual). Structured data refers to the numerical values typically captured in the operational systems and subsequently stored in a data warehouse. Unstructured content refers to information stored in textual comment fields, documents, annual reports, websites, and so on.
  • Users can find what they need through search, rather than through navigating a long list of reports.

Text analytics is closely related to search in that unstructured information or text can be transformed into quantitative data. For example, it allows for searching of information in a comment field to see how many times a customer praised a particular product. Text analytics is the numerical analysis of textual information.

Despite all the improvements in data warehousing and BI front-end tools, users continue to feel overwhelmed with reports yet undersatisfied with meaningful information. They don't know what's available or where. Similar reports are created over and over because users don't know which reports already exist or how, for example, the report "Product Sales" differs from "Product Sales YTD." Some of the most valuable information is hidden in textual data.

A BI Search interface promises to change the way users access information. Picture a Google interface to BI. Without any training in a BI tool, users can enter a phrase such as "Recent sales for customer A" and then be presented with either a list of predefined reports or, in some cases, a newly generated query. The added benefit is that in addition to displaying reports coming from the BI server, the search engine will also list textual information that may be relevant—a customer letter, sales call notes, headline news. When search capabilities are combined with text analytics, a report may include numerical data that scans the comment field to indicate number of complaints with number of positive comments. Never before has such unstructured data been so nicely accessible with structured or quantitative data.

If the integration of search and BI is successful, it is yet another innovation that will make BI accessible and usable by every employee in an organization. According to Tony Byrne, founder/president of CMS Watch, a technology evaluation firm focusing on enterprise search and content management systems, search as a technology has existed for more than 50 years.4 Consumer search (Google and Yahoo, for example) as a technology emerged with the Internet in the mid-1990s. In many respects, the success of consumer search has helped spur hype around enterprise search, in which companies deploy search technology internally to search myriad document repositories. Text analytics has existed for 25 years but with usage in limited sectors, particularly, the government. The convergence of search with business intelligence first emerged in 2006. Google is not the only enterprise search solution that BI vendors support but it is one that has the most consumer recognition and thus has helped business users to understand the possibilities. To illustrate the point, note that BI search was selected by only 27% of the Successful BI Survey respondents as a capability that would help foster greater success (see earlier Figure 14-1). Yet in discussing these technologies with individual executives who don't currently use business intelligence, a Google-like interface to BI generated the most enthusiasm.

The incorporation of text analytics with traditional business intelligence is still in its infancy. I place both BI search and text analytics close to the Monitor and Understand quadrant but in the Evaluate and Test quadrant in Figure 14-3. Again, both technologies independent of BI have existed for decades; it is that convergence with BI that is new. While the convergence is still relatively immature, the promise it brings for BI to reach more users and in the value of incorporating textual data is enormous.

The number of customers taking advantage of the BI Search and text analytics integration is only a handful. BlueCross BlueShield (BCBS) of Tennessee (TN) is an early adopter of these capabilities.5 BCBS of TN is a not-for-profit provider of health insurance. In 2006, it paid $17 billion in benefits for its 2 million commercial members.6 Managing claims and negotiating rates with providers is critical in ensuring BCBS can meet its obligations to the members it insures. While the insurer has had a mature business intelligence deployment for ten years,7 Frank Brooks, the senior manager of data resource management and chief data architect, recognized that there was value in bringing the text data stored in comment fields from call center notes together with information in the data warehouse.8 Given how new the technology is, Brooks asked their BI vendor, Cognos, along with IBM (who produces the search solution OmniFind) and SAS (who offers text analytics solution Text Miner) to work together to develop several prototypes and show the business users the concept of bringing BI, enterprise search, and text analytics together. With this capability, a business user can enter the key word "diabetes" in the OmniFind search box and be presented with a ranked list of things such as:

  • Cognos reports and OLAP cubes that show claims paid for diabetic treatments
  • Call center notes that involve diabetes
  • New research on improving care for diabetes patients

The business was enthusiastic. There has been a high degree of collaboration between BCBS of TN and its information technology partners to understand the new capabilities, develop the right infrastructure, and optimize the indexes to provide the best search performance.

Consistent with the evaluation framework in Figure 14-3, understanding new technologies requires a significant amount of evaluation and testing. BCBS of TN evaluated the capabilities for more than a year before developing plans for implementing in production.

Advanced Visualization

All leading BI tools have basic visualization capabilities: you can take tabular data and turn it into a bar chart, trend line, and so on. They also support some kind of conditional formatting of data: display positive numbers in green, display negative numbers in red, and enlarge those with the worst variances. Advanced visualization goes beyond a simple chart such as a bar or line chart to include things as:

  • Spark lines, a highly condensed trend line the size of a word.
  • Bullet graphs, a construct by Stephen Few that includes a target indicator within the bar chart. See Figure 3-3, Chapter 3.
  • Small multiples, which are series of small, similar graphics or charts.9 As they use the same scale and are positioned side-by-side, they facilitate visual discovery by letting users make comparisons at a glance.
  • Heat maps that display two variables as different intensifying colors.
  • Decomposition trees, a visualization that displays each drill-down akin to an ever-expanding organization chart.
  • Geographic maps that display things such as sales figures in a map form, using color to highlight sales performance. By mousing over a particular country, region, or state, you can see the individual data values.

Advanced visualization software and capabilities also help you apply best practices in data visualization, even for basic visualizations. As an example, many reports today are designed as a dense page of numbers. The dense page of numbers may not help facilitate insight, but they are what users are accustomed to. With visualization software, a dense set of numbers can quickly be converted to a more meaningful display. Figure 14-4 shows several charts created in Tableau Software. By displaying multiple graphs side-by-side, as "small multiples," you can more easily see which product category—technology in this case—is the top-selling product category for all regions. The dimensions (time and region) and scale within each graph are the same, allowing for a rapid comparison. Sales seem to have dipped in 2003, particularly in the eastern region. By toggling the quick filters (shown on the right in Figure 14-4), it's possible to focus on the individual customer segments to see that technology sales to corporations are on a steady decline, whereas consumer and small business segments show strength.

Creating this kind of display with standard BI software is theoretically possible, but one that would take many, many more steps. As well, if I am uncertain as to the best way to display the information, advanced visualization software can make suggestions. The capability to create easily such advanced visualizations is generally not available in BI suites. Users must rely on specialty products. In a theme similar to predictive analytics and search, visualization software has existed for years; the change is in its convergence with business intelligence such that advanced visualizations are appearing in dashboards and reports. In this regard, the emphasis for BI tools is changing from a focus of simply "getting to the data" to "what insights can I discover from the data and how can the most information be displayed in the smallest space."

Rich Reportlets

Report-based interactivity is a ho-hum term that warrants a better name. "Active reports," "on-report formatting," and "navigable reports" are similar terms that also don't fully capture the value of this capability. I suspect poor terminology and lack of awareness also explains why survey respondents ranked this capability on the low end of importance for emerging technologies (see Figure 14-1). So after much thought and brainstorming with some colleagues, I will refer to this capability as "rich reportlets." The difference in power and appeal with rich reportlets versus, say, green-bar paper reports and much of what is currently deployed over the Web, is comparable to the difference between a black Ford Model T and a red Mercedes sports coupe.

Rich reportlets are powered by Web 2.0 technologies to create rich Internet applications (RIA). When BI suites were first re-architected for the Web, report consumers could only view a static page. Given how static a display this was, more sophisticated users would export the data to Excel for analysis. Less sophisticated users would submit requests to IT or to the BI team to modify the report design. The Web in this case is only a delivery vehicle for data; it does not facilitate user adoption and insight. With rich reportlets, someone accesses a report over the Web but in a much more interactive and appealing way. At a simple click, data can be re-sorted, filtered, or graphed, without having to launch a complicated report editor. With the use of either Adobe Flex or Macromedia Flash, these reports come to life in ways that make business intelligence fun. I have seen, for example, a bubble chart that displays bubbles dancing across the screen as the time axis marches onward. Such animation makes BI appealing as well as insightful as users see the trend in action. In this regard, the term "report" doesn't do justice to the capability that is more akin to a mini application.

This type of interactivity affects all BI users, whether casual or power users. The appeal makes BI more engaging, and while some technologists may scoff at the importance of this, when other barriers to adoption exist, appeal matters. A lot! The ability to interact with the data in a simple and intuitive way facilitates greater insight at the hands of the decision maker. The report consumer is not forced to delay this insight until a power user can modify the report. Lastly, the cost of ownership is lowered because a single reportlet can be "tweaked" to that decision maker's needs, without IT having to maintain thousands of individualized reports.

The Future Beyond Technology

Technical innovation is only one aspect that will help increase BI's prevalence. In discussing future plans with many of the case study companies, much of their concern was not about technology, but rather, in finding new ways to use BI to address common business problems. For the more large-scale deployments, some expressed concern about managing the risk of making any kind of major change to such a business critical, complex application. With success, of course, comes greater demands on the systems and the people. Ensuring an effective way of prioritizing competing requests warrants constant attention. One business leader expressed frustration at his department's inability to make wise investments, while witnessing other departments, working in more unison and getting more value from business intelligence. Yet he remains optimistic that his business will get there and that BI will be the first thing people look at, even before email. "To have one screen I can get to with a single click, that shows sales, margin, price, opportunities in graphical form, with drill down—that would be magic!" His comments remind me that the technology is sometimes the easy part; getting the organization aligned is harder. Even the most successful BI companies, then, continue to have their battles.

Words of Wisdom

I hope this book will inspire you to ensure business intelligence has a profound impact on your organization. I'd like to see the techniques and insights shared by the most successful BI companies and the innovations from leading and niche vendors help move the industry beyond the current average of 25% usage rate within companies and 24% success rate to much higher rates. Business intelligence is all encompassing in its ability to improve an organization's efficiency, competitiveness, and opportunities. Through the process of writing and researching this book, these business intelligence visionaries have assured me that this way of thinking is not just analyst-speak or vendor hype; it can be a reality. Following are some words of wisdom that I hope will inspire and guide you as you strive to make business intelligence a wild success in your company.

"Slicing and dicing the data has to be easier—easier than picking up a piece of paper. Even then, there is an education that needs to happen—getting people to think business intelligence."
--Dr. Ray Iannaconne, vice president of operations, Emergency Medical Associates

"Senior management needs to take a leap of faith. It may take years, lots of time and money before you get value, but you have to ultimately know where you are going with this."
--Mike Gorman, senior director of customer relations, Continental Airlines

"The data is compelling. The business model is a challenge. We haven't proven the market wants this system. We get the feeling they do, but how to package it will be a challenge."
--Jeff Kennedy, CEO, FlightStats

"Implementing an integrated reporting system for the company was a significant undertaking, but the result, Dow's Shared Data Network, is now a fundamental building block to managing Dow on a global basis. It's a very significant capability to manage our strategy."
--Dave Kepler, CIO, The Dow Chemical Company

"Strong management and a cultural change have most contributed to our success. The CEO got Norway Post to be more business oriented, and the CFO drove the management system. In adopting this cultural change to one of accountability, sometimes we had to change the people."
--Dag Vidar Olsen, Norway Post, Manager Business Intelligence Competency Center

"I need to continue to push for BI to have a greater influence in the organization in order to maximize the returns on our BI investment. I see BI providing new insights into the business and serving as an enabler to profitable growth in the future."
--Robert VanHees, CFO, Corporate Express

"Our business intelligence initiative has been a terrific success in the way we can optimize our team and the greater sense of control of the business. We are always tweaking things. Before, too many decisions were based on assumptions, generality, anecdotal, off the gut. It's made us more agile as a company."
--Dave Walker, vice president of operations, 1-800 CONTACTS

Read other excerpts and download more sample chapters from our manufacturing ERP bookshelf

Dig Deeper on ERP software selection and implementation