Optimization Analytics Comes to the Mass Market

Optimization is the application of algorithms to sets of data to guide executives and managers in making the best decisions. It’s a trending topic because using optimization technologies and techniques to better manage a variety of day-to-day business issues is becoming easier. I expect optimization, once the preserve of data scientists and operations research specialists will become mainstream in general purpose business analytics over the next five years.

Optimization was first adopted by businesses in the middle of the 20th century, aided by the introduction of digital computers. The first technique that gained broad for a few specific purposes was linear programming, one of the most basic optimization methods. Linear programming enables analysts to quickly determine how to achieve the best outcome (such as maximum unit volume or lowest cost) in a given situation. They do so using a mathematical model that captures the key variables that go into the decision and any constraints that may affect that decision. A food processor, for instance, may use three types of cooking oil to make a product. To maximize its profit, the company needs to determine the exact proportions of the three oils that result in the lowest production cost. However, it can’t just choose the cheapest of the three in every case because for flavor and shelf-life requirements there’s a limit to the maximum percentage of each oil that it can use. Linear programming using the simplex algorithm quickly solves the problem.

As computing capabilities became increasingly affordable, companies could use more complex algorithms to handle ever more difficult optimization problems. For instance, the airline industry used it to determine how best to route aircraft between two cities and to staff flight crews. Not only can software find the best solution for scheduling these assets in advance, it also can rapidly re-optimize the solution when weather or mechanical issues force a change in how aircraft and crews are deployed. Airlines were also in the vanguard in the 1980s when they started using revenue management techniques. In this case, the optimization process was designed to enable established airlines to compete against low-cost startups. Revenue management enabled the large carriers to offer low fares to price-sensitive but flexible vacationers without sacrificing the higher fares that the less flexible businesspeople were willing to spend. The same approach was adopted by hotels in pricing their rooms. Starting in the 1990s markdown management software, which I have written about gained ground. It enables retailers to make more intelligent pricing decisions by monitoring the velocity of purchases of specific items and adjusting prices to maximize revenue. To be feasible, each of these optimization problems require large data sets and sufficient raw computing power.

We’re now on the cusp of “democratizing” what I call optimization analytics. Big data technologies are making it feasible and affordable for even midsize companies to work with much larger data sets than they have been able to in the past. Our benchmark research on big data analytics finds that about half of participating companies already use analytics with big data. This is partly the result of more powerful and affordable data processing resources but also because companies have invested in systems to automate many functions. The rich data sets created by these business applications provide corporations with the raw material for analysis. This data has the potential to enable businesses to make more intelligent decisions. From a practical standpoint, though, the value of these large data sets can only be realized by moving optimization analytics out of the exclusive realm of data scientists and into the hands of business analysts. These analysts are the ones who have a sufficient understanding of the business and the subtleties of the data to find useful and repeatable optimization opportunities. Three-fourths of companies in our research said that they need these business skills (“domain expertise”) to use big data analytics successfully.

Optimization analytics is a breakthrough technology with the potential to improve business performance and create a competitive advantage. You can’t do optimization in your head, and it’s not feasible in desktop spreadsheets for anything but the most basic use cases, such as linear programming optimizations on relatively small data sets. This is a good reason for almost any company to consider adopting optimization software.

Another reason why companies will find it attractive to apply optimization analytics broadly is that the results of applying optimization routines may be superior to using common rules of thumb or relying on instinct and experience. One of the most important lessons for executives about optimization analytics is that optimal solutions are sometimes (but – crucially – not always) counterintuitive to established norms. For instance, in markdown management, retailers often have found that smaller, more frequent price reductions maximize profits and produce a considerable improvement in sales over the end-of-the-season price slashing that was once considered to be the best practice. In financial services, charging your best customers more for loans and other services turns out to be the optimal choice for the bottom line of financial institutions. Another important insight from our collective experience with optimization is that while the value of these analytics as realized in a single event or transaction may be small, it can have a measurable impact on profitability and competitiveness when applied broadly in a business.

At this point optimization analytics is in a dual mode. On the one hand, there are proven examples of the narrow application of optimization such as those mentioned above. On the other, bringing optimization analytics to the masses is only beginning. Some vendors have made progress in simplifying their analytics, but mainstream products are only on the horizon. It’s also important to recognize that, as with past breakthroughs in information technology, there are bound to be more duds than success stories in initial attempts at using optimization analytics. Experience suggests that a small number of companies that have strong analytical skills and a rigorous approach to managing company data will prove to be the leaders in finding profitable opportunities for applying optimization technologies and techniques. Others will do well to find these examples and consider how to apply them to their own organizations.