A Position Paper

6 July 1997

Submitted to "Workshop on the Keys to the Commercial Success of Data Mining"

Held in Conjunction with the "Fourth International Conference on Knowledge Discovery and Data Mining"

Scott Cunningham

Srikant Sreedhar

Bill Smart

Knowledge Discovery Group

Human Interface Technology Center

NCR Corporation

Tej Anand

Golden Books, Inc.

Introduction

Algorithms for finding rules or affinities between items in a database are well known and well documented in the knowledge discovery community. A prototypical application of such affinity algorithms is in "market basket" analysis – the application of affinity rules to analyzing consumer purchases. Such analyses are of particular importance to the retail and consumer packaged goods industry. The manufacturers, distributors, retailers and wholesalers in this industry generated over 300 billion dollars of sales every year in the United States alone. Due to the economic importance of this industry, data mining solutions to key business problems are likely to have a huge impact . This position paper discusses some of the problems of the consumer packaged goods industry, notes a case study of some of the challenges presented to data miners within this industry, and critiques current knowledge discovery research in these areas.

Business Problem

The consumer packaged goods industry exists within a complex economic and informational environment. Mass merchandizing of products is in decline; U.S. consumers are increasing recognized as belonging to fifty (or more) distinct segments, each with its own demographic profile, buying power, product preferences and media access. Consumer products, are more diverse than ever before; a single category of food may easily contain hundreds of competing products. Within this highly differentiated environment, strong product brand names continue to offer a strong competitive advantage. By themselves temporary price reductions are not sufficient for establishing consumer loyalty to either a store or product. Consumers are knowledgeable, and mobile, enough to seek out the lowest possible prices for a product. Ultimately consumer value is gained by those retailers able to negotiate favorable terms with their suppliers. Retailers gain the requisite detailed knowledge of customers through the creation of consumer loyalty programs and the use of on-line transaction processing systems; this information about the consumer is a crucial component in retailer-supplier negotations.

Consumer packaged goods is a mature industry in the United States. Profit is no longer merely a matter of opening more stores, and selling to increasing numbers of consumers; the market is becoming saturated, and the available consumer disposable income largely consumed.

Maintenance of an existing customer base is more important than growing entirely new customers; this new phase of retail growth is based upon selling more and a greater variety of products to pre-existing consumers. The most profitable retailers are those that are able to maintain or reduce their operating costs. Economics of scope, not scale, determine profitability.

Data warehousing is one of the foremost technological means of increasing operational efficiency. Efficient consumer response systems, based upon data warehouses, are expected to save the industry $30 billion a year. Category management, an organizational strategy for enhancing retailer-wholesaler coordination, is another means of increasing operational efficiency. In the following two brief case studies we examine how data warehouses, category management, and data mining techniques show promise for answering the concerns of two large consumer package goods companies.
 
 

Case Studies

A major international food manufacturer, with significant brand equity and a wide variety of manufactured products, is interested in optimizing its product advertising budget. Like many package goods manufacturers, this manufacturer has an extensive and rapidly growing advertising budget. Essential to the endeavor is the cooperation of their independent retail outlets in the creation and design of product promotions. The manufacturer sought to create a suite of software tools for the design of promotions, utilizing the newest data mining technology, and to make these tools available in real time to their category managers and to the managers of their retail outlets. The business case suggested that there would be at least three sources of return in the creation of this tool: Improved coordination with retailers; more effective cross-sales across product categories; reduced promotional competition from other manufacturers; and enhanced promotional returns. NCR proposed and designed a state-of-the-art neural network for forecasting and optimizing planned promotions. The network met, or exceeded industry standards, for promotional forecasts (within 15% of actual sales, 85% of the time). Despite the statistical quality of the results the application was never put into production by the manufacturer; the software design necessary to implement the results was too complex. Part of the application complexity stemmed from the hierarchical data types necessitated by the varied products and markets; another component of the complexity was reconciling the different product world-views of manufacturer and retailer.

A major regional food retailer, a grocer, sought analyses of its consumer transactions within its produce and salad dressing departments. The retailer anticipated improved design of store layouts, improved promotional design, and an insight into the market role of the various highly differentiated products within the category. The retailer clearly anticipated a causal analysis which would reveal the products which, when purchased by consumers, would lead to additional add-on sales of other products. NCR produced a market basket analysis which revealed the distinctive purchasing profiles that are associated with each major brand of interest. The NCR analysis revealed that the best selling brands were not those that resulted in the greatest amount of attendant sales. The NCR analysis supported the existing category management plans by the retailer, and also independently confirmed the results of a demographic panel survey. Despite these successes the market basket analysis, by itself, did not produce any new actionable results for the retailer. In the next section, on data mining, key data mining algorithms and outputs are examined for their suitability for answering these, and other, consumer package goods questions.
 
 

Data Mining Solutions

Affinity algorithms are well-understood and well-documented by the data mining community. The quintessential application of affinity algorithms is in the area of market basket analysis. For instance, these algorithms when applied to market basket analysis produce rules such as "Those baskets producing product X are also 75% likely to contain product Y." Additional research has focused on optimizing the speed and efficiency with which these rules are found; however additional applied research is needed to the support decision making needs of the consumer goods industry (and other relevant business groups).

First, affinity algorithms produce individual, isolated rules; associations between groups of products are not revealed. While the analysis can be repeated across all products in a category, or even a store, the number of rules produced grows exponentially. Not only is this computationally complex, but the resulting welter of rules is hard to interpret as well. Second, the output of affinity algorithms seem to suggest causal relationships between products. Yet the algorithms themselves embody no causal assumptions. The nature of product affinities needs to reconsidered; either a new and causal form of affinities analysis needs to be produced, or a thorough understanding of non-causal applications and use of affinity rules needs to be obtained. Third, affinity algorithms lack robustness. The algorithms produce a point estimate of affinity; yet retailers need to understand how (and if) these rules apply across larger groups of transaction. A similar issue is the minimum sample size needed to produce robust results. Fourth, market basket analyses carry implicit information about consumer preferences. Even when consumer identification is missing from transaction data, the data can still be grouped or segmented using data mining techniques to reveal distinct groups of consumer preferences. Affinity algorithms imply that samples are taken from homogenous groups of customers; yet business knowledge suggests that consumers are highly varied in taste and expenditure. Fifth, the market basket analyses, for some set of business questions, may require the rigor of a properly designed statistical experiment. Reasoning from standard to promotional pricing, as well as reasoning from standard display conditions to promotional display conditions is unwarranted. Yet much of the potential of market basket analysis stems from the capacity of retailers to manipulate product pricing, display or even attributes to meet consumer need. Sixth, and finally, standard forecasting tools produce estimates of sales for single products across times. (This is not conventionally the domain of market basket or affinities analysis.) However retailers and manufacturers need to have forecasts for whole groups of products. Producing individual product forecasts, and then aggregating, will not produce optimum forecasts since sales of one product contains information about the potential sales of other products; indeed, the forecasts may not even aggregate correctly. Techniques such as "state space analysis" which combine forecasting with multivariate analysis, may prove useful.
 
 

Recommendations

The consumer package goods industry is an important, and expansive, industrial segment of the economy. This industry is dependent upon information for its continued economic growth. It is therefore making great progress in collecting large databases of relevant data about its industry. The corresponding questions the industry has about its data are both interesting, and economically fruitful. This paper considered two case studies of applying standard data mining techniques to industrial questions in the area of consumer package goods. The examples discussed a wholesaler and a retailer that sought better management of product categories, and a resulting improved economy of scope. Commercial success of data mining will in part, be dependent upon the capacity of algorithms to model complex, hierarchical arrangements of goods and products.