The technology described in this patent document relates generally to time series data mining, and more particularly to clustering time series based on forecast distributions.
Clustering time series is often an important task performed in time series data mining applications. Typically, a historical data set is used, either directly or indirectly, as the source of clustering operations. That is, clustering techniques typically focus on the similarity patterns of time series data that has already been observed. It may be desirable, however, to provide time series clustering for a specific time or period of time in the future based on forecast data, instead of clustering based on observed data over time.
In accordance with the teachings described herein, systems and methods are provided for clustering time series based on forecast distributions. A system may include one or more processors and one or more memory devices, the one or more memory devices including a database for storing time series data relating to one or more aspects of a physical process. A forecasting program may be stored in the one or more memory devices and executable by the one or more processors, when executed the forecasting program being configured to apply a forecasting model to the time series data to generate forecasted values and confidence intervals associated with the forecasted values, the confidence intervals being generated based on distribution information relating to the forecasted values. A divergence program may be stored in the one or more memory devices and executable by the one or more processors, when executed the divergence program being configured to generate a distance matrix that identifies divergence in the forecasted values, the distance matrix being generated based the distribution information relating to the forecasted values. The divergence program may generate the distance matrix using a symmetric Kullback-Leibler divergence algorithm. A clustering program may be stored in the one or more memory devices and executable by the one or more processors, when executed the clustering program being configured to perform a clustering operation on the plurality of forecasted values based on the distance matrix.
A method for clustering time series based on forecast distributions may include: receiving time series data relating to one or more aspects of a physical process; applying a forecasting model to the time series data to generate forecasted values and confidence intervals associated with the forecasted values, the confidence intervals being generated based on distribution information relating to the forecasted values; generating a distance matrix that identifies divergence in the forecasted values, the distance matrix being generated based the distribution information relating to the forecasted values; and performing a clustering operation on the plurality of forecasted values based on the distance matrix. The distance matrix may be generated using a symmetric Kullback-Leibler divergence algorithm.
It should be understood that the system blocks 102, 104, 106 illustrated in
The forecasting block 102 may be a software application that calculates forecasts and their associated confidence intervals 110 based on received time series data 108 utilizing a forecasting model, such as a moving average (weighted) model, an exponential smoothing model, an autoregressive model, an autoregressive integrated moving average model (ARIMA), or some other suitable type of forecasting model. The forecasting block 102 may, for example, be implemented using the SAS® Enterprise Miner™ software sold by SAS Institute Inc. of Cary, N.C. The forecasts 114 generated by the forecasting block 102 may be considered predictions over time, and the uncertainty of each prediction may be expressed by a confidence interval calculated through a distribution assumption in the model error term. A Gaussian error assumption is commonly used for the forecast confidence interval. Thus, when a forecasting application, such as SAS® Enterprise Miner™, produces forecast values 114, it also provides the confidence intervals of forecasts under the assumption of a Gaussian white noise process. That is, the confidence intervals of forecast values are calculated by the forecasting block 102 from the normal distributions of forecasts with their estimated means and variances.
The symmetric Kullback-Leibler divergence block 106 utilizes the forecast distributions 116 used by the forecasting block 102 in the calculation of forecast confidence intervals to perform a Kullback-Leibler divergence calculation. However, because the Kullback-Leibler divergence is not symmetric, and is thus not a metric, block 106 uses a symmetric version of Kullback-Leibler divergence (as described below) to calculate the dissimilarity matrix among forecast densities. The Kullback-Leibler divergence between two forecast densities is obtainable using the same distribution information used by the forecasting block 102 in generating the confidence intervals. The resulting Kullback-Leibler distance matrix 118 among all pairs of the forecasted values 114 may then be used as a forecast clustering input source to the clustering block 104. In this way, significant processing time and resources may be saved by utilizing the same distribution information to generate the distance matrix 118 for clustering as is used to generate the confidence intervals during the forecasting process.
The clustering block 104 may be implemented using a hierarchical clustering algorithm, such as single-link clustering, average-link clustering or complete-link clustering. It should be understood, however, that in other examples any distance-based clustering algorithm may be utilized. The clustering block 104 may, for example, be implemented using a software application that enables the user to select a particular clustering algorithm and/or to specify one or more parameters of the clustering operation. For instance, the clustering block 104 may be implemented using the SAS®/STAT or SAS® Enterprise Miner™ software sold by SAS Institute Inc. of Cary, N.C. In another example, the cluster block 104 may be implemented using a software module that is included in the same software application as the forecasting block 102 and/or the symmetric Kullback-Leibler divergence block 106. The output 120 generated by the cluster block 104 may, for example, include report information that illustrates the results of the hierarchical clustering operation.
To illustrate how the Kullback-Leibler distance matrix 118 may be generated by block 106, suppose that there are two normal densities, ρ0:N(μ0, σ02) and ρ1:N(μ1, σ12), then Kullback-Leibler divergence between two densities is defined and simplified as follows:
Since the Kullback-Leibler divergence is not symmetric, a symmetric version for use in generating the distance matrix 118 may be defined as:
As explained above, the forecasting confidence intervals generated by the forecasting block 102 are calculated based on normal distribution assumption in error. Therefore, the k-step ahead forecast, Ŷt(k), is assumed to follow N(Ŷt(k),V(Ŷt(k)). For example, if the forecasting block 102 utilizes a simple exponential smoothing model, Yt=μt+εt, then the smoothing equation is: St=αYt+(1−α)St-1, and the k-step ahead forecast of Yt is St, i.e., Ŷt(k)=St.
The simple exponential smoothing model uses an exponentially weighted moving average of past values. The model is equivalent to an ARIMA(0,1,1) model without a constant. Therefore, the model is: (1−B)Yt=(1−θB)εt, where θ=1−α, then Yt=εt+Σj=1∞αεt-j. Thus, the variance of Ŷt(k) is:
V(Ŷt(k))=V(εt[1+Σj=1k-1α2]=V(εt)[1+(k−1)α2].
Under the assumption that Ŷt(k) follows N(Ŷt(k),V (Ŷt(k)), the Kullback-Leibler divergence of two forecast densities at a future point may be obtained using equations (1) and (2), above.
As an example, consider retail sales data for two stores. In the example shown in
The input 212 to the symmetric Kullback-Leibler divergence block 106 may be used to define a specific future time point for calculating the K-L distance matrix 118 or to define a period of time in the future (i.e., a lead time period) over which to calculate the K-L distance matrix 118. An example illustrating how the system of
An example output 600 that may be generated by the clustering block is illustrated in
An example of forecasting lead 1100 and clustering lead point 1200 parameter values is illustrated in
At 1416, a decision is made as to whether the forecast clustering is to be performed at a specific future time point (i.e., a lead point) or over a future time period (i.e., a lead time period). This decision may, for example, be based on one or more parameter settings, such as a forecasting lead parameter, as described above. If the forecast clustering is based on a specific lead point, then the method proceeds to 1418. Otherwise, if the forecast clustering is based on a lead time period, then the method proceeds to 1422.
If clustering at a specific time point, the specific lead point is specified at 1418. The specific future time point may, for example, be specified by one or more parameter settings, such as a clustering lead point parameter, as described above. Then, at 1420 a K-L distance matrix is calculated using the forecast densities at the specified lead point. Otherwise, if clustering over a future time period, the K-L distance matrix is calculated at 1422 using a summation, average, or some other linear combination of forecasts over the lead time period. In either case, the K-L distance matrix may be calculated using a symmetric Kullback-Leibler divergence algorithm that utilizes the same distribution information that is used during the forecasting process to generate the confidence interval.
At 1424, the K-L distance matrix from 1420 or 1422 is provided to a clustering algorithm to generate a clustered output at 1426. The clustering operation at 1424 may be implemented using a hierarchical clustering algorithm, such as single-link clustering, average-link clustering or complete-link clustering, or in other examples may be implemented using another distance-based clustering algorithm.
It should be understood that similar to the other processing flows described herein, one or more of the steps and the order in the flowchart may be altered, deleted, modified and/or augmented and still achieve the desired outcome.
A disk controller 2060 interfaces one or more optional disk drives to the system bus 2052. These disk drives may be external or internal floppy disk drives such as 2062, external or internal CD-ROM, CD-R, CD-RW or DVD drives such as 2064, or external or internal hard drives 2066. As indicated previously, these various disk drives and disk controllers are optional devices.
Each of the element managers, real-time data buffer, conveyors, file input processor, database index shared access memory loader, reference data buffer and data managers may include a software application stored in one or more of the disk drives connected to the disk controller 2060, the ROM 2056 and/or the RAM 2058. Preferably, the processor 2054 may access each component as required.
A display interface 2068 may permit information from the bus 2052 to be displayed on a display 2070 in audio, graphic, or alphanumeric format. Communication with external devices may optionally occur using various communication ports 2072.
In addition to the standard computer-type components, the hardware may also include data input devices, such as a keyboard 2073, or other input device 2074, such as a microphone, remote control, pointer, mouse and/or joystick.
This written description uses examples to disclose the invention, including the best mode, and also to enable a person skilled in the art to make and use the invention. The patentable scope of the invention may include other examples. Additionally, the methods and systems described herein may be implemented on many different types of processing devices by program code comprising program instructions that are executable by the device processing subsystem. The software program instructions may include source code, object code, machine code, or any other stored data that is operable to cause a processing system to perform the methods and operations described herein. Other implementations may also be used, however, such as firmware or even appropriately designed hardware configured to carry out the methods and systems described herein.
The systems' and methods' data (e.g., associations, mappings, data input, data output, intermediate data results, final data results, etc.) may be stored and implemented in one or more different types of computer-implemented data stores, such as different types of storage devices and programming constructs (e.g., RAM, ROM, Flash memory, flat files, databases, programming data structures, programming variables, IF-THEN (or similar type) statement constructs, etc.). It is noted that data structures describe formats for use in organizing and storing data in databases, programs, memory, or other computer-readable media for use by a computer program.
The computer components, software modules, functions, data stores and data structures described herein may be connected directly or indirectly to each other in order to allow the flow of data needed for their operations. It is also noted that a module or processor includes but is not limited to a unit of code that performs a software operation, and can be implemented for example as a subroutine unit of code, or as a software function unit of code, or as an object (as in an object-oriented paradigm), or as an applet, or in a computer script language, or as another type of computer code. The software components and/or functionality may be located on a single computer or distributed across multiple computers depending upon the situation at hand.
It should be understood that as used in the description herein and throughout the claims that follow, the meaning of “a,” “an,” and “the” includes plural reference unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise. Finally, as used in the description herein and throughout the claims that follow, the meanings of “and” and “or” include both the conjunctive and disjunctive and may be used interchangeably unless the context expressly dictates otherwise; the phrase “exclusive or” may be used to indicate situation where only the disjunctive meaning may apply.
Number | Name | Date | Kind |
---|---|---|---|
5991740 | Messer | Nov 1999 | A |
5995943 | Bull et al. | Nov 1999 | A |
6052481 | Grajski et al. | Apr 2000 | A |
6128624 | Papierniak et al. | Oct 2000 | A |
6151584 | Papierniak et al. | Nov 2000 | A |
6169534 | Raffel et al. | Jan 2001 | B1 |
6189029 | Fuerst | Feb 2001 | B1 |
6208975 | Bull et al. | Mar 2001 | B1 |
6216129 | Eldering | Apr 2001 | B1 |
6286005 | Cannon | Sep 2001 | B1 |
6308162 | Ouimet et al. | Oct 2001 | B1 |
6317731 | Luciano | Nov 2001 | B1 |
6334110 | Walter et al. | Dec 2001 | B1 |
6397166 | Leung et al. | May 2002 | B1 |
6400853 | Shiiyama | Jun 2002 | B1 |
6526405 | Mannila et al. | Feb 2003 | B1 |
6539392 | Rebane | Mar 2003 | B1 |
6542869 | Foote | Apr 2003 | B1 |
6564190 | Dubner | May 2003 | B1 |
6591255 | Tatum et al. | Jul 2003 | B1 |
6640227 | Andreev | Oct 2003 | B1 |
6735738 | Kojima | May 2004 | B1 |
6775646 | Tufillaro et al. | Aug 2004 | B1 |
6792399 | Phillips et al. | Sep 2004 | B1 |
6826575 | Waclawski | Nov 2004 | B1 |
6850871 | Barford et al. | Feb 2005 | B1 |
6850937 | Hisamitsu et al. | Feb 2005 | B1 |
6878891 | Josten et al. | Apr 2005 | B1 |
6928398 | Fang et al. | Aug 2005 | B1 |
7072863 | Phillips et al. | Jul 2006 | B1 |
7103222 | Peker | Sep 2006 | B2 |
7171340 | Brocklebank | Jan 2007 | B2 |
7216088 | Chappel et al. | May 2007 | B1 |
7236940 | Chappel | Jun 2007 | B2 |
7251589 | Crowe et al. | Jul 2007 | B1 |
7260550 | Notani | Aug 2007 | B1 |
7480659 | Chmura et al. | Jan 2009 | B2 |
7610214 | Dwarakanath et al. | Oct 2009 | B1 |
7711734 | Leonard | May 2010 | B2 |
8245135 | Cai et al. | Aug 2012 | B2 |
20020107853 | Hofmann et al. | Aug 2002 | A1 |
20020169657 | Singh et al. | Nov 2002 | A1 |
20030105660 | Walsh et al. | Jun 2003 | A1 |
20030110016 | Stefek et al. | Jun 2003 | A1 |
20030187719 | Brocklebank | Oct 2003 | A1 |
20030200134 | Leonard et al. | Oct 2003 | A1 |
20040086185 | Sun | May 2004 | A1 |
20040172225 | Hochberg et al. | Sep 2004 | A1 |
20050102107 | Porikli | May 2005 | A1 |
20050249412 | Radhakrishnan et al. | Nov 2005 | A1 |
20060063156 | Willman et al. | Mar 2006 | A1 |
20060064181 | Kato | Mar 2006 | A1 |
20060112028 | Xiao et al. | May 2006 | A1 |
20060143081 | Argaiz | Jun 2006 | A1 |
20060247900 | Brocklebank | Nov 2006 | A1 |
20070239753 | Leonard | Oct 2007 | A1 |
20070282785 | Fayyad et al. | Dec 2007 | A1 |
20070291958 | Jehan | Dec 2007 | A1 |
20080147702 | Bernhard | Jun 2008 | A1 |
20080294651 | Masuyama et al. | Nov 2008 | A1 |
20090150368 | Orumchian et al. | Jun 2009 | A1 |
20090164171 | Wold et al. | Jun 2009 | A1 |
20100036643 | Marvasti et al. | Feb 2010 | A1 |
20100036857 | Marvasti et al. | Feb 2010 | A1 |
20100161652 | Bellare et al. | Jun 2010 | A1 |
20100189354 | de Campos et al. | Jul 2010 | A1 |
20110035379 | Chen et al. | Feb 2011 | A1 |
20110196872 | Sims et al. | Aug 2011 | A1 |
20120102066 | Eronen et al. | Apr 2012 | A1 |
20130198152 | McGhee et al. | Aug 2013 | A1 |
Entry |
---|
Product Brochure, ForecastPRO, (2000) 12 pp. |
Automatic Forecasting Systems Inc., Autobox 5.0 for Windows User's Guide, 82 pp. (1999). |
Quest Software, “Funnel Web Analyzer: Analyzing the Way Visitors Interact with Your Web Site”, download from http://www.quest.com/funnel—web/analyzer 2 pp. |
Funnel Web Web site Analysis Report, Funnel Web Demonstration, Clients History, download from http://www/quest.com/funnel—web/analyzer/sample.CiientHist.html 2pp. |
Funnel Web Web site Analysis Report, Funnel Web Demonstration, Authenticated Users History, download from http://www.quest.com/funnel—web/analyzer/sample/UserHist.html 1 pg. |
SAS Institute Inc., SAS/ETS User's Guide, Version 8, pp. 577-631 and 1307-1323 (1999). |
Keogh, E. & Pazzani, M. (2001). Derivative Dynamic Time Warping. In First SIAM International Conference on Data Mining (SDM'2001), Chicago, USA, pp. 5-6. |
Oates et al., Clustering Time Series with Hidden Markov Models and Dynamic Time Warping, 1999, Computer Science Department, LGRC University of Massachusetts, in Proceedings of the IJCAI-99, pp. 1-5. |
Crowe, Keith E. et al., U.S. Appl. No. 11/431,089 filed May 9, 2006 entitled “Computer-Implemented System and Method for Generating Forecasts”. |
Jackson, Wilma S. et al., U.S. Appl. No. 11/431,127 filed May 9, 2006 entitled “Computer-Implemented Systems and Methods for Defining Events”. |
Park, Youngjin et al., U.S. Appl. No. 11/431,116 filed May 9, 2006 entitled “Computer-Implemented Systems and Methods for Processing Time Series Data”. |
Crowe, Keith E. et al., U.S. Appl. No. 11/431,123 filed May 9, 2006 entitled “Computer-Implemented Systems and Methods for Storing Data Analysis Models”. |
“Data Mining Group”, www.dmg.org, printed May 9, 2005, 3 pp. |
Kumar, Mahesh, “Combining Forecasts Using Clustering”, Rutcor Research Report 40-2005, Dec. 2005, cover p. and pp. 116. |
Leonard, Michael, “Promotional Analysis and Forecasting for Demand Planning: A Practical Time Series Approach”, with exhibits 1 and 2, SAS Institute Inc., Cary, North Carolina, 2000, 50 pp. |
Leonard, Michael, “Large-Scale Automatic Forecasting Using Inputs and Calendar Events”, abstract and presentation, International Symposium on Forecasting Conference, Jul. 4-7, 2004, 56 pp. |
Leonard, Michael et al., “Mining Transactional and Time Series Data”, abstract and presentation, International Symposium of Forecasting, 2003, 23 pp. |
Leonard, Michael, “Large-Scale Automatic Forecasting: Millions of Forecasts”, abstract and presentation, International Symposium of Forecasting, 2002, 156 pp. |
Leonard, Michael et al., “Mining Transactional and Time Series Data”, abstract, presentation and paper, SUGI, Apr. 10-13, 2005, 142 pp. |
Leonard, Michael, “Predictive Modeling Markup Language for Time Series Models”, abstract and presentation, International Symposium on Forecasting Conference, Jul. 4-7, 2004, 35 pp. |
Leonard, Michael, “Large Scale Automatic Forecasting Using Inputs and Calendar Events”, White Paper, 2005, pp. 1-27. |
van Wijk, Jarke J. et al., “Cluster and Calendar based Visualization of Time Series Data”, IEEE Symposium on Information Visualization (INFOVIS '99), San Francisco, pp. 1-6 [Oct. 25-26, 1999]. |
Choudhury, J. Paul et al., “Forecasting of Engineering Manpower Through Fuzzy Associative Memory Neural Network with ARIMA: A Comparative Study”, Neurocomputing, vol. 47, Iss. 1-4, pp. 241-257 (Aug. 2002). |
Harrison, H.C. et al., “An Intelligent Business Forecasting System”, ACM Annual Computer Science Conference, pp. 229-236 (1993). |
Jacobsen, Erik et al., “Assigning Confidence to Conditional Branch Predictions”, IEEE, Proceedings of the 29th Annual International Symposium on Microarchitecture, 12 pp. (Dec. 2-4, 1996). |
Kobbacy, Khairy A.H., et al., “Towards the development of an intelligent inventory management system”, Abstract, Integrated Manufacturing Systems, vol. 10, Issue 6, (1999) 11 pp. |
Lu, Sheng et al., “A New Algorithm for Linear and Nonlinear ARMA Model Parameter Estimation Using Affine Geometry”, IEEE Transactions on Biomedical Engineering, vol. 48, No. 10, pp. 1116-1124 (Oct. 2001). |
Malhotra, Manoj K. et al., “Decision making using multiple models”, European Journal of Operational Research, 114, pp. 1-14 (1999). |
McQuarrie, Allan D.R. et al., “Regression and Time Series Model Selection”, World Scientific Publishing Co. Pte. Ltd., 40 pp. (1998). |
Park, Kwan Hee, “Development and evaluation of a prototype expert system for forecasting models”, Abstract, Mississippi State University, (1990) 1 pg. |
Tashman, Leonard J. et al., “Automatic Forecasting Software: A Survey and Evaluation”, Abstract, International Journal of Forecasting, vol. 7, Issue 2, (Aug. 1991) 1 pg. |
Wang, Liang et al., “An Expert System for Forecasting Model Selection”, IEEE, pp. 704-709 (1992). |
Non-Final Office Action of Aug. 29, 2009 for U.S. Appl. No. 11/696,951, 28 pages. |
Notice of Allowance of Mar. 10, 2010 for U.S. Appl. No. 11/696,951, 10 pages. |
Alonso, A.M. et al., “Time series clustering based on forecast densities,” Computational Statistics & Data Analysis, vol. 51, Issue 2, pp. 1-22 (Nov. 2006). |
Kullback, S. et al., “On Information and Sufficiency,” The Annals of Mathematical Statistics, pp. 79-86 (1951). |
Quest Software, “Funnel Web Analyzer: Analyzing the Way Visitors Interact with Your Web Site”, download from http://www.quest.com/funnel—web/analyzer 2 pp (Accessed Way Visitors on Mar. 18, 2002). |
Funnel Web Web site Analysis Report, Funnel Web Demonstration, Clients History, download from http://www/quest.com/funnel web/analyzer/sample.CiientHist.html 2 pp (Accessed on Mar. 18, 2002). |
Funnel Web Web site Analysis Report, Funnel Web Demonstration, Authenticated Users History, download from http://www.quest.com/funnel—web/analyzer/sample/UserHist.html 1 pg (Accessed on Mar. 18, 2002). |
Number | Date | Country | |
---|---|---|---|
20120310939 A1 | Dec 2012 | US |