Patent Application
20230037619
Portions of the material in this patent document are subject to copyright protection under the copyright laws of the United States and of other countries. The owner of the copyright rights has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the United States Patent and Trademark Office publicly available file or records, but otherwise reserves all copyright rights whatsoever. The copyright owner does not hereby waive any of its rights to have this patent document maintained in secrecy, including without limitation its rights pursuant to 37 C.F.R. § 1.14.
According to some aspects described herein, it is appreciated that it would be useful to be able to store timeseries data in a non-relational database format. Timeseries information is used by a number of systems for recording data retrieved over time, such is done in multiple types of systems/industries such as the Internet of Things (IoT), manufacturing, utilities, energy, retail, advertising, E-commerce, financial services, banking, stock brokerages, among others that store and analyze data over time.
Historically, nonrelational database formats such as those provided by MongoDB include NoSQL formats which were previously non-conducive for storing timeseries collections, as many of these formats are based upon documents, not time. Conventionally, timeseries data would be stored natively in an SQL database format or converted to such a format to perform SQL-like functions. In some embodiments described herein, event data may be stored in a data structure defined by documents. It is appreciated also that other document-based databases or other database formats may be modified to or suitably use timeseries information.
In some implementations, events measured at various points in time may be organized in a data structure that defines an event represented by a document. In particular, events can be organized in columns of documents referred to as buckets. These buckets may be indexed using B-trees by addressing metadata values or value ranges. Buckets may be defined by periods of time. Documents may also be geoindexed and stored in one or more locations in a distributed computer network. One or more secondary indexes may be created based on time and/or metadata values within documents.
A number of additional functionalities may be defined that processes elements of the storage format, such as, for example methods for manipulating timeseries data in association with an aggregation pipeline of operations, such as an aggregation pipeline provided in NoSQL systems commercially available from MongoDB. Aggregation pipelines and their operations are more fully described in U.S. Pat. No. 10,366,100, entitled “AGGREGATION FRAMEWORK SYSTEM ARCHITECTURE AND METHOD,” filed May 25, 2017, incorporated by reference by its entirety. Using pipelines, the database may create an on-demand materialized view of the data which comprises an independent collection upon which operations can be performed. Further, methods may be provided for sampling data elements over buckets, performing bucket unpacking operations, performing densification operations on data sets, archiving data buckets to cold storage, performing fast deletes of bucket data, performing windowing operations, among other functionalities that can be used with timeseries data.
According to one aspect a system is provided. The system comprises a database engine configured to store, in a database, a plurality of timeseries events as a plurality of documents within a bucket, the database engine being further configured to: store, in a columnar format, the plurality of timeseries events represented by the plurality of respective documents, and index the plurality of timeseries events represented by the plurality of respective documents based on time values.
According to one embodiment the database is a non-relational database comprising the plurality of documents. According to one embodiment the database engine is configured to index the plurality of documents using a B-tree. According to one embodiment the database engine is configured to store a time-based event that is represented by a single logical document. According to one embodiment the database engine is configured to index the plurality of documents.
According to one embodiment the database engine is configured to index the plurality of documents based on a user-defined entity. According to one embodiment the user-defined entity includes metadata values within the plurality of documents, and wherein the metadata values include at least one of the group comprising a data source and a data region. According to one embodiment the database engine is further configured to create an on-demand materialized view of the plurality of documents. According to one embodiment the on-demand materialized view of the plurality of documents is an independent collection of data. According to one embodiment the independent collection of data is created within a pipeline processing stage using at least one pipeline operator.
According to one embodiment each bucket of documents represents data collected at a particular moment of time. According to one embodiment the database engine is adapted to sample the database comprising the plurality of documents within buckets. According to one embodiment the bucket includes a group of measurements each having the same metadata over a limited period of time. According to one embodiment each bucket is indexed with a respective key. According to one embodiment the database engine is configured to perform a random sampling of buckets. According to one embodiment the database engine is configured to perform an unpacking of the bucket using a pipeline operator.
According to one embodiment the database engine is configured to perform windowing operations using window bounds based on time and/or the plurality of documents. According to one embodiment the database engine is adapted to perform a windowing operation that produces an output stage that depends upon a range of input documents defined by the window bounds and a partition key. According to one embodiment the bucket defines a window of a predetermined amount of time. According to one embodiment at least one or more buckets associated with a plurality of windows are overlapping with respect to time. According to one embodiment the database engine is configured to index the plurality of time series events based on geographically-based indices. According to one embodiment the database engine is configured to archive data associated with a selected one or more buckets to a cold storage entity and delete, from a hot storage location, the selected one or more buckets. According to one embodiment the database engine is configured to archive data to a cold storage entity based on one or more parameters based on the documents.
According to one aspect a method is provided. The method comprises storing, by a database engine in a database, a plurality of timeseries events as a plurality of documents within a bucket, the database engine being further configured to perform acts of: storing, in a columnar format, the plurality of timeseries events represented by the plurality of respective documents, and indexing the plurality of timeseries events represented by the plurality of respective documents based on time values.
According to one embodiment the database is a non-relational database comprising the plurality of documents. According to one embodiment, the method further comprises indexing the plurality of documents using a B-tree. According to one embodiment the database engine is configured to store a time-based event that is represented by a single logical document. According to one embodiment, the method further comprises an act of indexing, by the database engine, the plurality of documents based on a user-defined entity. According to one embodiment the user-defined entity includes metadata values within the plurality of documents, and wherein the metadata values include at least one of the group comprising a data source and a data region. According to one embodiment, the method further comprises an act of creating an on-demand materialized view of the plurality of documents.
According to one aspect a system is provided. The system comprises a database engine configured to store, in a database, a plurality of timeseries events as a plurality of documents within a bucket, the database engine being further configured to: store, in a columnar format, the plurality of timeseries events represented by the plurality of respective documents, and to sample the database comprising the plurality of documents within buckets. According to one embodiment the database engine is configured to perform a sampling of buckets. According to one embodiment the database engine is configured to perform a random sampling of buckets. According to one embodiment the database engine is configured to perform an unpacking of the bucket using a pipeline operator. According to one embodiment the plurality of documents are stored in the database as BSON objects. According to one embodiment the database engine is further configured to implement a sampling function as part of an aggregation operation of a BSON database.
According to one embodiment the bucket is constructed by hashing measurements on a predetermined field. According to one embodiment the database engine is further configured to perform a sampling algorithm that causes the database engine to: select a random bucket of a plurality of buckets in the database, place a cursor on the selected random bucket, generate a random integer, determine bucket depth, and if the random integer is determined to be less than the bucket depth, mark a sampling iteration as a miss and move the cursor to the next bucket for sampling, if the random integer is determined not to be less than the bucket depth, extract a sampled element from selected bucket, and place the sampled element in an output sample set. According to one embodiment the database engine is further configured to eliminating duplicate samples from a sampled data set. According to one embodiment the database engine is configured to determine a duplicate sample based on a determined hash value of a bucket and an associated measurement. According to one embodiment the database engine is configured to determine the depth of a bucket by inferring an upper and lower bound. According to one embodiment the database engine is configured to determine an average bucket fullness of a plurality of buckets associated with the database.
According to one embodiment the database engine is configured to implement a sampling algorithm based on the determination of average bucket fullness. According to one embodiment the database is a non-relational database comprising the plurality of documents. According to one embodiment the database engine is configured to store a time-based event that is represented by a single logical document. According to one embodiment the database engine is further configured to create an on-demand materialized view of the plurality of documents. According to one embodiment the on-demand materialized view of the plurality of documents is an independent collection of data. According to one embodiment the independent collection of data is created within a pipeline processing stage using at least one pipeline operator. According to one embodiment each bucket of documents represents data collected at a particular moment of time. According to one embodiment the database engine is adapted to sample the database comprising the plurality of documents within buckets. According to one embodiment the bucket includes a group of measurements each having the same metadata over a limited period of time.
According to one aspect a method is provided. The method comprises storing, by a database engine in a database, a plurality of timeseries events as a plurality of documents within a bucket, the database engine being further configured to perform acts of: storing, in a columnar format, the plurality of timeseries events represented by the plurality of respective documents, and sampling the database comprising the plurality of documents within buckets.
According to one embodiment, the method further comprises performing, by the database engine, a sampling of buckets. According to one embodiment, the method further comprises performing, by the database engine, a random sampling of buckets. According to one embodiment performing an unpacking of the bucket using a pipeline operator. According to one embodiment, the method further comprises an act of storing the plurality of documents in the database as BSON objects. According to one embodiment implementing a sampling function as part of an aggregation operation of a BSON database. According to one embodiment, the method further comprises an act of constructing a bucket responsive to hashing measurements on a predetermined field. According to one embodiment, the method further comprises an act of performing a sampling algorithm that causes the database engine to perform acts of: selecting a random bucket of a plurality of buckets in the database, placing a cursor on the selected random bucket, generating a random integer, determining bucket depth, if the random integer is determined to be less than the bucket depth, marking a sampling iteration as a miss and move the cursor to the next bucket for sampling, if the random integer is determined not to be less than the bucket depth, extracting a sampled element from selected bucket, and place the sampled element in an output sample set. According to one embodiment, the method further comprises an act of eliminating duplicate samples from a sampled data set. According to one embodiment, the method further comprises an act of determining a duplicate sample based on a determined hash value of a bucket and an associated measurement.
According to one aspect a system is provided. The system comprises a database engine configured to store, in a database, a plurality of timeseries events as a plurality of documents within a bucket, the database engine being further configured to: store, in a columnar format, the plurality of timeseries events represented by the plurality of respective documents; and perform an unpacking of the bucket using a pipeline operator.
According to one embodiment the database engine is configured to unpack one or more of the plurality of timeseries events from the bucket. According to one embodiment the database engine is configured to identify one or more buckets of a data collection identified in the database to unpack. According to one embodiment the database engine is configured to unpack the one or more of the plurality of timeseries events from the bucket one event at a time. According to one embodiment the database engine is configured to inspect a top-level data region of the bucket based on field names and wherein the database engine is further configured to construct a list of events to unpack from the bucket. According to one embodiment the database is a non-relational database comprising the plurality of documents. According to one embodiment the database engine is configured to store a time-based event that is represented by a single logical document. According to one embodiment the database engine is configured to create an on-demand materialized view of the plurality of documents responsive to an unpacking event. According to one embodiment the on-demand materialized view of the plurality of documents is an independent collection of data. According to one embodiment the independent collection of data is created within a pipeline processing stage using at least one pipeline operator. According to one embodiment the bucket includes a group of measurements each having the same metadata over a limited period of time.
According to one aspect a method is provided. The method comprises storing, by a database engine in a database, a plurality of timeseries events as a plurality of documents within a bucket, the database engine being further configured to perform acts of: storing, in a columnar format, the plurality of timeseries events represented by the plurality of respective documents, and performing an unpacking of the bucket using a pipeline operator.
According to one embodiment, the method further comprises an act of unpacking one or more of the plurality of timeseries events from the bucket. According to one embodiment, the method further comprises an act of identifying one or more buckets of a data collection identified in the database to unpack. According to one embodiment, the method further comprises an act of unpacking the one or more of the plurality of timeseries events from the bucket one event at a time. According to one embodiment, the method further comprises an act of inspecting a top-level data region of the bucket based on field names and constructing a list of events to unpack from the bucket.
According to one embodiment the database is a non-relational database comprising the plurality of documents. According to one embodiment, the method further comprises an act of storing a time-based event that is represented by a single logical document. According to one embodiment, the method further comprises an act of creating an on-demand materialized view of the plurality of documents responsive to an unpacking event. According to one embodiment the on-demand materialized view of the plurality of documents is an independent collection of data. According to one embodiment the independent collection of data is created within a pipeline processing stage using at least one pipeline operator. According to one embodiment the bucket includes a group of measurements each having the same metadata over a limited period of time.
Still other aspects, examples, and advantages of these exemplary aspects and examples, are discussed in detail below. Moreover, it is to be understood that both the foregoing information and the following detailed description are merely illustrative examples of various aspects and examples and are intended to provide an overview or framework for understanding the nature and character of the claimed aspects and examples. Any example disclosed herein may be combined with any other example in any manner consistent with at least one of the objects, aims, and needs disclosed herein, and references to “an example,” “some examples,” “an alternate example,” “various examples,” “one example,” “at least one example,” “this and other examples” or the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the example may be included in at least one example. The appearances of such terms herein are not necessarily all referring to the same example.
Various aspects of at least one embodiment are discussed herein with reference to the accompanying figures, which are not intended to be drawn to scale. The figures are included to provide illustration and a further understanding of the various aspects and embodiments and are incorporated in and constitute a part of this specification but are not intended as a definition of the limits of the invention. Where technical features in the figures, detailed description or any claim are followed by references signs, the reference signs have been included for the sole purpose of increasing the intelligibility of the figures, detailed description, and/or claims. Accordingly, neither the reference signs nor their absence are intended to have any limiting effect on the scope of any claim elements. In the figures, each identical or nearly identical component that is illustrated in various figures is represented by a like numeral. For purposes of clarity, not every component may be labeled in every figure. In the figures:
As discussed, various aspects relate to storing timeseries data in non-relational database formats such as NoSQL. In some embodiments, timeseries event information is stored as a discrete document within a database. The database may be arranged in buckets which represent periods of time in which the events occur, and therefore the documents are collected within the buckets. The documents may include timestamp information as well as one or more metadata values (e.g., a key-value pair) which can be defined that describe the timeseries. For instance, in the case of an IoT device, one or more measurements may be stored as metadata within a particular document that represents the event. Measurement data associated with measured values during events (and are stored in documents) may include key-value pairs observed at a specific time (e.g., by an IoT device). A compilation of measurement data may be stored as a timeseries defined as a sequence of measurements over time.
Further, a bucket may be defined which includes a number of measurements having the same metadata types measured over a limited period of time. A bucket collection may be defined that is used for storing multiple buckets in a time series collection. In some embodiments, database operations such as replication, sharding, and indexing may be performed at the level of buckets in the bucket collection.
Buckets may be stored in a columnar format and may be indexed by a B-tree for easy retrieval. Further, the data structure may be indexed based on time and/or one or more metadata values within the documents. Further, as discussed, one or more pipeline operators may be used to perform operations associated with the timeseries data. In some embodiments, an on-demand materialized view that comprises an independent collection of data may be operated on by the system using one or more pipeline operators and/or stages.
In some embodiments as described herein, one or more data sources may generate timeseries event data 114 which is then processed and stored by database engine (e.g., database engine 106). For example, timeseries data may be generated by one or more systems such as those that may typically create event data such as in the manufacturing, financial services, or other types of systems. In some embodiments, one or more IoT systems (e.g., systems 113 (elements 113A-113C)) may generate events which are stored within the distributed system 101. For example, it is appreciated that there may be a number of systems that can generate and store timeseries data that may be stored by distributed system 101, and various embodiments are not limited to any particular number or type of data generating systems.
Timeseries event data is passed to the distributed system 101, received by an interface (e.g., interface 105) and forwarded to a database engine 106 which is configured to perform one or more database operations. Database engine 106 may include a number of elements including processors, elements such as routers, or other elements. Database engine 106 may include any entity related to storing data, may include hardware and/or software. In some embodiments, the database engine may include one or more processes and one or more storage entities that manage and store database entities such as documents. In some embodiments, the database engine may include a modified mongod process (commercially available from MongoDB) that is executed by a processor. Data is stored in a distributed storage entity 107 which includes one or more systems and/or storage elements.
In some embodiments, a logical structure is defined referred to herein as a bucket (e.g. bucket 108) which defines a period of time in which event data may be stored. Storage 107 may store one or more buckets (e.g., bucket A (element 110A), bucket B (element 110B)). These buckets may contain one or more documents 109 that correspond to event data collected from one or more systems. Further, system 101 may include one or more indexes used to index timeseries data, one or more pipeline operators used to perform operations on timeseries data, and other elements used to facilitate timeseries operations (e.g., windowing commands).
As discussed, by defining timeseries data as a collection of buckets and associated documents, other operations and functions may be performed on this timeseries data. For example, methods may be provided for sampling data elements over buckets, performing bucket unpacking operations, performing densification operations on data sets, archiving data buckets to cold storage, performing fast deletes of bucket data, performing windowing operations, among other functionalities that can be used with timeseries data.
In some embodiments, distributed system 301 includes a hot-storage-type database as well as a cold-storage-type database for fulfilling database requests. In one embodiment, the distributed system provides a single access interface 105 performing database operations on both types of databases. In some examples, the online database is a DaaS-type database and may include, for example, cluster-based system. Online database engine 302 may be provided that performs read and write operations to storage entities configured in a database cluster (e.g., a cluster-based database such as the ATLAS database commercially available from MongoDB).
In some embodiments, an archive manager (e.g., archive manager 304) is provided that controls how data is archived from the online database to a data archive (e.g., data archive 305). In some implementations, the data archive may be implemented as cloud-based storage elements. For example, the data archive may use data buckets defined on S3 to create one or more archives associated with an online database. In some embodiments, a capability is provided for archiving data by the database management system that reduces management effort on behalf of application creators. In some embodiments, an archive manager 304 is provided that automatically archives data from an online database to an off-line database while maintaining a single point of interface to the database. In this manner, archiving operations are transparent to end user applications.
Further, a database may be provided that fulfills data read operations from one or more hot and cold data sources. In some embodiments, a data lake (e.g., data lake 303) is provided that provides a single view of offline and online storage. As is known, data lakes generally have the ability to store both structured and unstructured data. In some embodiments, the data lake may service read operations that reference an online database. In some embodiments, the database is a DaaS-based database that implements online storage using a cluster of nodes (e.g., online database (cluster) 302). Further, the data lake services read operations to a data archive (e.g., data archive 305, such as for example, one or more S3 databuckets). In some embodiments, the data lake may be used as a single view of online cluster data and archive data.
Managing the lifecycle of time series data is an important aspect to time series customers and their workloads. The inventors appreciate that time series data will often exponentially grow to large volumes quickly and lead to considerable performance degradation and increased costs without action and planning from the user.
Users who process time series data have a few primary options when it comes to managing their data lifecycle often based on how long they wish or need to maintain data. Users can continue to scale, vertically or horizontally as their data volume grows and they choose to maintain that data in hot storage. Additionally, users can choose, if their requirements allow, to delete data directly from hot storage. Lastly, users can choose to age out data by archiving older data and keep only the freshest or most frequently accessed data stored in their active cluster.
To implement a timeseries data storage system, it is noted that the following should be considered:
The following example is implemented in a MongoDB Atlas online archive database. In some embodiments, timeseries support is provided for users to create, manage and process timeseries data within a non-relational database. Some of the implementations may have some of the following behaviors, either alone or in combination with other functionality:
Also, it may be desired to sample timeseries data for the purpose of determining certain attributes regarding the measurement data. Samples are critical for understanding flexible schemas of document collections. They can also be used for other purposes, including cardinality estimation.
At block 403, the system unpacks the bucket, and at block 404 sample measurement is taken from at least one document within the bucket. It should be appreciated that this process can involve accidentally sampling duplicate entries, therefore a system and process may be provided for eliminating duplicate samples such as at block 405. For instance, sampled items can be tracked and if selected again, the system may proceed without performing a duplicate sample. At block 406, it is determined whether the sample set is complete. If yes, process 400 ends at block 407. If not, the system proceeds to select another random bucket at block 402.
As discussed in more detail below, some aspects relate to an ARHASH algorithm that speeds up sampling over buckets preserving desirable statistical properties.
In some embodiments, a NoSQL (e.g., non-relational) database may be modified to support for time series collections. MongoDB tools such as Compass, BI Connector, and Charts rely on $sample to sample collections in order to present views to the user or infer schema information from sampled documents in collections. To support $sample of time series collections in these types of products (and for users interested in sampling time series collections) it would be beneficial to have a way to efficiently generate random samples without replacement for time series collections.
According to some embodiments, a random sampling algorithm called ARHASH may be adapted to implement $sample pushdown for time series collections. The new ARHASH-based algorithm as described herein shows a significant speed improvement over the current implementation and in some cases a 300× speedup.
According to some embodiments, time series collections are implemented as non-materialized views that are backed by a system level collection that store “buckets”. A user can create a time series collection by issuing a createCollection command with TimeseriesOptions,
Buckets hold multiple pivoted time series measurements where each measurement field _id, time, A, and B are stored in a BSONObj column since the metaField is the same for each pivoted measurement, tags are only stored once. As an example, suppose we insert three measurements into the point_data collection,
Each measurement is pivoted into a bucket according to the data in the metaField specified during collection creation, in this case the metaField is tags and because, in the example, each measurement has the same data for tags the buckets collection stores all three entries in a single bucket. Each measurement field aside from the metadata value will be pivoted into columns stored as a BSONObj indexed by a “row key” which is a zero-based null-terminated decimal string.
Notice that in the example the bucket compresses the missing B value from the measurement with row key “1”. Another important detail that is helpful is that the time column is non-sparse. This is due to the fact that, in some embodiments, the timeField is required for each measurement, and we can use this knowledge to determine how many measurements (bucketDepth) in a given bucket.
In some implementations, the time series collection view is backed by a new stage called $_internalUnpackBucket. This new $_internalUnpackBucket stage unpacks all measurements in a given bucket from the system.buckets.X collection where X is the name of the time series collection. For example, an $_internalUnpackBucket stage can be used in pipelines with $sample and other stages. For example, if one wishes to extract a random sample of size 100 from a time series collection all the user needs to do run the following aggregation pipeline against their collection:
In the explain output, it can be seen that the pipeline is rewritten to have an $_internalUnpackBucket stage at the front of the pipeline followed by a $sample. If this pipeline were to run by itself, it would unpack each bucket in the db.point_data collection and run top-k sorting and choose the first 100 ranked measurements for the sample set. Now imagine that the collection stored millions of measurements and the system runs the top-k algorithm for this small sample size. The top-k sorting algorithm will materialize every single measurement in the collection only to select the top 100 ranked measurements and discard all other BSONObjs.
For better performance, it would be beneficial to find a suitable algorithm that will have the following characteristics:
The ARHASH algorithm as described below is an iterative algorithm that attempts to extract a measurement from a random bucket repeatedly until it successfully builds a sample of the target sample size, m. It should be appreciated that the algorithm performs roughly O(m) work instead of doing roughly O(n*log n) work during top-k sorting where n is not the sample size, but rather n the number of measurements in the collection.
The algorithm is best described by example below that shows the algorithm performance through a series of diagrams as shown in
There is a sample set created (e.g., of size=4, element 1006)
When the system has selected a measurement that has already been selected, the system advances the RandomCursor to bucket 0 and then generates a random Int, 2, attempting to extract the measurement with row key 2 from bucket 0. Duplicates are tracked by hashing a std::pair<BucketId, MeasurementId> in a std::unordered_map. When a duplicate is encountered the system counts it as a miss and proceeds on.
The ARHASH algorithm is terminated when the system has built the entire sample set, and in this case the system would have sampled a final measurement from bucket 2 by generating a random Int 1 and extracting the measurement at row key 1, and the algorithm terminates.
In some embodiments, this algorithm may be implemented as, in at least one implementation, using a WiredTiger RandomCursor, a pseudo-random number generator, and a mechanism to extract the k-th element from a bucket which can be implemented through an abstraction called a BucketUnpacker.
On implementing ARHASH the it is appreciated that two interesting challenges that may be addressed:
The first challenge of computing the bucketDepth at each iteration may be accomplished by exploiting the structure of the bucket format's timestamp column. In some implementations, this approach may be used as, in some implementations, the bucket format does not track additional metadata around the measurement count or other stats of that nature. Alternatively, the system could walk the timestamp BSONObj and count the number of elements to infer the bucketDepth.
Computing the bucketDepth in O(1)
Rather than walking the timestamp column, some facts may be used about this
Given these three facts, a recurrence relation can be written for the timestamp column size Si as a function of the number of row keys i. For simplicity and in order to eliminate log 10s from the equation, the recurrence may be written at powers of 10,
S
0=5
S
1
=S
0+110
S
i
=S
i-1+(10i−10(i-1))*(10+i),where i>1
Now to compute the bucketDepth, the system can read the objsize header from the timestamp BSONObj and find an upper and lower bound for the actual object size by searching the table. Once the objsize is bounded we use linear interpolation to infer the bucket size.
Now with all of the pieces discussed, consider the situation when the buckets are not entirely full and the ARHASH algorithm realizes a majority of misses. This was the second hurdle encountered and in this case, the ARHASH algorithm spun and never beat top-k sorting. To complicate things further, there is no way to know upfront if this case exists without inspecting buckets on the fly or by tracking statistics which seemed to be error prone.
Non-Full bucket Case
It is appreciated that buckets could be mostly empty as some workloads can generate buckets that are nearly empty due to the fact that measurements are stored by met aField value. It is appreciated that buckets not are opened and filled one-at-a-time as one may believe. This phenomenon is illustrated in
Notice that a new term is defined, referred to herein as average bucket fullness (ABF). This parameter is useful for utilizing the TrialStage to determine if ARHASH would be chosen over top-k. The use of the TrialStage helped us overcome the half-full bucket case, and relied on running a runtime trial for 100 iterations of the algorithm to essentially check if there is an ABF˜0.5 and use the ARHASH algorithm instead of the top-k algorithm.
Before getting into how the ABF may be used by the TrialStage, the TrialStage operation is now discussed.
In this case, the system can pushdown a TrialStage (e.g., TrialStage 1704) into a find layer (element 1702) with two branches,
When the TrialStage is instantiated, in one implementation, it runs a trial of 100 work units and it tracks how many times the trial branch selects a measurement by counting how many ADVANCED states are reached. It also tracks the total number of work units in total to compute an ADVANCED:Work ratio. The trial branch is successful if the ADVANCED:Work ratio exceeds some threshold on [0, 1]. That threshold was chosen to be an ABF of 0.5.
A question one might have at this point is why does the ABF approximate the ADVANCED:Work ratio of the TrialStage? To answer this, the SampleFromTimeseriesBucket stage may be configured to return an ADVANCED state if a measurement is sampled (a hit), and NEED_TIME otherwise (a miss). So if the timeseriesBucketMaxCount is 1000 on average the system ADVANCE the ARHASH algorithm more than half the time, the system would be generating random ints sufficiently to sample buckets with at least 500 measurements in them, and by extension, the system would are observing an ABF of 0.5. Now the system can control the TrialStage to select the trial branch if the ADVANCED/WORK ratio is at or above 0.5, otherwise the system would fallback to top-k sampling.
ARHASH can be executed for a variety of bucket fill factors and number of measurements inserted into time series collections. Interestingly, it was found that ARHASH beat top-k when roughly sampling <1% of the measurement count of the collection and when the ABF >=0.5. Heuristics are not always perfect, but the system may attempt to use ARHASH via the TrialStage when the sample size was <1% of the collection size as something to try.
The plot 1800 shown in
Notice that the query planner has selected a pipeline with a $cursor plan stage that contains only a COLLSCAN (e.g., element 1711) ollowed by an $_internalUnpackBucket stage followed by a $sample stage. Here, the system is sampling a sample size that is significantly larger than %1 of the collection so the system may avoid running the ARHASH algorithm. Suppose the system samples 1000 measurements now,
It can be observed that the query planner chose the pushed down plan with the TrialStage, and the winning plan is TrialStage where the trial branch containing the SampleFromTimeseriesBucket stage (e.g., element 1707) that implements ARHASH was chosen. Now the system can execute two pipelines, one that inhibits the optimization before $sample so the system forces the query planner to run top-k and another that does not inhibit the optimization before the $s ample stage which will run the ARHASH algorithm via the TrialStage.
It should be appreciated that by adapting the ARHASH algorithm to sample time series collections, a faster $sample path for time series collections may result. However, it should be appreciated that other similar algorithms may be used.
Many queries can benefit from events being ordered on time, in particular, when $sort on time is present, e.g. for visualization, or window functions over the time dimensions need to be computed or data need to be grouped on time-bins (e.g., months). An operation that can unpack buckets and perform sorting would be beneficial for performing such operations. Because buckets are stored in a clustered index based on the minimum time of events stored in each bucket, scanning and unpacking results in an ordered collection. Bucket unpacking can be extended to sort the events as it is consuming them to satisfy the needs of subsequent stages. This makes unpacking a blocking operator but, in practice, it should only hold events until a bucket with a greater minimum timestamp gets opened. This approach can be extended to the cases when events need to be grouped prior to sorting and buckets are retrieved from a secondary index on corresponding metadata fields and timestamp.
In some embodiments, a component is provided (hereinafter the “BucketUnpacker”) that is used to unpack the time series bucket storage format. It essentially takes a bucket from a time series collection and unpacks time series measurements one at a time. The BucketUnpacker is intended to be used in two phases:
1. An initialization phase that will analyze the owned bucket and set up the necessary BSONObjIterators needed to materialize measurements.
2. The main iteration that is driven by BucketUnpacker::getNext( ) calls to unpack the bucket one measurement at-a-time.
The initialization phase is done once per bucket in BucketUnpacker::reset( ). This phase caches the metadata value because it's repeated across all measurements in a bucket. Subsequently, a BSONObjIterator is initialized for the timestamp column for the purpose of providing the canonical ordering of row keys used to track which values are needed for the measurement being processed in the iteration. The value from this time field iterator can be used to materialize the time value if needed.
To determine which columns to unpack, BucketUnpacker::reset( ) computes the set of BSONObjIterators needed by traversing the top-level data region of the bucket, inspecting the field names, and comparing them against the provided include/exclude list. Once the initialization phase is complete, BucketUnpacker::getNext( ) can be called until exhaustion. This is a forward iterative process, the iterator does not peek, nor go backwards during the iteration because these features are not needed at the moment.
The mechanics of the iteration during BucketUnpacker::getNext( ) calls can be realized in a single-pass algorithm using constant space (auxiliary state alongside the materialized document) that accumulates the materialized document in a MutableDocument:
1. Populate a MutableDocument with the metadata fields if the metaField is either provided in the include list or it is not explicitly listed in the exclude list.
2. Get the current row key and timestamp by reading the next item from the timeField BSONObjIterator. If the time field should be present in the resulting document, place it at the front of the resulting document.
3. Loop over the bucket BSONObjIterators to unpack measurements
a. Check to see if there is a matching row key in the current BSONObjIterator.
b. If there is a next value the system appends it to the MutableDocument and advance the current BSONObjIterator. Otherwise, the field is missing for the current unpacked measurement and the system skips advancing this BSONObjIterator.
c. The whole iterator is exhausted when both the timestamp BSONObjIteratorand all column iterators are exhausted.
As discussed, it may be desired, depending on the operation to be performed, to permit analysis of timeseries according to one or more window-based operations.
Such functions differ from the MongoDB aggregation operators (e.g., $group, $set) in the following ways:
$expMovingAvg computes an exponentially weighted moving average.
Windowing Functions
Time series calculations using windows could be classified into two main categories:
According to some implementations, some use cases cover not only IoT data but also common analytics scenarios both categories may be implemented.
According to some embodiments as described herein, window functions are exposed as a new aggregation stage capable of partitioning incoming data similar to $group, and apply one or more functions to defined windows within each partition. However these window functions do not change the granularity of the data like $group does.
In the following example, assume the user wants to calculate a 3-period moving average of temperature, looking forward 1 document and back 1 document (+1 current document) for each sensor.
The syntax for this behavior is as follows:
This solution works well when data points are evenly spaced and without missing values. According to some embodiments, functionality may be provided that is similar to SQL environments, but in a document-based, no SQL architecture.
Assume the user wants to compute a 10-second moving average of temperature looking back 10 seconds from the current timestamp for each sensor. This could be to smooth the signal or calculate the difference between the current data point and the moving average (e.g. to determine if the current point is a potential outlier).
The syntax for this behavior is as follows:
Unlike the previous option, this solution is less sensitive to missing data as it qualifies surrounding values by not just their position in an ordered list but their temporal distance, which is significantly different than traditional SQL-based analytical functions.
If no documents fall in the range (empty window) or the input values are of an incompatible type (e.g., $sum over strings), the returned value is function-dependent but will be consistent with $group where appropriate:
If the current value for the sortBy dimension is non-numeric and non-datetime all functions return null as the window itself is undefined.
In another example, if the user wanted to see the unique products browsed in the past 1 hour by each customer, the syntax would be as follows:
If the user wanted to understand how movements of two series are directionally coordinated with each other, the syntax would be as follows:
If the user wanted to track the price of an investment (like a stock or commodity) for volatility in the last 20 periods with more weight on recent events, the syntax would be as follows:
If the user wanted to compute the temperature change around a power meter make sure for any given 3 minute time window the temperature doesn't change too fast to indicate a fire in the facility, the syntax would be as follows:
Derivative can be computed by taking the first and last points in the sorted window; and does not consider many points that might be in between.
For sortBy fields which are timestamps, unit can be specified to indicate the desired time unit for the divisor. unit is optional and should be left off for non-date sortBy fields. If the unit is present but the boundary values are not dates, $derivative will return null. It defaults to ‘milliseconds’ for sortBy fields which are dates. The unit will be treated as a scaling factor, meaning fractional values are allowed and preserved as the divisor, but also meaning that units “months” and “years” cannot be used since these cannot be converted to a fixed number of milliseconds.
If the user wanted to compute the running sum of units manufactured for each Assembly Line the syntax would be as follows:
If the user wanted to compute the total power production from a solar panel over the course of past hour, the syntax would be as follows:
This function computes the approximate integral of the window by using the trapezoidal rule, and it is a cumulative function e.g. if there are 3 data points, it first computes the integral for the first window (between 1-2), then between the next pair (2-3) then sums them up.
Similar to $derivative, for sortBy fields which are timestamps the unit can be specified. The semantics of unit are identical to those described above in $derivative, except here the unit applies to a multiplication instead of the divisor.
If the user wanted to build a path out of pairs of latitude-longitude coordinates e.g. trajectory of a traveling vehicle, the syntax would be as follows:
If the user wanted to compute a percent contribution of each production facility to the region they are part of the syntax would be as follows:
Rank creates a ranking of the rows based on a provided column. It starts with assigning “1” to the first row in the order and increments the following rows using different logic depending on the type of the ranking method used.
$documentNumber is the equivalent of the ROW_NUMBER( ) function in SQL and returns the unique document number based on the order of documents in the current partition. This is a monotonically increasing counter.
$rank returns the ranking of the current document within the current partition. Documents with the same value(s) for the sortBy by fields will result in the same ranking. For example, the sequence of values [25, 25, 50, 75, 75, 100] when ranked will return [1, 1, 3, 4, 4, 6]. Note that the values 2 and 5 are skipped.
$denseRank is the equivalent of the DENSE_RANK( ) function in SQL and handles ties differently from $rank by assigning duplicates the next subsequent rank value. If we take the same sequence of values from the previous example: [25, 25, 50, 75, 75, 100], denseRank will return [1,1,2,3,3,4]. Note that unlike $rank no values are skipped.
If the user wanted to compute the days since the first time an event had occurred (recency analysis) the syntax to retrieve the first event would be as follows:
Note that a NULL value could be first or last depending on the sort order and BSON spec collation behavior. There may be a provided option to override this default sort behavior.
Assume the user wants to compute the difference between the current period and 4 periods earlier for each Weather Station to get Q1 2020-Q1 2019, Q2 2020-Q2 2019 and so on. Shift has an implied window frame of the entire partition and will error if the user explicitly specifies one (see error messages).
The syntax for to access the value of temperature from 4 periods prior is as follows:
At a partition boundary some referenced points may not exist.
The syntax for to access the value of temperature from two periods prior on data partition by Year is as follows:
At a partition boundary some referenced points may not exist as shown in
In this case the process returns the default value of NULL. The user can override it by using the default option. This may be typically used when the beginning of a period implies a value because that is where measurement began (e.g. count, sales total etc. being 0) or in some cases if the metric is decrementing could be starting at a higher value.
A 5-second moving average of temperature per sensor reported every 5 seconds (tumbling window).
This example estimates bin boundaries exactly the same way dateTrunc does, locations of truncated bins are represented by dashed lines in
In some embodiments, it will be possible to achieve this using the following syntax combining dateTrunc and a group:
One approach is to offer some “syntactic sugar” and handle these as by extending the existing $bucket aggregation stage.
A 10-second moving average of a temperature per sensor reported every 5 seconds (hopping window).
In
At minimum, it will be possible to achieve this result using the following syntax combining time based sliding window and a group:
This comes with the limitation that window size has to be a factor of hop size (e.g. a 5 second hop) could have 10, 15, 20, etc. second windows and if the user provides a 13 second window, it will not raise any error and instead behave as if it were a 15 second window, since the dates between 10 and 15 seconds would all be truncated to 10.
One approach is to offer some “syntactic sugar” and error handling when hop and window sizes do not support the aforementioned constraint in which case the syntax will look as follows.
At minimum, it will be possible to achieve this result using the following syntax combining time-based sliding window and a group:
One goal is to offer some “syntactic sugar”, with the help of which 3-second wide, 5-second hopping window could look as follows:
As you can tell, this piggy-backs on the existing bucket calculation e.g. with similar extension to bucket, user can bin a numeric field into bin sizes of 20 to create a histogram like output using:
In this case, the bucket assignments would be the equivalent of CEILING(mySales/20)*20.
If the user wanted to count the number of clicks in a session that terminates if there is no activity for more than 45 seconds, the syntax is as follows.
In some embodiments, the same will be possible to achieve this using the following syntax combining shift, datediff and group as a workaround:
In some embodiments, secondary indexes only store one key per bucket, or group of measurements. For secondary indexes on time and other metrics, this key is a compound value of the minimum and maximum values in the bucket. This allows indexes to be orders of magnitude smaller than data, and allows optimized queries to skip vast ranges of data. The order of the minimum and maximum in the compound index key is reversed for descending indexes, allowing for efficient top-k queries.
In some additional embodiments, secondary indexes on geo-spatial metrics compute a small set of S2 cells, each a spherical quadrilateral, representing a region that covers all location values of the indexed field over the measurements included in the bucket. This allows indexing just a few cells per bucket, rather than a cell per metric.
As time-series collections will support secondary indexes by default, this project will use the existing featureFlagTimeseriesCollection flag from PM-1952.
createIndexes
An example below uses a time-series collection abc created with the following options as an example:
When creating a new index on abc, the index specification will be transformed into an index on the underlying bucket collection in the db.system.buckets.abc namespace.
A createIndexes command with a single metadata field:
will be equivalent to the follow operation on the underlying buckets collection:
By definition, the metadata field in the bucket collection is always named meta.
If the index requested is a compound index with both the metadata and the time fields:
The index specification created on the bucket collection will be:
Because buckets in the underlying bucket collection may contain overlapping time ranges, we include both lower and upper bounds in the index to support the query optimizer's ability to order measurements. In this example, including control.min.time allows us to create bounds on the minimum time, Including control.max.time does not help provide an order, given the assumption that control.min.time will often be unique to a bucket. Instead it will allow us to confirm/reject a bucket without having to fetch it on the basis of time range.
Conversely if the time field has to be indexed in descending order, the system would transform a compound index:
Any data but time-series, in particular, often misses documents because measurements were not captured. For example, two time-series for two products tracked over a certain period of time may not have events for each timestamp. That makes the data hard to compare and visualize. Also, there may be events missing for common timestamps in a given domain, say months, which gaps would skew people's perception of visualized data or make computation of window functions harder. This is separate but often followed by filling in missing values in documents without certain measurements.
It is pretty common for time series data to be uneven and have gaps. In order to perform analytics on such data and ensure correctness of results, quite often the gaps need to be filled first.
We will solve this by introducing two new aggregation stages: $densify and $fill. $densify will create new documents to eliminate the gaps in the time or numeric domain at the required granularity level. $fill will set values for the fields when value is null or missing. Filling of missing values can be done using linear interpolation, carrying over the last observation (or carrying backward next observation), or with constant. Densification and gap filling will be supported on both time series collections and regular collections as they will also benefit the ongoing real-timeanalytics effort.
The syntax used below is a placeholder to help think through the use cases and will most likely be changed as part of team review and/or during technical design. Those changes will be reflected in this document to avoid confusion among future readers.
For example: densify date field “ts” to get to 5 minutes granularity for each series defined by fields “meta.location” and “meta.model” within a time range.
The last value that we generate for ‘ts’ field will be less or equal to the right bound of the range.
Created documents will look like this:
As an example, here are four different options for densification over the numeric field ‘altitude’ in a ‘coffee’ collection:
As shown by 2700 in
As shown by 2800 in
As shown by 2900 in
As shown by 3000 in
In some embodiments, gap filling may be performed on some time series collections. In one implementation, a new stage called $fill sets values for null or missing fields. It can fill values using linear interpolation, last observation carried forward (“locf”), or expression that evaluates to a constant.
Note: interpolation and locf can be achieved with window functions, and filling with constant can be done with $ifNull. $fill stage is a syntactic sugar.
Stage $fill can fill in multiple fields at once using different methods.
Below are a number of example use cases where densification may be used. Such use cases refer to the various examples provided below.
Report hourly average temperature, total motion and maximum of stock items for each room. If data is missing, fill in the average temperature using interpolation, total motion as 0, and for maximum stock of items—carry over the last observed value.
A customer has 3 metrics (motion, temperature, and number of items stored in the room). Some timestamps have some data points missing. They want to get a view of each timestamp with all metrics filled in. Densification of timestamps is not required.
If data for motion is missing, it should be filled in with 0, temperature should be interpolated, and number of items should be carried forward from the last known value.
Here interpolation was based on the number of documents. What is ignored here is the fact that the points for temperature are not distributed evenly over ‘ts’: 03:01, 03:06, 03:10, 03:21, 03:30.
Ideally interpolation will take into consideration the distance between documents along the ‘ts’ time field. For instance,
Join prices of two different assets with non-matching timestamps to plot as a 15 minute time series chart with no gaps where no data means carrying the last non-NULL value forward.
Collection “price_tracker”
if there are two collections, we can start with $unionWith stage to combine them together db.price_tracker_AB.aggregate([{$unionWith: {coll: ‘price_tracker_XY’}}])
It may be desired to determine how many times did Coca Cola promotions overlap with Pepsi promotions in the last year. To determine this:
To find overlapping promotions we can use $lookup
If we want to get the timeline to plot on the graph, we can use densification.
Step1: Use $unionWith to combine two collection into one ‘promotions’ collection:
Step2: Transform into time series structure, limit to records for year 2020.
Now there are two documents for each promotion: one for start date, one for end date.
Step3: densify timeline for each promotion separately: partition by ‘brand’ and ‘promoId’, and fill in within the range of each partition.
Step4: apply $densify again to fill gaps in the full timeline with documents like this:
It is desired that the system create a proper histogram (gaps are zero filled, not skipped) with a bin size of 100: number of Airbnb listings over $1000/night shown in buckets by price.
In another example, it is desired to obtain the Top 5 worst selling products for each month. 0 sales is the worst and if there are no sales in a given time frame, there will be no record of that product in that time frame.
Note: We are using $topN here though it doesn't exist yet.
Two Collections:
$densify stage can include reference to a collection that will provide a full domain for densification.
The same scenario can be implemented with two densification stages: first to generate missing dates, and second to create records for products that are not present in the sales collection:
Compare each product to every other product pairwise (cartesian join) e.g. to compute a correlation or covariance matrix.
This can be achieved with current functionality of aggregation without densification.
$densify stage could support this functionality by producing all combinations of ‘field’ and ‘partition’ when ‘step’ and ‘unit’ are not specified.
How many cars have we had in the parking lot waiting to be rented daily for the past 60 days (using check out-check in dates for each vehicle, and assuming there were 20 cars in the lot initially)?
We will bin the data into daily granularity. Compute sum of check-ins and sum of checkouts. Gapfill datebins. Compute (total cars managed+running sum of check-ins—running sum ofcheck-outs) to find # cars sitting in the parking lot over time.
How many cars did we have in the parking lot last year on average (each day) for each XBZ Rental car facility (using check out-check in dates for each vehicle)?
We will reuse the calculation for number cars in a lot from the previous example, and then add calculation of average grouped by location.
For each day that has at least one sale, show how many products of each size were sold (assuming a product of every size has at least one sale entry in the range that is being analyzed)
Sample document in collection ‘sales’:
With no ‘step’ and ‘unit’, and with range: “full” densification will produce all combinations of ‘field’ and ‘partitionFields’
This can be achieved using $lookup with a pipeline like this:
For each day that has at least one sale, show how many products of each size were sold. Assuming a full list of sizes can be found in the product catalog collection.
$densify stage can include reference to a collection that will provide a full domain for densification.
For each day that has at least one sale, show how many products of each size were sold. Assuming a full list of sizes is generally known and is small enough to be passed in the query.
The guiding idea for this approach is for any document we pull from the previous stage, we will generate all documents in its partition between the last seen value in its partition and its value.
New stage: $_internalDensify
In some implementations, this stage will only be generated from $densify and will not be externally accessible. The code block is only to convey the structure of the stage.
{$_internalDensify: {field: <fieldName>, min: <value>, max: <value>, partitionFields: [<field>, . . . ], step: <value>, unit: <unit>, range: <rangeSpec>}
When built, $_internalDensify will be populated with all of the information needed to generate the new documents. This stage will assume that it is preceded by a ‘$sort’ stage, and therefore the input is sorted on ‘{field: 1}’, or on {partitionFieldA: 1, partitionFieldB: 1, . . . , field: 1}. On construction the stage will build the in memory table as a ValueMap holding Values. In some implementations, the table will start empty.
The stage will track the highest value seen for each partition in the in memory table. Upon receiving a new value in an unseen partition, the system stores the found document (in its entirety) in the Document Generator as the max value. On future getNext calls the stage will populate the missing values (between the minimum densification value and the first seen value in the partition). When the document generator is done generating documents, it will output its final value (the value pulled from the previous stage) and then on subsequent values have the “done” flag set. This has the advantage of preserving the sort order (for a given partition) for the user going into the next stage.
Similarly when a new document is seen for an already known partition, the stage will note the range of values between the previously seen value for the partition and the value in the current document. Before outputting the current document, all values between will be outputted on subsequent getNext calls before pulling new documents from the previous stage.
In the non-full case, the system will check every document to see if it is starting a new partition. If so, the system can remove all information from the in-memory table about the preceding partition (after filling to the max value in the explicit range case). Once EOF is reached (and finish filling in the explicit range case), the system is done generating documents.
In the full case once the system has processed all the input, the stage will loop over the stored partition values and output all documents between the existing values in the table and the maximum value in the range (over multiple getNext calls).
In
If the generator is not done at block 4003, it is determined whether the partition is in the table at block 4004, if not, the document is returned from the generator at block 4009. If the partition is in the table at 4004, it is determined whether the value min is divisible the step. If so, the generated (or final) value is added to the partition table at block 4008, and the document is returned by the generator.
If at block 4015, the range value is a full value, the document generator is built for the first partition in the table with its value+step as its minimum and the max as the global maximum (at block 4013). If at block 4015, the range is an explicit range, then the document generator is built for the first partition in the table with its value+step as its minimum and the max as the maximum value within the range (at block 4014). At block 4012, the partition is removed from the table.
If at block 4007 there was not seen an EOF, the system pulls the document from the previous stage. If the document is at EOF, then the process proceeds to block 4017 as discussed above. If not, the process proceeds to block 4020 where it determined whether the document partition is already in the table. If so, the process determined what the rage value is at block 4022. If the range value is export range, it is determined whether the current value is inside, below, or above the range at block 4021. Depending on the result, a generator is created as identified at blocks 4029, 4030. If below, the current document is stored in the generator as the last output value at block 4031, and the first document is returned from the generator at block 4032.
If at block 4020, the document partition is not in the table, the system determines the range value a block 4023. If the range value is full, the system builds a new generator with a max as this value and the global minimum at block 4024. If the range value is partition, the current document is stored in the generator as the last output value. If the range value is an explicit range, it is determined whether the current value is inside, below or above the range at block 4025. Depending on the result, a generator is created as identified in blocks 4026 and 4027. If below, the current document is stored in the generator as the last output value at block 4031, and the first document is returned from the generator at block 4032.
As discussed above with reference to
In the {range: “full”} case we will offer no guarantees on the output order of the documents. In the other range options, the system according to some implementations will guarantee the sort order on {partitionFieldA, partitionFieldB, densifyField}.
In most cases, the system can be configured to desugar to the same pipeline:
In all examples “unit” is not shown. Without “unit” the value is assumed to be numeric (and will error if a date is encountered). If “unit” is specified, we assume fields are dates and will error if a numeric is encountered.
Generally, the system will not be configured to push any computation to the shards, as all computation will depend on seeing the range of values in the field that is being densified, possibly in each partition. Even if the shard key is the exact set of fields in the “partitionFields” array, the system would still need to see all the documents across the collection in the “full” range case.
If the shard key is exactly or a subset of the partitionFields array, and range is not “full”, then the system could send the desugared _internalDensify stages to the shards. Each shard would be able to generate the documents for the partitions it is holding and send back the full result set. This would result in much more traffic between the shards and the merging shard (in some cases), but would allow for generation in parallel.
In a situation where all of the data for a given partition is on one shard, the logic that has to be done on that shard is the same as the logic in the general case. In some implementations, any special code is not required in the sharded case apart from moving the sort and the _internalDensify stage into the “shards part” of the pipeline.
The general idea behind this approach is to use $lookup cross product to generate the entire domain of documents. Each side of the cross product will be generated either by a $group or a new stage that willgenerate documents in a range.
New stage: $_internalGenerateBounds
This stage will only be generated from $densify and will not be externally accessible. The code block is only to convey the structure of the stage.
{$_internalGenerateBounds: {field: <fieldName>, unit: <unit>, step: <stepSize>}
There may be two versions of this stage, or possibly two separate stages (GenerateBoundsFromDocs, GenerateBoundsFromArgs).
GenerateBoundsFromArgs: When built it will take in a field name (the field to densify) and the step size, and the range to generate. It will then generate all the documents in that range, with only the densification field present. The expectation is that it will be put into a $lookup for a cross product with all of the unique partitions. This stage could be replaced by a $literal stage or the $search project. In this case the system could implement a $range operator that returned documents in order over a range.
GenerateBoundsFromDocs: This version will always be preceded by a $group that calculates a “min” field and a “max” field, stored in two different UUID (generated) field names. For each partition (if present), it will create the documents between the min and max from the preceding group. Depending on if the partition fields are present in the group, this may or may not need a cross-join to finish generating the domain. See examples below.
Either way we will expect the stage to generate documents of the form {fieldToDensify: <value>, p1: <value>, p2: <value> . . . }
$lookup enhancement: Full join
In order to support any case with a “step” value, the system would need to implement a full join. Users may want a “step” without sufficient granularity such that the step values will have a corresponding value from the source collection (and vice-versa).
One of the cons of this approach is that $lookup is slow, but one join implementation is a nested loop join. It is possible that if a hash join was used, performance could be improved.
New stage: $_internalMinMax
This stage will only be generated from $densify and, in some implementations, will not be externally accessible. The code block is only to convey the structure of the stage.
$_internalMinMax: {field: <fieldName>}
When built it will take in a field name (the field to densify) and a pointer to some shared state with $_internalDensify. It will find the min and max values of the field, and populate those values in the shared state. It will pass on the documents as it gets them, making no changes to those documents.
New stage: $_internalDensify
This stage will only be generated from $densify and will not be externally accessible. The code block is only to convey the structure of the stage.
{$_internalDensify: {field: <fieldName>, min: <value>, unit: <unit>max: <value>, partitionFields: [<field>, . . . ], step: <value>, range: <rangeSpec>}
When built internalDensify will be populated with all of the information needed to generate the new documents. This will be done either on construction, or by an $_internalMinMax stage that precedes it in the pipeline (depending on the ‘range’ argument of the original $densify). This stage will also assume that it is preceded by a ‘$sort’ stage, and therefore the input is sorted on ‘{partitionFieldA: 1, <other partition fields>, field: 1}’.
The stage will use the existing PartitionIterator to iterate over the documents in each partition. For each document it will create the missing values between itself and the previous document (or the minimum value for the first document) before outputting the current document without any changes.
As part of this work the partition iterator will be modified to accept multiple partition expressions. We could also add a cacheless mode for the partition iterator, but it wouldn't be that much of an optimization as we expect to need to cache at most the current plus the previous document (implementation dependent)
Desugaring will depend on the existence of partitionFields and the range value specified. Note that none of the below examples include a “unit” field, but “unit” will appear alongside field whenever it is a date type.
The following describes several implementation options that can be used with various embodiments described herein. Such options and implementations may be used alone or in combination with any other options.
A time-series collection is implemented as a kind of writable non-materialized view on an underlying collection holding a more efficient representation of the time-series data. The creation of a time-series collection mydb.X results in the creation of a collection mydb.system.buckets.X as well. Additionally, if the database has no views yet, a collection mydb.system.views is created. Like other system collections, users should not directly interact with these: they perform all insert and query operations on their time-series collection mydb.X.
Creating a time-series collection X with options as described above, consists of these steps:
The global bucket catalog has an in-memory thread-safe ordered map indexed by a tuple <nss, metadata, _id>. For each bucket it contains:
The catalog also has an “idle bucket” queue with references to all buckets that do not have writers. This queue allows expiring entries in the bucket catalog if their total size exceeds some (big) threshold. On step-down this queue is flushed, so the bucket catalog is empty.
The catalog serves two main purposes:
One goal is to transform an insert of a measurement into a time-series collection (which is much like a writable non-materialized view) into an insert or update on the bucket collection. The measurement is first added to the in-memory bucket's vector of measurements to be inserted, then one writer takes all pending inserts, turns them into an upsert and commits, repeating until all writes are committed.
For inserting a measurement X with metadata M with time field value T in a time-series collection C, atomically do the following:
If the above action returns with number of uncommitted measurements equal to 1, the operation that did the insert has become a committer and must perform the following actions until the number of uncommitted measurements is zero:
See below for caveats on updating buckets.
When the number of uncommitted measurements is greater than 1, the insert operation is a waiter, and must wait until the number of committed measurements is such that it includes the insert. In that case, the waiter does the following:
The drop command will automatically drop both the time-series collection and the underlying buckets collection. We'll prohibit users from dropping the system.views collection if there are time-series collections. Interruption of the dropDatabase command can cause an orphaned buckets collection, as can some other scenarios such as interrupted initial sync, or a partial restore. Retrying the operation will fix that.
A special-purpose computer system can be specially configured as disclosed herein. According to one embodiment the special-purpose computer system is configured to perform any of the described operations and/or algorithms. The operations and/or algorithms described herein can also be encoded as software executing on hardware that defines a processing component, that can define portions of a special purpose computer, reside on an individual special-purpose computer, and/or reside on multiple special-purpose computers.
Computer system 800 may also include one or more input/output (I/O) devices 802-804, for example, a keyboard, mouse, trackball, microphone, touch screen, a printing device, display screen, speaker, etc. Storage 812 typically includes a computer readable and writeable nonvolatile recording medium in which computer executable instructions are stored that define a program to be executed by the processor or information stored on or in the medium to be processed by the program.
The medium can, for example, be a disk 902 or flash memory as shown in
Referring again to
The computer system may include specially-programmed, special-purpose hardware, for example, an application-specific integrated circuit (ASIC). Aspects of the invention can be implemented in software, hardware or firmware, or any combination thereof. Although computer system 800 is shown by way of example, as one type of computer system upon which various aspects of the invention can be practiced, it should be appreciated that aspects of the invention are not limited to being implemented on the computer system as shown in
It should be appreciated that the invention is not limited to executing on any particular system or group of systems. Also, it should be appreciated that the invention is not limited to any particular distributed architecture, network, or communication protocol.
Various embodiments of the invention can be programmed using an object-oriented programming language, such as Java, C++, Ada, or C# (C-Sharp). Other programming languages may also be used. Alternatively, functional, scripting, and/or logical programming languages can be used. Various aspects of the invention can be implemented in a non-programmed environment (e.g., documents created in HTML, XML or other format that, when viewed in a window of a browser program, render aspects of a graphical-user interface (GUI) or perform other functions). The system libraries of the programming languages are incorporated herein by reference. Various aspects of the invention can be implemented as programmed or non-programmed elements, or any combination thereof.
A distributed system according to various aspects may include one or more specially configured special-purpose computer systems distributed among a network such as, for example, the Internet. Such systems may cooperate to perform functions related to hosting a partitioned database, managing database metadata, monitoring distribution of database partitions, monitoring size of partitions, splitting partitions as necessary, migrating partitions as necessary, identifying sequentially keyed collections, optimizing migration, splitting, and rebalancing for collections with sequential keying architectures.
Having thus described several aspects and embodiments of this invention, it is to be appreciated that various alterations, modifications and improvements will readily occur to those skilled in the art. Such alterations, modifications, and improvements are intended to be part of this disclosure and are intended to be within the spirit and scope of the invention. Accordingly, the foregoing description is by way of example only.
Use of ordinal terms such as “first,” “second,” “third,” “a,” “b,” “c,” etc., in the claims to modify or otherwise identify a claim element does not by itself connote any priority, precedence, or order of one claim element over another or the temporal order in which acts of a method are performed, but are used merely as labels to distinguish one claim element having a certain name from another element having a same name (but for use of the ordinal term) to distinguish the claim elements.
This application is a Non-Provisional of Provisional (35 USC 119(e)) U.S. Application Ser. No. 63/220,332, filed Jul. 9, 2021, entitled “SYSTEMS AND METHOD FOR PROCESSING TIMESERIES DATA”, which is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63220332 | Jul 2021 | US |
