This application is a national phase entry of International Application No. PCT/CN2018/122272 filed on Dec. 20, 2018, which is based on and claims priority to Chinese Patent Application No. 201711384490 filed on Dec. 20, 2017 and entitled “Columnar Storage and Query Method and System for Time Series Data”, which is incorporated herein by reference in its entirety.
The present disclosure relates to the field of data storage technologies, and specifically, to a columnar storage method and a query method and system for time series data.
With the development of computer technology and industrial informatization, the amount of data generated by the industrial field is increasing. Time series data is widely used in the industrial applications and is one of the main data type of industrial data. As the main data type in industrial big data, its storage and query problems have become a key topic in the research on industrial big data.
The time series data is usually generated by a machine sensor at some frequencies, and once a data point is generated, its value is almost unnecessary to be modified. Secondly, the time series data oriented applications mostly require analytical query other than fine-grained insertion, deletion, modification and selection operations. Therefore, in combination with the workload of the time series data insertion and query in industrial applications, using columnar file format is more suitable for storing massive time series data.
Currently, the popular columnar file formats such as Parquet, ORCFile, etc. do not optimize for time series data. When the time series data is stored in such a columnar format, a complicated nested structure is adopted, and data values and timestamps of a group of time series data cannot be stored in one column at the same time, data values and timestamps of the group of time series data cannot be indexed simultaneously in a column of data and thus these problems will slow down the query speed of the time series data and more system resources are consumed.
The present disclosure provides a columnar storage method and system for time series data and a columnar query method and system for time series data that overcome the problems above or at least partially solve the problems above, and can improve the query speed of the time series data.
According to a first aspect of the present disclosure, a columnar storage method for time series data is provided, including:
dividing a time series data into a plurality of pages, wherein each page stores a part of data points of the time series data. That is, all the data points in these pages form a time series; and
Each page consists of two parts, i.e., a page header and a page body. For each page, storing summary information of all the data points in the page as an index into the page header of the page and storing data value information of all the data points in the page in the page body of the page. Based on the technical solution above, the present disclosure can also be improved as follows.
Further, the summary index information in the page header of each page includes the number of all the data points in the page, a maximum value in timestamps corresponding to all the data points, a minimum value in the timestamps corresponding to all the data points, a maximum value in data values of all the data points and a minimum value in the data values of all the data points; detailed data value histogram can also be stored into the page header; and
the data value information of all the data points in the page body of each page includes a timestamp set for all the data points of the page and a data value set for all the data points of the page.
Further, all the timestamps in the timestamp set have the same data type and the data value set can include different data types of data values; the storage method further includes:
performing data encoding on all the timestamps in the timestamp set; and
encoding respective data value using a corresponding data encoding method according to the data type of the respective data value in the data value set.
Further, each page occupies the same fixed storage space.
According to a second aspect of the present disclosure, a query method for time series data is provided, including:
For each page, reading its summary index information to determining whether data points that satisfy the query filter condition are present in the page according to the summary index information in the page.
If not satisfied, then skip the page directly and continue to check the next page. If satisfied partially, then check the data points in the page and add the qualified data into the result set; If satisfied totally, then add the data points in the page into the result set.
Traverse all pages to get the final query result.
Further, the query filter condition includes a given timestamp range and/or a given data value range, and the determining whether data points that satisfy the query filter condition are present in the page according to the summary index information in the page specifically includes:
determining whether a timestamp range corresponding to all the data points in each page and the given timestamp range have an overlapping portion according to the maximum value in timestamps corresponding to all the data points and a minimum value in the timestamps corresponding to all the data points in the summary index information in each page and determining whether the data value range corresponding to all the data points in each page and the given data value range have an overlapping portion according to a maximum value in data values of all the data points and a minimum value in the data values of all the data points in the summary index information in each page; and
determining that the data points that satisfy the query filter condition are present in the page if the timestamp range corresponding to all the data points in each page and the given timestamp range have the overlapping portion and/or the data value range corresponding to all the data points in each page and the set data value range have the overlapping portion, and otherwise, determining that no data points that satisfy the query filter condition are present in the page.
Further, the query method further includes:
decoding data information of all the data points in the page to acquire decoded data information if the data points that satisfy the query filter condition are present in the page, wherein the data information includes a timestamp set for all the data points and a data value set for all the data points in the page.
Further, the screening all the data points that satisfy the query filter condition and adding them into the query result set if the data points that satisfy the query filter condition are present in the page specifically include:
determining whether a timestamp of each data point is within the given timestamp range according to the timestamp of each data point in the timestamp set of the page and determining whether a data value of each data point is within the set data value range according to the data value of each data point in the data value set of the page; and
screening all the data points of which the timestamp is within the given timestamp range and the data value is within the set data value range in the page as the data points that satisfy the query filter condition.
According to a third aspect of the present disclosure, a columnar storage system for time series data is provided, including:
a dividing module configured to divide a column of time series data into a plurality of pages, wherein each page stores a part of data points of the column of time series data and the sum of the data points stored in all the pages is all the data points in the column of time series data; and
a storage module configured to set two parts, i.e., a page header and a page body, for each page, store summary index information of all the data points in the page in the page header of the page and store data value information of all the data points in the page in the page body of the page.
According to a fourth aspect of the present disclosure, a query system for time series data is provided, including:
a reading module configured to read one page of a column of time series data,
a determining module configured to determine whether data points that satisfy a query filter condition are present in the page according to summary index information in the page; and
a screening module configured to screen all the data points that satisfy the query filter condition and add them into a query result set if the determining module determines that the data points that satisfy the query filter condition are present in the page; and call the reading module to read a next page if the determining module determines that no data points that satisfy the query filter condition are present in the page; and
a traversing module configured to traverse all pages to get a final query result set.
By the columnar storage method and system for time series data and the query method and system for time series data provided by the present disclosure, timestamps and specific data values of a group of time series data can be stored in one column and thus the I/O of a magnetic disc is reduced during data query and the query speed of data is accelerated by dividing the data according to pages and establishing summary index information, respectively.
The specific implementations of the present disclosure are further described in detail below with reference to the drawings and embodiments. The following embodiments are intended to illustrate the disclosure, but are not intended to limit the scope of the disclosure.
Referring to
The time series data refers to time series data and is a data column recorded in a time order by the same unified indicator, and each data point of a group of time series data includes a timestamp and a specific data value of the data point. Since only the timestamps of all the data points can be stored as one column, and the data values of all the data points can be stored as another column according to the existing columnar storage method, the query efficiency of data is reduced.
In order to solve the problem above, the embodiment adopts paging storage, specifically, dividing a column of time series data into a plurality of pages, wherein each page stores a part of data points of the column of time series data and the sum of the data points stored in all the pages is all the data points in the column of time series data. Wherein, each page occupies the same fixed storage space. The number of the data points stored in each page is not necessarily the same since each data point has different sizes.
Each page includes two parts, i.e., a page header and a page body, wherein the page header mainly stores summary index information of all the data points in the page and the page body mainly stores data value information of all the data points in the page.
Based on the embodiment above, in an embodiment of the present disclosure, the summary index information in the page header of each page includes the number of all the data points in the page, a maximum value in timestamps corresponding to all the data points, a minimum value in the timestamps corresponding to all the data points, a maximum value in data values of all the data points and a minimum value in the data values of all the data points; and the data value information of all the data points in the page body of each page includes a timestamp set for all the data points of the page and a data value set for all the data points of the page.
In an embodiment of the present disclosure, the timestamps and data values corresponding to all the data points in each page are stored using a respective set, i.e., the timestamps of all the data points in one page are stored in a timestamp set of the page body and the data values of all the data points in one page are stored in a data value set of the page body. In an embodiment of the present disclosure, each timestamp in the timestamp set is in one-to-one correspondence with each data value in the data value set, identification information can be set for each data value and the timestamps in the timestamp set may correspond to the data values in the data value set according to the identification information.
Based on the various embodiments above, in an embodiment of the present disclosure, all the timestamps in the timestamp set have consistent data types and the data value set includes different data types of data values. The storage method further includes: performing data encoding on all the timestamps in the timestamp set using the same data encoding method; and encoding respective data value using a corresponding data encoding method according to the data type of the respective data value in the data value set.
In this embodiment, the timestamp of each data point in the timestamp set of the page body of each page has the same data type and all the timestamps are represented by the same data type. While all the data values in the data value set in each page may include various data types, for example, the data type may be an int type, a long type, a float type, and the like. In this embodiment, data encoding is performed according to different data types in the timestamp set and the data value set, respectively. Data encoding needs to be performed on all the timestamps only using one data encoding method since each timestamp in the timestamp set has the same data type. While the data value is encoded using a corresponding data encoding method according to different data types since all the data values in the data value set may have various different data types. In this embodiment, an optimal encoding method is selected in combination with different data types, and by encoding the data through different data encoding methods, the compression ratio of the data can be improved and the use of the storage space is reduced.
Referring to
The embodiments above describe the columnar storage process of the time series data. When the data query is performed, a first page of the column of time series data is sequentially read, it is determined first whether the data points that satisfy the query filter condition are present in the page according to the summary index information of all the data points in the page stored in the page header of the page and the page is skipped and a next page is read if no data points that satisfy the query filter condition are present in the page. All the data points that satisfy the query filter condition are screened and added into the query result set if the data points that satisfy the query filter condition are present in the page. All the pages are traversed to acquire the data points that satisfy the query filter condition in each page, and finally, all the data points that satisfy the query filter condition in all the pages are returned as the query result.
Based on the various embodiments above, in another embodiment of the present disclosure, the query filter condition includes a given timestamp range and/or a set data value range, and the determining whether data points that satisfy the query filter condition are present in the page according to the summary index information in the page specifically includes: determining whether a timestamp range corresponding to all the data points in each page and the given timestamp range have an overlapping portion according to a maximum value in timestamps corresponding to all the data points and a minimum value in the timestamps corresponding to all the data points in the summary index information in each page and determining whether a data value range corresponding to all the data points in each page and the set data value range have an overlapping portion according to a maximum value in data values of all the data points and a minimum value in the data values of all the data points in the summary index information in each page; and determining that the data points that satisfy the query filter condition are present in the page if the timestamp range corresponding to all the data points in each page and the given timestamp range have the overlapping portion and/or the data value range corresponding to all the data points in each page and the set data value range have the overlapping portion, and otherwise, determining that no data points that satisfy the query filter condition are present in the page. Since the maximum value in the timestamps of all the data points, the minimum value in the timestamps of all the data points, the maximum value in the data values corresponding to all the data points and the minimum value in the data values corresponding to all the data points are included in the summary index information, the query filter condition in this embodiment is the given timestamp range or the set data value range, and can also be the given timestamp range and the set data value range at the same time. The following takes the query filter condition being the given timestamp range as an example. According to the maximum value and the minimum value in the timestamp set of each page, it can be known that the timestamps corresponding to all the data points in the page are within the range between the maximum value and the minimum value and thus it is determined that whether the timestamp range of all the data points in each page and the given timestamp range have an overlapping portion, the page is a page that satisfies the query filter condition if the overlapping portion exists and pages that satisfy the query filter condition in all the pages are filtered out.
If the query filter condition is the set data value range and the query filter condition is the given timestamp range and the set data value range, the query method is similar to that in which the query filter condition is the given timestamp range, and is no longer described herein.
Based on the various embodiments above, in an embodiment of the present disclosure, the query method further includes: decoding data information of all the data points in the page to acquire decoded data information if the data points that satisfy the query filter condition are present in the page, wherein the data information includes the timestamp set for all the data points and the data value set for all the data points in the page.
After all the pages that satisfy the query filter condition are filtered out according the query filter condition, all the data points in the pages that satisfy the query filter condition are decoded to acquire an original timestamp set and a data value set corresponding to all the data points. In this embodiment, the pages that satisfy the query filter condition are filtered out firstly according to the summary index information of each page, and then the data points in the pages that satisfy the query filter condition are decoded without decoding the data points in the pages that do not satisfy the query filter condition, thereby avoiding unnecessary decoding process, saving time, and speeding up the query speed of the entire data.
Based on the various embodiments above, in an embodiment of the present disclosure, the screening all the data points that satisfy the query filter condition and adding them into the query result set if the data points that satisfy the query filter condition are present in the page specifically include: determining whether the timestamp of each data point is within the given timestamp range according to the timestamp of each data point in the timestamp set of the page and determining whether the data value of each data point is within the set data value range according to the data value of each data point in the data value set of the page; and screening all the data points of which the timestamp is within the given timestamp range and/or the data value is within the set data value range in the page as the data points that satisfy the query filter condition.
Still taking the above-mentioned query filter condition being the given timestamp range as an example, it is determined whether the timestamp of each data point is within the given timestamp range according to the timestamp of each data point in the pages that satisfy the query filter condition, this data point is the data point that satisfies the query filter condition if it is within the given timestamp range, the data points that satisfy the query filter condition of all the data points in each page are filtered out, and the data points that satisfy the query filter condition of all the pages are summarized to acquire all the data points that satisfy the query filter condition in the column of the time series data.
If the query filter condition is the set data value range and the query filter condition is the given timestamp range and the set data value range, the query method is similar to that in which the query filter condition is the given timestamp range, and is no longer described herein.
Referring to
The dividing module 31 is configured to divide a column of time series data into a plurality of pages, wherein each page stores a part of data points of the column of time series data and the sum of the data points stored in all the pages is all the data points in the column of time series data.
The storage module 32 is configured to set two parts, i.e., a page header and a page body, for each page, store summary index information of all the data points in the page in the page header of the page and store data value information of all the data points in the page in the page body of the page.
Referring to
The dividing module 31 is configured to divide a column of time series data into a plurality of pages, wherein each page stores a part of data points of the column of time series data and the sum of the data points stored in all the pages is all the data points in the column of time series data.
The storage module 32 is configured to set two parts, i.e., a page header and a page body, for each page, store summary index information of all the data points in the page in the page header of the page and store data value information of all the data points in the page in the page body of the page.
In an embodiment of the present disclosure, the summary index information in the page header of each page includes the number of all the data points in the page, a maximum value in timestamps corresponding to all the data points, a minimum value in the timestamps corresponding to all the data points, a maximum value in the data values of all the data points and a minimum value in the data values of all the data points; and the data value information of all the data points in the page body of each page includes a timestamp set for all the data points of the page and a data value set for all the data points of the page.
All the timestamps in the timestamp set have consistent data types and the data value set includes different data types of data values. Correspondingly, the encoding module 33 is configured to perform data encoding on all the timestamps in the timestamp set using the same data encoding method; and is further configured to encode respective data value using a corresponding data encoding method according to the data type of respective data value in the data value set.
Referring to
The reading module 41 is configured to read one page of a column of time series data.
The determining module 42 is configured to determine whether data points that satisfy a query filter condition are present in the page according to summary index information in the page; and
The screening module 43 is configured to screen all the data points that satisfy the query filter condition and add them into a query result set if the determining module determines that the data points that satisfy the query filter condition are present in the page; and call the reading module to read a next page if the determining module determines that no data points that satisfy the query filter condition are present in the page.
The traversing module 44 is configured to traverse all pages to get a final query result set.
Referring to
In this embodiment, the reading module 41 is configured to read one page of a column of time series data.
The determining module 42 is configured to determine whether data points that satisfy a query filter condition are present in the page according to summary index information in the page.
The screening module 43 is configured to screen all the data points that satisfy the query filter condition and add them into a query result set if the determining module determines that the data points that satisfy the query filter condition are present in the page; and call the reading module 41 to read a next page if the determining module determines that no data points that satisfy the query filter condition are present in the page.
The traversing module 44 is configured to traverse all pages to get a final query result set.
In this embodiment, the query filter condition includes a given timestamp range and/or a set data value range. Correspondingly, the determining module 42 is configured to determine whether a timestamp range corresponding to all the data points in each page and the given timestamp range have an overlapping portion according to a maximum value in timestamps corresponding to all the data points and a minimum value in the timestamps corresponding to all the data points in the summary index information in each page and determine whether a data value range corresponding to all the data points in each page and the set data value range have an overlapping portion according to a maximum value in data values of all the data points and a minimum value in the data values of all the data points in the summary index information in each page. It is determined that the data points that satisfy the query filter condition are present in the page if the timestamp range corresponding to all the data points in each page and the given timestamp range have the overlapping portion and/or the data value range corresponding to all the data points in each page and the set data value range have the overlapping portion, and otherwise, it is determined that no data points that satisfy the query filter condition are present in the page. The decoding module 45 is configured to decode data information of all the data points in each page that satisfies the query filter condition to acquire decoded data information, wherein the data information includes a timestamp set for all the data points and a data value set for all the data points in the page.
The determining module 42 is further configured to determine whether the timestamp of each data point is within the given timestamp range according to the timestamp of each data point in the page and determine whether the data value of each data point is within the set data value range according to the data value of each data point in each page, determine that the data point satisfies the query filter condition if the timestamp of the data point in this page is within the given timestamp range and/or the data value is within the set data value range, and otherwise, determine that the data point does not satisfy the query filter condition.
The present disclosure also provides a non-transitory computer readable storage medium storing computer instructions for enabling a computer to perform the columnar storage method for time series data provided by the corresponding embodiments above, for example, including: dividing a column of time series data into a plurality of pages, wherein each page stores a part of data points of the column of time series data and the sum of the data points stored in all the pages is all the data points in the column of time series data; and setting two parts, i.e., a page header and a page body, for each page, storing summary index information of all the data points in the page in the page header of the page and storing data value information of all the data points in the page in the page body of the page.
In addition, the computer instructions enable the computer to perform the columnar storage method for time series data provided by the corresponding embodiments above, for example, including: reading one page of a column of time series data, determining whether the page satisfies a query filter condition according to summary index information in the page; skipping and reading a next page if the query filter condition is not satisfied; screening all the data points that satisfy the query filter condition and adding them into a query result set if the query filter condition is satisfied; and traversing all pages to get a final query result set.
It can be understood by those of ordinary skill in the art that all or part of the steps of implementing the method embodiments above may be completed by using hardware related to the program instructions. The foregoing program may be stored in a computer readable storage medium, and the program performs foregoing steps including the method embodiments above when executed. The foregoing storage medium includes: a medium that can store program codes, such as a ROM, a RAM, a magnetic disk, or an optical disk.
Through the description of the embodiments above, those skilled in the art can clearly understand that the various embodiments can be implemented by means of software and a necessary general hardware platform, and of course, by hardware. Based on such understanding, the technical solution of the present disclosure or a part of the technical solution, which is essential or contributes to the prior art, may be embodied in the form of a software product, which is stored in a storage medium such as ROM/RAM, magnetic Discs, optical discs, etc., including several instructions to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to perform various embodiments or a part of the methods described in various embodiments.
By the columnar storage method and system for time series data and the query method and system for time series data provided by the present disclosure, timestamps and specific data values of a group of time series data can be stored in one column and thus the I/O of a magnetic disc is reduced during data query; during the query, the data is divided according to pages, summary index information is established respectively, the pages that satisfy a query filter condition are firstly filtered out according to summary index information of each page, and then all the data points in the pages that satisfy the query filter condition are decoded and thus unnecessary decoding is avoided and the query speed of data is accelerated; and the data is divided according to pages, the timestamp and the data value are encoded separately and optimal coding methods can be selected in combination with different types of the data, thereby improving the compression ratio of the data and reducing the use of storage space.
Finally, the methods of the present disclosure are only preferred embodiments and are not intended to limit the scope of the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and scope of the present disclosure are included within the scope of the present disclosure.
Number | Date | Country | Kind |
---|---|---|---|
201711384490.X | Dec 2017 | CN | national |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/CN2018/122272 | 12/20/2018 | WO |
Publishing Document | Publishing Date | Country | Kind |
---|---|---|---|
WO2019/120240 | 6/27/2019 | WO | A |
Number | Name | Date | Kind |
---|---|---|---|
6301578 | Harris | Oct 2001 | B1 |
20120330931 | Nakano | Dec 2012 | A1 |
20170270183 | Kuroda | Sep 2017 | A1 |
Number | Date | Country |
---|---|---|
103003803 | Mar 2013 | CN |
103092905 | May 2013 | CN |
103942343 | Jul 2014 | CN |
105242882 | Jan 2016 | CN |
107256233 | Oct 2017 | CN |
107871022 | Apr 2018 | CN |
Entry |
---|
Li, Q., et al., “Data Structures and Classical Algorithms,” General Higher Education “Eleventh Five-Year” National Planning Textbook Computer Series (2015), pp. 278-289, Tsinghua University Press, Beijing, China. |
“In depth analysis of parquet column storage format,” https://www.infoq.cn/article/in-depth-analysis-of-parquet-column-storage-format (Aug. 6, 2015). |
Number | Date | Country | |
---|---|---|---|
20210157811 A1 | May 2021 | US |