Patent Application
20140101177
This document generally relates to systems and methods for use with data profiling. More specifically, this document relates methods and system for performing data profiling in-memory.
Data profiling is the process of analyzing source data and gathering information inherent to its internal metadata, data structures, content, relationships, and derivation rules. Profiling allows the user to validate data quality and track anomalies, but also to discover metadata. Currently, data profiling requires that a system first extract data from a data source and place it into a memory. The process of transferring this data can take a lot of time. The result is that data profiling is a slow process, especially as the amount of data grows, including performing table-wide or even cross-table profiling.
The present disclosure is illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements and in which:
The description that follows includes illustrative systems, methods, techniques, instruction sequences, and computing machine program products that embody illustrative embodiments. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments of the inventive subject matter. It will be evident, however, to those skilled in the art that embodiments of the inventive subject matter may be practiced without these specific details. In general, well-known instruction instances, protocols, structures, and techniques have not been shown in detail.
In an example embodiment, data profiling is performed in-memory, meaning that the data profiling occurs in the memory that is storing the source data. This reduces and possibly eliminates any need to transfer the data to another memory prior to profiling it. Additionally, this allows for the flexibility of allowing multiple clusters to work in parallel to perform the data profiling, further improving the speed of such profiling.
Here, an in-memory database system 100 may be coupled to one or more client applications 102a, 102b. The client applications 102a, 102b may communicate with the in-memory database system 100 through a number of different protocols, including Structured Query Language (SQL), Multidimensional Expressions (MDX), Hypertext Transfer Protocol (HTTP), Representational State Transfer (REST), and Hypertext Markup Language (HTML). In an example embodiment, unique SQL commands are created allowing the client application 102a, 102b to request that a particular data attribute calculation be performed by the in-memory database system 100. In essence, this allows the client application 102a, 102b to request that data profiling be performed without actually performing any of the data profiling calculations itself.
Data profiling computers different attributes from the data, such as minimum, maximum, mean, mode, percentile, standard deviation, frequency, variation, count, and sum. Additional metadata could include data type, length, discrete values, uniqueness, occurrence of null values, representation, and duplicates. For purposes of this document, all of these data profiling aspects may be performed as what is termed “data attribute calculations”. Of course, one of ordinary skill in the art will recognize that the data attribute calculations could also include other calculations not specifically described here.
Also depicted is a studio 104, used to perform modeling by accessing the in-memory database system 100.
The in-memory database system 100 may comprise a number of different components, including index server 106, XS engine 108, statistics server 110, preprocessor server 112, and name server 114. These components may operate on a single computing device, or may be spread among multiple computing devices (e.g., separate servers).
In an example embodiment, the index server 106 contains the actual data and the engines for processing the data. It also coordinates and uses all the other servers. In an example embodiment, new procedures are defined in the index server 106 to perform various data attribute calculations when requested by a client application 102a, 102b. In a further example embodiment, these new procedures are designed to receive and interpret SQL commands from the client applications 102a, 102b.
The name server 114 holds information about the database topology. This is used in a distributed system with instances of the database on different hosts. The name server 114 knows where the components are running and which data is located on which server.
The statistics server 110 collects information about status, performance, and resource consumption from all the other server components. The statistics server 110 can be accessed from the studio 104 to obtain the status of various alert monitors.
The preprocessor server 112 is used for analyzing text data and extracting the information on which the text search capabilities are based.
The XS engine 108 allows clients to connect to the database system 100 using HTTP.
The client requests can be analyzed and executed by a set of components summarized as request processing and execution control 206. The SQL processor 208 checks the syntax and semantics of the client SQL statements and generates a logical execution plan. Multidimensional expressions (MDX) is a language for querying and manipulating multidimensional data stored in OLAP cubes. As such, an MDX engine 210 is provided to allow for the parsing and executing of MDX commands. A planning engine 212 allows financial planning applications to execute basic planning operations in the database layer. One such operation is to create a new version of a dataset as a copy of an existing dataset, while applying filters and transformations.
A calc engine 214 implements the various SQL script and planning operations. The calc engine 214 creates a logical execution plan for calculation models derived from SQL script, MDX, planning, and domain-specific models. This logical execution plan may include, for example, breaking up a model into operations that can be processed in parallel.
The data is stored in relational stores 216, which implement a relational database in main memory.
Each SQL statement may be processed in the context of a transaction. New sessions are implicitly assigned to a new transaction. The transaction manager 218 coordinates database transactions, controls transactional isolation, and keeps track of running and closed transactions. When a transaction is committed or rolled back, the transaction manager 218 informs the involved engines about this event so they can execute necessary actions. The transaction manager 218 also cooperates with a persistence layer 220 to achieve atomic and durable transactions.
An authorization manager 222 is invoked by other database system components to check whether the user has the required privileges to execute the requested operations. The database system allows for the granting of privileges to users or roles. A privilege grants the right to perform a specified operation on a specified object.
The persistence layer 220 ensures that the database is restored to the most recent committed state after a restart and that transactions are either completely executed or completely undone. To achieve this goal in an efficient way, the persistence layer 220 uses a combination of write-ahead logs, shadow paging, and save points. The persistence layer 220 also offers a page management interface 224 for writing and reading data to a separate disk storage 226, and also contains a logger 228 that manages the transaction log. Log entries can be written implicitly by the persistence layer 220 when data is written via the persistence interface or explicitly by using a log interface.
L infrastructure 320 includes a number of components to aid in the running of L procedures, including L-runtime (system mode) 332, L compiler 324, and L-runtime (User mode) 326.
As described briefly above, in an example embodiment, a technique is provided allowing data profiling to be performed in-memory, providing high quality calculations in little time. In this example embodiment, all relevant data is kept in main memory, so that read operations can be executed without requiring disk input and output. Additionally, data is stored in columns. This allows for the use of high compression techniques, high performance reading contiguous data for column operations, and parallelization of the column processing. In a column store, data is already vertically partitioned. This means that operations on different columns can be easily processed in parallel. If multiple column need to be searched or aggregated, each of these operations can be assigned to a different processor core and/or computer cluster. In addition, the execution of operations within a single column can be further parallelized by dividing the column into multiple sections that are processed by different processor cores.
In an example embodiment, specialized client software may be run by one or more client machines interfacing with the database system. This specialized client software may be known as an information steward, and may combine the functionality for data profiling, metadata management, and data quality monitoring in a unified framework. The solution provides tools that help companies perform data assessments, view data lineage, and improve data traceability to support governance initiatives.
In another example embodiments, business objects data services software i run on the client. In such an embodiment, a single enterprise-class solution is used for data integration, data quality, data profiling, and text data processing that allows users to integrate, transform, improve, and deliver trusted data to critical business processes.
Since both types of software run data profiling and are located on clients, the various data profiling techniques described herein can be run on either type of software. More particularly, SQL or other database control statements can be implemented in either business objects data services software or information steward software, which then can be used to interface with an index server containing data attribute calculations. Therefore, while in the past such software would perform the data profiling itself, causing the previously mentioned delays, now the software can leverage the strength of the index server to greatly improve performance of data profiling techniques.
Because the tests cover the replacement of the information steward profiling engine, it focuses on the same profiling attributes. Other profiling attributes, of course, can be derived using the same techniques.
In an example embodiment, any data attribute calculation that can be calculated using a SQL query can be implemented in a way to allow the information steward to generate such a SQL query. This would include data attribute calculations such as min, max, count, null counts, blanks, zeroes, and string length. For other data attribute calculations that may not be computed using pure SQL script, procedures and functions can be utilized. Examples of such other data attribute calculations include median and pattern.
SQL script is a high-level language used to control and manipulate the dataflow, allowing a developer to create high performance and concurrent execution code. When data transformation and complex data manipulation are necessary, extra support can be provided using the scalar language L. L is natively compiled, allowing the developer to implement high performance operators called from SQL script.
It should be noted that implementations using alternative programming languages, such as C++ and R, are also foreseen.
As described above, the procedures used in the index server to perform the data attribute calculation may be written in SQL script and L language. What follows are example procedures, and these examples are not intended to be limiting:
Example Table types
Embodiments may also, for example, be deployed by Software-as-a-Service (SaaS), Application Service Provider (ASP), or utility computing providers, in addition to being sold or licensed via traditional channels. The computer may be a server computer, a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), cellular telephone, or any processing device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device. Further, while only a single computer is illustrated, the term “computer” shall also be taken to include any collection of computers that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.
The example computer processing system 800 includes processor 802 (e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both), main memory 804 and static memory 806, which communicate with each other via bus 808. The processing system 800 may further include graphics display 810 (e.g., a plasma display, a liquid crystal display (LCD) or a cathode ray tube (CRT)). The processing system 800 also includes alphanumeric input device 812 (e.g., a keyboard), a user interface (UI) navigation device 814 (e.g., a mouse, touch screen, or the like), a storage unit 816, a signal generation device 818 (e.g., a speaker), and a network interface device 820.
The storage unit 816 includes machine-readable medium 822 on which is stored one or more sets of data structures and instructions 824 (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. The instructions 824 may also reside, completely or at least partially, within the main memory 804 and/or within the processor 802 during execution thereof by the processing system 800, the main memory 804 and the processor 802 also constituting computer-readable, tangible media.
The instructions 824 ma further be transmitted or received over network 826 via a network interface device 820 utilizing any one of a number of well-known transfer protocols (e.g., HTTP).
While the machine-readable medium 822 is shown in an example embodiment to be a single medium, the term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a cenralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions 824. The term “machine-readable medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the computer and that cause the computer to perform any one or more of the methodologies of the present application, or that is capable of storing, encoding or carrying data structures utilized by or associated with such a set of instructions. The term “machine-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media.
While various implementations and exploitations are described, it will be understood that these embodiments are illustrative and that the scope of the claims is not limited to them. In general, techniques for maintaining consistency between data structures may be implemented with facilities consistent with any hardware system or hardware systems defined herein. Many variations, modifications, additions, and improvements are possible.
Plural instances may be provided for components, operations or structures described herein as a single instance. Finally, boundaries between various components, operations, and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of the claims. In general, structures and functionality presented as separate components in the exemplary configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the claims.
While the embodiments are described with reference to various implementations and exploitations, it will be understood that these embodiments are illustrative, and that the scope of claims provided below is not limited to the embodiments described herein. In general, the techniques described herein may be implemented with facilities consistent with any hardware system or hardware systems defined herein. Many variations, modifications, additions, and improvements are possible.
The term “computer readable medium” is used generally to refer to media embodied as non-transitory subject matter, such as main memory, secondary memory, removable storage, hard disks, flash memory, disk drive memory, CD-ROM and other forms of persistent memory. It should be noted that program storage devices, as may be used to describe storage devices contaiing executable computer code for operating various methods, shall not be construed to cover transitory subject matter, such as carrier waves or signals, “Program storage devices” and “computer-readable medium” are terms used generally to refer to media such as main memory, secondary memory, removable storage disks, hard disk drives, and other tangible storage devices or components.
Plural instances may be provided for components, operations, or structures described herein as a single instance. Finally, boundaries between various components, operations, and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of the claims. In general, structures and functionality presented as separate components in the exemplary configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the claims and their equivalents.
