Patent Application
20200097446
The field relates generally to data processing and, more particularly, to data valuation.
Data is a valuable resource for an enterprise. Typically, data received and generated by an enterprise is stored in a data lake maintained by the enterprise. A data lake is typically considered to be a functionally centralized data storage system for unstructured and structured data. However, there are relatively few, if any, industry standards, tools, or technologies that can assist in quantifying the actual value of data in real-time.
Embodiments of the invention provide techniques for metadata-based data valuation. For example, in one embodiment, a method comprises the following steps. At least one application data set stored in a data repository is obtained. The application data set is analyzed to generate at least one metadata node. The metadata node is combined with at least one other related node to form a hierarchical data structure. One or more valuation algorithms are executed against the hierarchical data structure to calculate a value for the data set represented in the hierarchical data structure.
In another embodiment, a method comprises the following steps. At least one application data set stored in a data repository is obtained. The application data set contains data generated by a plurality of application program types comprising: a source type, an intermediate type, and a destination type, wherein at least one source type application generates source data, at least one destination type application generates end-user deliverable data, and at least one intermediate type application generates driver data (e.g., intermediate data that helps drive analytic results) in between the source data and the end-user deliverable data. At least a portion of the source data generated by the source type application is analyzed to generate one or more source metadata attributes. At least a portion of the driver data generated by the intermediate type application is analyzed to generate one or more driver metadata attributes. At least a portion of the end-user deliverable data generated by the destination type application is analyzed to generate one or more end-user deliverable metadata attributes. A metadata hierarchical structure is formed comprising a source level of valuation nodes, a driver level of valuation nodes, and an end-user level of valuation nodes. The one or more source metadata attributes populate the source level valuation nodes, the one or more driver metadata attributes populate the driver level valuation nodes, and the end-user deliverable metadata attributes populate the end-user level valuation nodes. One or more source level valuation nodes point to one or more driver level valuation nodes, and one or more driver level valuation nodes point to one or more end-user level valuation nodes. Values are assigned to the valuation nodes at each level of the metadata hierarchical structure, and a data valuation is determined for at least a portion of the application data set stored in the data repository based on the values assigned to at least a subset of the valuation nodes of the metadata hierarchical structure.
Advantageously, illustrative embodiments provide a data value analysis model using a metadata-based approach. In the illustrative approach, data value is quantified in real-time by creating and organizing metadata using a pre-defined model, applying various analytics to quantify real-time data value, and normalizing the analyses using data management architecture and algorithms.
These and other features and advantages of the invention will become more readily apparent from the accompanying drawings and the following detailed description.
Illustrative embodiments may be described herein with reference to exemplary cloud infrastructure, data repositories, data centers, data processing systems, computing systems, data storage systems and associated servers, computers, storage units and devices and other processing devices. It is to be appreciated, however, that embodiments of the invention are not restricted to use with the particular illustrative system and device configurations shown. Moreover, the phrases “cloud infrastructure,” “data repository,” “data center,” “data processing system,” “computing system,” “data storage system,” “data lake,” and the like as used herein are intended to be broadly construed, so as to encompass, for example, private and/or public cloud computing or storage systems, as well as other types of systems comprising distributed virtual infrastructure. However, a given embodiment may more generally comprise any arrangement of one or more processing devices.
As used herein, the following terms and phrases have the following illustrative meanings:
“metadata” illustratively refers to data that describes or defines data;
“valuation” illustratively refers to a computation and/or estimation of something's worth or value; in this case, data valuation is a computation and/or estimation of the value of a data set for a given context;
“context” illustratively refers to time, place, surroundings, circumstances, environment, background, settings, and/or the like, that determine, specify, and/or clarify something; in this case, for example, context is used to determine a value of data;
“structured data” illustratively refers to data that resides in fixed fields within a document, record or file, e.g., data contained in relational databases and spreadsheets; and
“unstructured data” illustratively refers to data that is not considered structured data (in which case, some “semi-structured” data asset may also be considered unstructured data).
As mentioned above, there are relatively few, if any, current methodologies that can assist in quantifying the actual value of data in real-time. All approaches currently available to perform any quantifiable value analysis on data revolve around the actual data itself. For many enterprises, enormous amounts of data (of varying complexity and variety) are being generated every second. As such, it is almost impossible to keep up with the speed of ingest by continually running valuation algorithms against the content itself. This results in a lack of capability to get timely valuation results. Furthermore, it is realized that performing valuation approaches against all data volumes will create an enormous compute load. Also, given that valuation algorithms focusing on parsing content would need to continually access that content, these algorithms will be in competition with other production activities (e.g., standard reads and writes against the content). It is highly likely that the valuation algorithms will slow down the performance of the production applications. Still further, often times the users that wish to calculate value do not have full access to the content or encryption keys to access the content. This prevents specific business users from running valuation algorithms. Lastly, valuation algorithms that focus on specific content are not relevant and/or not portable to other enterprise data lakes containing different content that is specific or applicable to different vertical markets.
Embodiments of the invention overcome these and other drawbacks of existing approaches by quantifying data value via metadata as opposed to the data itself. This results in numerous advantages over valuation via production content. Advantages include, but are not limited to, a higher speed of analysis, less computing needs, quick addition or deletion or modification (update) of a metadata attribute, very infrequent access to the direct data itself, fast value calculations, and the ability to add value-specific locks to the data.
In particular, embodiments of the invention analyze an application data set to generate at least one metadata node, which is combined with at least one other related node to form a hierarchical data structure, e.g., a graph. One or more valuation algorithms are executed against the graph to calculate a value for the data set represented in the graph. As will be explained in further embodiments, the graph can have multiple metadata-based valuation nodes at multiple interrelated levels depending on what type of application program (application) yielded the application data.
In an illustrative embodiment, the generation of valuation metadata starts with classifying three types of applications that are typically involved in analytic activity. Source type applications are applications that generate “raw” or “source” data (e.g., SAP® applications). Intermediate type applications are applications that analyze raw data and create “driver” data which is intermediate data that helps drive analytic results/visualizations (e.g., Hive' applications). These intermediate type applications can also recursively analyze driver data and generate additional intermediate data. Destination type applications are applications that analyze raw or driver data and create end-user reports and/or visualizations (e.g., Tableaux applications). Assume these three types of applications are contributing content (e.g., application data 115) into a data lake (e.g., data lake 110).
One of the main concepts of the metadata valuation approach described herein is the creation of valuation metadata that describes the content created by each one of the three application classes mentioned above.
At the bottom-most level, in one embodiment, one source valuation node is created and maintained for each data source generated by a primary or source application.
Accordingly, the end-user valuation nodes (at level 502) contain metadata describing the data assets at the top of the chain which are the end results of all calculations to derive some business value, for example, but not limited to, applications, reports, dashboards, etc. The driver valuation nodes (at level 402) contain metadata describing all data assets that are results of calculations out of source data assets which are used to reach the final outcome, for example, but not limited to, data warehouse tables, fact tables, etc. The source valuation nodes (at level 302) contain metadata about assets which store the data in its native form as generated by transactional and operational systems, for example, but not limited to, ERP (enterprise resource planning) data, log files, etc.
We now describe how each level of valuation node in the hierarchical data structure 500 can be populated.
The bottom-most valuation nodes (source metadata) are populated by analyzing the raw data in a data lake (source data). It is not necessary to parse all of the data to create these valuation nodes (but certainly this approach can be done initially).
For structured data (e.g., an Oracle® database), the source valuation node is created by gathering such metadata as the definitions of the tables 632, definitions of views 634, and more granular information (definitions) about the fields 636.
For unstructured data 638 (e.g., Hadoop Distributed File System or HDFS), common terms or occurrences can be extracted from the unstructured store and placed into the metadata node. In addition, pointers 640 are stored in association with the source node that create a “data chain” to driver and end-user metadata nodes. The pointers 640 can be implemented, for example, as doubly-linked lists that allow navigation in a top-down or bottom-up method. These definitions are essentially “technical metadata” descriptions that are stored inside a valuation node.
Finally, the source metadata node has a set of assigned values 642 (e.g., [0 . . . N]) that are calculated using the one or more valuation algorithms described below. These assigned values 642 allow the metadata to describe some aspect of the actual data's value.
Driver nodes can be created and populated using similar techniques. When ETL or analytic activity occurs against source data sets, the results cause the population of identical fields (as those shown in
End-user valuation nodes are created when a visualization or report tool (e.g., Tableau® applications) creates a user-visible asset from analyzing some combination of source or driver data assets. During this operation, end-user valuation nodes are created and populated with “business metadata” (as opposed to the “technical metadata” created for source and driver nodes). This “business metadata” contains tables, fields, views, and terms which propagate to the end-user level and are consumed in some fashion by the end user.
Once the full hierarchy of valuation nodes has been created (e.g., as illustratively shown in
The first approach navigates the hierarchical data structure from the top down (i.e., from end-user valuation nodes to source valuation nodes) and assigns value to metadata nodes during that process.
Once the metadata topology (e.g., 500 in
The next step (step 704) is to traverse the topology to find all the drivers and/or the source metadata nodes in the chain. For each of these next level nodes, steps 706 through 710 examine the metadata attributes, compare the metadata attributes with the end-user (or root) metadata node attributes (business metadata), and assign weights to each node in accordance to how much of a contribution each node made to the final end-user value. Now that the weights of each node contributing to the end-user data value are known, the algorithm calculates the data value for each node by using the formula: Data_Value=Weight×Root_Node_Data_Value.
These steps are repeated for the source data nodes if the above steps were calculated for driver nodes. In this case, the driver data node is given as a root node to this same algorithm.
Another way to calculate valuation scores is to count the dependencies in the data chain and use that number to assign value. This is a bottom-up approach (i.e., from source valuation nodes to end-user valuation nodes).
By traversing (step 802) all source data nodes to their higher level nodes (which could be driver nodes or end-user nodes), the algorithm can count or otherwise calculate (step 804) the number of times each source node has participated in contributing to the higher level nodes in the data chain (contributing factor). The algorithm now assigns (step 806) the value of each source node based on the contributing factor to the higher level nodes. The formula to calculate the source node value is: Source_Data_Value=(Contributing_Factor/#_of_Higher_Level_Nodes)*100.
For each higher level node (driver or end-user), the algorithm applies aggregation or otherwise calculates (step 808) the data value by using the formula: Data_Value=Sum of (Source_Data_Value).
A third approach to assigning value is to use input provided directly by the end user as to the perceived or actual value of a top-level end-user asset.
Lastly, one algorithm can run all three approaches (algorithms 700, 800 and 900) to calculate an overall data value score, as well as adding in other optional approaches. This composite algorithm 1000 is illustrated in
When using the multiple valuation approaches, each node stores a different valuation result. Note that the ability to store multiple valuation scores in one node is part of the data structure depicted in
One of the primary benefits of the metadata node approach is that the valuation metadata is kept separate from the actual data. This allows for segmentation of the users accessing the data (e.g., data scientists) and the administrators that calculate value.
Valuation algorithms that focus solely on content not only have a scalability problem (ability to process all data in a reasonable amount of time), but they have a portability problem as well. The metadata node approach is neutral to any enterprise vertical market (e.g., medical, oil & gas, retail) etc.
Once a full map of valuation metadata nodes is up and running, over time it is possible to identify candidates for pruning. Likely candidates for pruning include driver data sets that were used to calculate intermediate results but are no longer being used. By navigating all valuation nodes and identifying low-value data sets, a list of candidate data sets can be provided to an administrator for pruning (i.e., deletion).
However, an end user may consider an end-user data set as important, even if it is identified by the system as a low-value data set. In such case, embodiments of the invention allow the end user to place a “lock” on that data set and any other data sets (driver or source) that were part of it in a data chain. By setting a “lock” flag on the top-most valuation node, a set of cascading locks can be set on all intermediate and source nodes that were involved in the generation of the end-user data. This prevents pruning low-value data in the case where a critical user has identified a high-value end-user data set.
Above layer 1102 is a technical metadata hub 1104, which maintains a technical metadata store (i.e., contains the source and driver valuation nodes). In addition, the metadata lineage can be kept separately or within the technical metadata store. Lastly, all of this metadata is indexed and a query application programming interface (API) allows access to the metadata. Alongside the technical metadata hub 1104 is business metadata hub 1106 where end-user valuation nodes are kept in the business metadata store. Hub 1106 also contains the valuation algorithms, which can use tools such as Spring XD® to perform queries and shuffle data between the technical and business metadata hubs.
Data valuation techniques can be used across these layers to perform different forms of analytics for calculating value. As shown, for example, data mining, machine learning, and regression analysis are analytics 1110 that access the metadata hubs 1104 and 1106 through a metadata management center (interface or portal) 1108 in order to determine data value 1112 for data sets stored in data lake 1102.
Further, illustrative embodiments allow a user (e.g., enterprise administrator) to execute an operation, through system 1100, such as Value(data), where Value( ) is a function written in a particular programming language and exposed as one or more REST APIs, and the object ‘data’ is the input on which valuation is requested. It is also to be appreciated that value of a data could depend on the time (instance or period) and where it is requested. In such an embodiment, time and place are considered attributes of the super class node in the hierarchical data structure. These attributes could also contribute in the weight calculations described above. Still further, it is to be understood that hierarchical data structures such as those described herein can have relationships assigned. The assigned relationships would define relations (by designated name) amongst source, driver and end-user nodes. The system is also configured to enable a user to define new types of nodes other than the ones (source, driver, end-user) defined for the current hierarchical data structure.
Advantageously, embodiments of the invention therefore enable the entire process of data valuation by defining and loading a metadata model into a distributed system (e.g., a distributed processing platform as will be illustrated below in the context of
We now provide an exemplary use case in the context of
In this use case, assume that John (a salesperson for an enterprise that provides data storage infrastructure for customers) will visit a particular customer site next month and that he wishes to read information beforehand to decide the strategy with that customer site, such as the installed base at that site, revenue generated, customer satisfaction, etc. The data to provide these details includes: (i) Install Base Data to see the products installed at the site; (ii) Service Request Data to see the service tickets and customer satisfaction; and (iii) Finance Data to see the revenue generated from Customer Site. This data is made available to John in a Customer Account report 1202 as depicted in visualization 1200.
This report 1202 carries five metadata attributes:
The report 1202 also points to Install Base Data (data set) 1214 and Service Request (data set) 1216. Using the metadata-based valuation approach according to one or more illustrative embodiments described herein, a metadata model 1300 representing report 1202 is shown in
As an example of a processing platform on which a metadata-based data valuation system and its corresponding environment (e.g., 100 in
It is to be appreciated that the data valuation methodologies described herein may be executed in one such processing device 1402, or executed in a distributed manner across two or more such processing devices 1402. The cloud infrastructure environment may also be executed in a distributed manner across two or more such processing devices 1402. The various functionalities described herein may be executed on the same processing devices, separate processing devices, or some combination of separate and the same (overlapping) processing devices. It is to be further appreciated that a server, a client device, a computing device or any other processing platform element may be viewed as an example of what is more generally referred to herein as a “processing device.” As illustrated in
The processing device 1402-1 in the processing platform 1400 comprises a processor 1410 coupled to a memory 1412. The processor 1410 may comprise a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array
(FPGA) or other type of processing circuitry, as well as portions or combinations of such circuitry elements. Components of systems as disclosed herein can be implemented at least in part in the form of one or more software programs stored in memory and executed by a processor of a processing device such as processor 1410. Memory 1412 (or other storage device) having such program code embodied therein is an example of what is more generally referred to herein as a processor-readable storage medium. Articles of manufacture comprising such processor-readable storage media are considered embodiments of the invention. A given such article of manufacture may comprise, for example, a storage device such as a storage disk, a storage array or an integrated circuit containing memory. The term “article of manufacture” as used herein should be understood to exclude transitory, propagating signals.
Furthermore, memory 1412 may comprise electronic memory such as random access memory (RAM), read-only memory (ROM) or other types of memory, in any combination. The one or more software programs when executed by a processing device such as the processing device 1402-1 causes the device to perform functions associated with one or more of the components/steps of system/methodologies in
Processing device 1402-1 also includes network interface circuitry 1414, which is used to interface the device with the network 1404 and other system components. Such circuitry may comprise conventional transceivers of a type well known in the art.
The other processing devices 1402 (1402-2, 1402-3, . . . 1402-N) of the processing platform 1400 are assumed to be configured in a manner similar to that shown for computing device 1402-1 in the figure.
The processing platform 1400 shown in
Also, numerous other arrangements of servers, clients, computers, storage devices or other components are possible in processing platform 1400. Such components can communicate with other elements of the processing platform 1400 over any type of network, such as a wide area network (WAN), a local area network (LAN), a satellite network, a telephone or cable network, or various portions or combinations of these and other types of networks.
Furthermore, it is to be appreciated that the processing platform 1400 of
As is known, virtual machines are logical processing elements that may be instantiated on one or more physical processing elements (e.g., servers, computers, processing devices). That is, a “virtual machine” generally refers to a software implementation of a machine (i.e., a computer) that executes programs like a physical machine. Thus, different virtual machines can run different operating systems and multiple applications on the same physical computer. Virtualization is implemented by the hypervisor which is directly inserted on top of the computer hardware in order to allocate hardware resources of the physical computer dynamically and transparently. The hypervisor affords the ability for multiple operating systems to run concurrently on a single physical computer and share hardware resources with each other.
An example of a commercially available hypervisor platform that may be used to implement portions of the processing platform 1400 in one or more embodiments of the invention is the VMware vSphere (VMware Inc. of Palo Alto, Calif.) which may have an associated virtual infrastructure management system such as the VMware vCenter. The underlying physical infrastructure may comprise one or more distributed processing platforms that include storage products such as VNX and Symmetrix VMAX (both available from EMC Corporation of Hopkinton, Mass.). A variety of other computing and storage products may be utilized to implement the one or more cloud services that provide the functionality and features described herein.
It was noted above that portions of the data valuation system and cloud environment may be implemented using one or more processing platforms. A given such processing platform comprises at least one processing device comprising a processor coupled to a memory, and the processing device may be implemented at least in part utilizing one or more virtual machines, containers or other virtualization infrastructure. By way of example, such containers may be Docker containers or other types of containers.
It should again be emphasized that the above-described embodiments of the invention are presented for purposes of illustration only. Many variations may be made in the particular arrangements shown. For example, although described in the context of particular system and device configurations, the techniques are applicable to a wide variety of other types of data processing systems, processing devices and distributed virtual infrastructure arrangements. In addition, any simplifying assumptions made above in the course of describing the illustrative embodiments should also be viewed as exemplary rather than as requirements or limitations of the invention. Numerous other alternative embodiments within the scope of the appended claims will be readily apparent to those skilled in the art.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15072557 | Mar 2016 | US |
| Child | 16697372 | US |
