Patent Application
20240013231
An enterprise may store a substantial amount of information. For example, a business might store information about customers, purchase orders, human resources, financial data, etc. The enterprise may use Master Data Management (“MDM”) techniques to ensure the uniformity, accuracy, stewardship, semantic consistency, and accountability of the data. A data validation rule may be a part of that process and act as a gatekeeper to make sure that data are entered into the system in good quality manner and help track the current status of data quality. Note that an enterprise environment may frequently change which can cause challenges and problems requiring continuous updates to existing data validation rules. In some cases, an enterprise may use a simulation designed to verify whether a rule change meets expectations before it is put into production usage. However, such simulations may not clearly indicate the consequences of a rule change, making decisions about implementing rule changes difficult.
It would therefore be desirable to perform data validation for an enterprise in an improved, efficient, and accurate manner.
A data repository may contain electronic records, each electronic record including a record identifier and a set of record characteristics. A data validation rule data store may contain at least one active data validation rule. A data validation server may access the active data validation rule from the data validation rule data store. The server receives an adjustment to the active data validation rule from a user to create an inactive data validation rule. The server may also receive at least one filter condition defining a subset of the electronic records in the data repository. The server can then automatically simulate execution of the active data validation rule on record characteristics of the subset to obtain an active result and automatically simulate execution of the inactive data validation rule to obtain an inactive result. The system then displays the active result and the inactive result to the user.
Some embodiments comprise: means for accessing, by a computer processor of a data validation server, an active data validation rule from a data validation rule data store; means for receiving, from a user via a user interface, an adjustment to the active data validation rule to create an inactive data validation rule; means for receiving, from the user via the user interface, at least one filter condition defining a subset of the electronic records in a data repository, wherein the data repository contains electronic records, each electronic record including a record identifier and a set of record characteristics; means for automatically simulating execution of the active data validation rule on record characteristics of the subset of the electronic records in the data repository to obtain an active result; means for automatically simulating execution of the inactive data validation rule on record characteristics of the subset of the electronic records in the data repository to obtain an inactive result; and means for arranging for the active result and the inactive result to be displayed to the user.
Some technical advantages of some embodiments disclosed herein are improved systems and methods to perform data validation for an enterprise in an improved, efficient, and accurate manner.
In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of embodiments. However, it will be understood by those of ordinary skill in the art that the embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the embodiments.
One or more specific embodiments of the present invention will now be described. In an effort to provide a concise description of these embodiments, all features of an actual implementation may not be described in the specification. It should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another. Moreover, it should be appreciated that such a development effort might be complex and time consuming, but would nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill having the benefit of this disclosure.
Some embodiments described herein provide a quality preview of data validation rule changes in substantially real-time and may notify corresponding managers about potential impacts to data quality. Furthermore, embodiments may automate the quality preview whenever rules are changed.
At (A), a data validation server 150 accesses the active data validation rule from the data validation rule data store. The data validation server 150 may also receive, from a user via a user interface, an adjustment to the active data validation rule to create an inactive data validation rule. The data validation server 150 may also receive, from the user via the user interface, at least one filter condition defining a subset of the electronic records in the data repository 110. For example, the user might select the data representing all items produced at a certain manufacturing plant during a certain time period.
At (B), the data validation server 150 may retrieve the subset of electronic records from the data repository 110 and automatically simulate execution of the active data validation rule on record characteristics of the subset to obtain an “active” result. A used herein, the term “automatically” may refer to a device or process that can operate with little or no human interaction. Similarly, the data validation server 150 may automatically simulate execution of the inactive data validation rule on record characteristics of the subset of the electronic records in the data repository 110 to obtain an “inactive” result. At (C), the data validation server 150 may arrange for the active result and the inactive result to be displayed to the user. This might help a data steward, for example, determine if the adjustment to the active data validation rule improved or degraded the quality of the enterprise data.
According to some embodiments, devices, including those associated with the system 100 and any other device described herein, may exchange data via any communication network which may be one or more of a Local Area Network (“LAN”), a Metropolitan Area Network (“MAN”), a Wide Area Network (“WAN”), a proprietary network, a Public Switched Telephone Network (“PSTN”), a Wireless Application Protocol (“WAP”) network, a Bluetooth network, a wireless LAN network, and/or an Internet Protocol (“IP”) network such as the Internet, an intranet, or an extranet. Note that any devices described herein may communicate via one or more such communication networks.
The elements of the system 100 may store data into and/or retrieve data from various data stores (e.g., the data repository 110 and the data validation rule data store 120), which may be locally stored or reside remote from the data validation server 150. Although a single data validation server 150 is shown in
An operator (e.g., a database administrator) may access the system 100 via a remote device (e.g., a Personal Computer (“PC”), tablet, or smartphone) to view data about and/or manage operational data in accordance with any of the embodiments described herein. In some cases, an interactive graphical user interface display may let an operator or administrator define and/or adjust certain parameters (e.g., to set up or adjust various mapping relationships) and/or provide or receive automatically generated recommendations, results, and/or alerts from the system 100. According to some embodiments, the operator may generate instructions to adjust data quality rules (e.g., pre-determined tolerances) or set thresholds that, when triggered, may manually or automatically result in an adjustment of data validation rules.
At S210, a computer processor of a data validation server may access an active data validation rule from a data validation rule data store. The active data validation rule might indicate, for example, that an item price should approximately equal an item cost plus a profit margin. The rule might be “active” in that it is currently being used to validate enterprise information in a production environment. At S220, an adjustment to the active data validation rule may be received from a user (via a user interface) to create an inactive data validation rule. For example, the adjustment might indicate that an item price should approximately equal an item cost plus a profit margin plus a shipping cost. The rule might be “inactive” in that it is not currently being used to validate enterprise information.
At S230, at least one filter condition defining a subset of the electronic records in a data repository may be receiving from the user (via the user interface). The data repository may, for example, contain electronic records, each electronic record including a record identifier and a set of record characteristics. The filter condition might include, for example, an item identifier condition (e.g., all items having an identifier beginning with “X123 . . . ”, a location condition (e.g., all items manufactured in Europe), an item type condition (e.g., all items marked as “sold”), a date condition (e.g., all items manufactured in a particular month), etc. According to some embodiments, the data repository may be associated master data management, master data governance, a master data steward, etc.
At S240, the system may automatically simulate execution of the active data validation rule on record characteristics of the subset of the electronic records to obtain an active result. Similarly, at S250 the system may automatically simulate execution of the inactive data validation rule on record characteristics of the subset of the electronic records to obtain an inactive result. The active result and/or the inactive result might include, for example, an indication that an electronic record passed a validation rule, an indication that an electronic record did not pass a validation rule, an indication that a validation rule was not applied to an electronic record (e.g., an item associated with that record was not within the scope of the rule), a score calculated for a validation rule, etc.
The system may then arrange for the active result and the inactive result to be displayed to the user at S260. According to some embodiments, the display of the active result and the inactive result is provided graphically via the user interface. Moreover, selection of a portion of the graphic display may result in further details about the active result or inactive result being provided to the user.
According to some embodiments, the data validation server may further, responsive to an indication received from the user, replace logic of the active data validation rule with logic of the inactive data validation rule. That is, the adjusted rule will now be used to validate information in a production environment of the enterprise. Moreover, in some embodiments the data validation server is further to receive, from the user via the user interface, a further adjustment to the inactive data validation rule to create a new inactive data validation rule (e.g., a second revised version of the current active rule). The system can then automatically simulate execution of the new inactive data validation rule on the subset of the electronic records to obtain a new inactive result. According to some embodiments, the data validation server is further to automatically detect an anomaly in substantially real-time (e.g., when a validation score moves beyond a threshold value. The anomaly might be associated with, for example, a security anomaly (e.g., a potential cyber security problem in connection with the enterprise data), a regulatory compliance anomaly (e.g., a legal requirement to maintain certain records in accordance with a governmental regulation or statute), etc.
Some embodiments provide a user interface to simulate the rule and visualize the impact. For example,
Selection of a “Start” icon 450 may initiate execution of the simulation (e.g., for the active version of the rule).
According to some embodiments, a user can select a portion of the simulation results 530 to obtain a “drill down” view of supporting data.
Selection of a “Next” icon 550 may initiate execution of the simulation for the newly defined inactive version of the rule.
The display 700 also includes a number of “Not OK” records for both the active and inactive business rules 730 along with a number of “OK” records for both the active and inactive business rules 740. According to some embodiments, the display 700 provides an action 750 that can be performed by the user (e.g., to stop a currently running simulation, re-run a simulation, etc.). In this way, the display 700 may help a user make informed data validation rule decisions. In some embodiments, simulations will be automatically executed whenever a change happens in the rule, and the system may automatically notify a corresponding quality manager about the quality preview of the change (e.g., via a communication link that is automatically established using a communication address associated with the appropriate quality manager). In some embodiments, the display 700 may include additional columns such as when the simulation was started, who ran the simulation, records that “Not in Scope” (active and inactive), when the simulation was completed, whether a simulation should be automatically executed, information about alerts that should be automatically generated and/or transmitted, etc.
Note that the embodiments described herein may be implemented using any number of different hardware configurations. For example,
The processor 810 also communicates with a storage device 830. The storage device 830 can be implemented as a single database, or the different components of the storage device 830 can be distributed using multiple databases (that is, different deployment data storage options are possible). The storage device 830 may comprise any appropriate data storage device, including combinations of magnetic storage devices (e.g., a hard disk drive), optical storage devices, mobile telephones, and/or semiconductor memory devices. The storage device 830 stores a program 812 and/or data validation engine 814 for controlling the processor 810. The processor 810 performs instructions of the programs 812, 814, and thereby operates in accordance with any of the embodiments described herein. For example, the processor 810 may access an active data validation rule from a data validation rule data store 900. The processor 810 receives an adjustment to the active data validation rule from a user to create an inactive data validation rule. The processor 810 may also receive at least one filter condition defining a subset of the electronic records in a data repository 860. The processor 810 can then automatically simulate execution of the active data validation rule on record characteristics of the subset to obtain an active result and automatically simulate execution of the inactive data validation rule to obtain an inactive result. The processor 810 then displays the active result and the inactive result to the user (e.g., via the user interface 824).
The programs 812, 814 may be stored in a compressed, uncompiled and/or encrypted format. The programs 812, 814 may furthermore include other program elements, such as an operating system, clipboard application, a database management system, and/or device drivers used by the processor 810 to interface with peripheral devices.
As used herein, data may be “received” by or “transmitted” to, for example: (i) the platform 800 from another device; or (ii) a software application or module within the platform 800 from another software application, module, or any other source.
In some embodiments (such as the one shown in
Referring to
The data validation rule identifier 902 might be a unique alphanumeric label or link that is associated with a business data validation rule. The rule name 904 may describe the rule and the rule 906 may define the conditions associated with that rule. The rule status 908 might indicate if that rule is currently active or inactive. For example, rule “DVR_101” named “Shipping Weight” (Shipping Weight>Product Weight) is currently active while rule “DVR_101.1” named “Shipping Weight Modified” (Shipping Weight>Product Weight+Package Weight) is inactive (e.g., it might currently be undergoing evaluation by a data steward). In contrast, “DVR_102.1” named “Total Price Modified” (Price≈Cost+Profit+Shipping) has been made active (and has replaced prior version “DVR_102” after being approved by a data steward).
Thus, embodiments may provide substantially real-time insight of potential impacts on data quality that are caused by business rule changes. Embodiments may let a data quality manager or steward compare current data quality with future data quality to support informed decisions. In some embodiments, automatic quality preview alerts may be generated whenever changes happen to business rules.
The following illustrates various additional embodiments of the invention. These do not constitute a definition of all possible embodiments, and those skilled in the art will understand that the present invention is applicable to many other embodiments. Further, although the following embodiments are briefly described for clarity, those skilled in the art will understand how to make any changes, if necessary, to the above-described apparatus and methods to accommodate these and other embodiments and applications.
Although specific hardware and data configurations have been described herein, note that any number of other configurations may be provided in accordance with some embodiments of the present invention (e.g., some of the data associated with the databases described herein may be combined or stored in external systems). Moreover, although some embodiments are focused on particular types of business rules, any of the embodiments described herein could be applied to other types of business rules (including relatively complex business rules). Moreover, the displays shown herein are provided only as examples, and any other type of user interface could be implemented. For example,
In some embodiments, a data steward may manually define data validation rule changes and/or evaluate active and inactive simulation results (e.g., do decide which business rules should be deployed in a production environment). In other embodiments, artificial intelligence and/or machine learning algorithms and predictive models may be used to automate either of these processes (and feedback information may be used to improve the performance of the system).
The present invention has been described in terms of several embodiments solely for the purpose of illustration. Persons skilled in the art will recognize from this description that the invention is not limited to the embodiments described but may be practiced with modifications and alterations limited only by the spirit and scope of the appended claims.
