Patent Application
20230004845
This related generally to AI explainability, and more specifically to AI-augmented auditing platform including techniques for providing AI-explainability for processing data through multiple layers.
Known techniques for processing data and generating adjudications based on the processed data include enterprise resource planning (ERP) systems with data-warehouse and/or data mart capabilities, which allow drill-down and rollup of digital data. These systems may provide direct linkage of aggregated digital data or data processed by simple mathematical formulas only. Furthermore, known techniques include content description frameworks for multimedia content including text, images, and video (see, e.g., “Multimedia Content Description Framework” U.S. Pat. No. 6,564,263 (L. D. Bergman, M.Y. Y. Kim, C.-S. Li, R. Mohan, J. R. Smith), which forms the basis for the multimedia metadata standard MPEG-7. Furthermore, known techniques include business rules engines that allow forward and backward chaining of business rules as part of reasoning processes. Furthermore, known techniques include conducting testing with tick-marks, pulldown menus, and manual entries of explanations on workpaper, or with circling or highlighting areas of interest. Known automation approaches use ad hoc approaches for identifying results needing follow-up, or are based on single-layer taxonomy/vocabulary for defining outputs.
As explained above, known techniques for processing data and generating adjudications include enterprise resource planning (ERP) systems and may provide direct linkage of aggregated data or data processed by simple mathematical formulas only. Furthermore, known techniques include content description frameworks for multimedia content including text, images, and video. Furthermore, known techniques include business rules engines and include conducting testing manual entries of explanations. Known automation approaches use ad hoc approaches for identifying results needing follow-up, or are based on single-layer taxonomy/vocabulary for defining outputs.
However, known business rules engines do not retain a provenance of the deduction processes that they apply—e.g., they do not retain a log of rules that have been applied by the engine. Moreover, known techniques do not provide for traceability (including full traceability) and provenance of interpretation, reasoning, deliberation, and adjudication through various layers, including from raw evidence through final observation, conclusion, and recommendation. Furthermore, known techniques do not provide flexibility in presenting generated recommendations and/or remediations, adaptability in light of evolving needs, nor minimization and localization of necessary changes within systems. Furthermore, known techniques do not provide robust and flexible frameworks for applying different taxonomy/vocabulary/logic at each of a plurality of layers, generating output data and explainability outputs at each of the plurality of layers, and allowing localized modification of any of the layers without modification of other layers.
Accordingly, there is a need for improved systems and methods for tracking the reasoning and adjudication processes by which data is processed through a plurality of interconnected layers, wherein said improved systems address one or more of the above-identified shortcomings.
Disclosed herein are systems and methods for tracking the reasoning and adjudication processes by which data is processed through a plurality of interconnected layers. In some embodiments, the techniques disclosed herein may be applied to tracking the reasoning and adjudication processes by which evidence data and information are processed through a plurality of layers during an audit process. The fine-grained explainability of the reasoning process is critical for the human auditor to understand and believe in the conclusion and recommendation. The decoupling of the tracking at each stage of the deliberation enable maximal flexibility while minimize the necessary modification when frequent changes of the required summary, conclusion, and recommendation are required.
In some embodiments, a first system for providing explainability for processing data through multiple layers is provided, the first system comprising one or more processors configured to cause the first system to provide: an input layer configured to: receive an evidence data set comprising a plurality of evidence items; apply one or more evidence processing models to the evidence data set to generate evidence understanding data; and generate an input-layer output for presentation to a user, wherein the input layer output represents the processing of the evidence data set by the input layer; a presentation layer configured to: receive data, wherein the received data includes one of: the evidence understanding data, and data generated based on the evidence understanding data; apply one or more presentation generation models to the received data to generate presentation data; and generate a presentation output for presentation to the user, wherein the presentation output comprises the presentation data.
In some embodiments of the first system, the one or more processors are configured to provide: one or more intermediate layers configured to: receive the evidence understanding data generated by the input layer; apply one or more intermediate-layer processing models to the evidence understanding data to generate the data received by the presentation layer; provide the data received by the presentation layer to the presentation layer; and generate an intermediate-layer output for presentation to the user, wherein the intermediate-layer output represents the processing of the evidence understanding data by the one or more intermediate layers.
In some embodiments of the first system, the input layer and the presentation layer are each configured to apply a respective ontology.
In some embodiments of the first system, the one or more processors are configured to: receive a user input comprising an instruction to modify the input layer; modify the input layer in accordance with the user input without modifying the presentation layer.
In some embodiments of the first system, the one or more processors are configured to: receive a user input comprising an instruction to modify the presentation layer; and modify the presentation layer in accordance with the user input without modifying the input layer.
In some embodiments of the first system, the one or more processors are configured to initialize the presentation layer by applying one or more machine learning models to classify output data from one or more prior analyses performed by the system.
In some embodiments of the first system, the one or more processors are configured to: receive utilization data representing a manner in which the presentation output is utilized by one or more users; and automatically modify the presentation layer in accordance with the utilization data.
In some embodiments, a non-transitory computer-readable storage medium storing instructions for providing explainability for processing data through multiple layers is provided, wherein the instructions are configured to be executed by a system comprising one or more processors to cause the system to provide: an input layer configured to: receive an evidence data set comprising a plurality of evidence items; apply one or more evidence processing models to the evidence data set to generate evidence understanding data; and generate an input-layer output for presentation to a user, wherein the input layer output represents the processing of the evidence data set by the input layer; a presentation layer configured to: receive data, wherein the received data includes one of: the evidence understanding data, and data generated based on the evidence understanding data; apply one or more presentation generation models to the received data to generate presentation data; and generate a presentation output for presentation to the user, wherein the presentation output comprises the presentation data.
In some embodiments, a method for providing explainability for processing data through multiple layers is provided, wherein the method is performed by a system comprising one or more processors, the method comprising: by an input layer of the system: receiving an evidence data set comprising a plurality of evidence items; applying one or more evidence processing models to the evidence data set to generate evidence understanding data; and generating an input-layer output for presentation to a user, wherein the input layer output represents the processing of the evidence data set by the input layer; by a presentation layer of the system: receiving data, wherein the received data includes one of: the evidence understanding data, and data generated based on the evidence understanding data; applying one or more presentation generation models to the received data to generate presentation data; and generating a presentation output for presentation to the user, wherein the presentation output comprises the presentation data.
In some embodiments, a second system for providing explainability for processing data through multiple data-processing layers is provided, the second system comprising one or more processors configured to cause the second system to cause the second system to: at an input layer: receive an evidence data set comprising a plurality of evidence items; apply one or more evidence processing models to the evidence data set to generate evidence understanding data; and generate input-layer explainability data, wherein the input-layer explainability data represents information about the processing of the evidence data set by the input layer; at a presentation layer: receive data, wherein the received data includes one of: the evidence understanding data, and data generated based on the evidence understanding data; apply one or more presentation generation models to the received data to generate presentation data; and generate presentation-layer explainability data, wherein the presentation-layer explainability data represents information about the processing of the received data by the input layer; cause display of the presentation data; and cause display of one or more of: the input-layer explainability data and the presentation-layer explainability data.
In some embodiments, any one or more of the features, characteristics, or aspects of any one or more of the above systems, methods, or non-transitory computer-readable storage media may be combined, in whole or in part, with one another and/or with any one or more of the features, characteristics, or aspects (in whole or in part) of any other embodiment or disclosure herein.
Various embodiments are described with reference to the accompanying figures, in which:
As explained above, known techniques for processing data and generating adjudications include enterprise resource planning (ERP) systems and may provide direct linkage of aggregated data or data processed by simple mathematical formulas only. Furthermore, known techniques include content description frameworks for multimedia content including text, images, and video. Furthermore, known techniques include business rules engines and include conducting testing manual entries of explanations. Known automation approaches use ad hoc approaches for identifying results needing follow-up, or are based on single-layer taxonomy/vocabulary for defining outputs.
However, known business rules engines do not retain a provenance of the deduction processes that they apply—e.g., they do not retain a log of rules that have been applied by the engine. Moreover, known techniques do not provide for traceability (including full traceability) and provenance of interpretation, reasoning, deliberation, and adjudication through various layers, including from raw evidence through final observation, conclusion, and recommendation. Furthermore, known techniques do not provide flexibility in presenting generated recommendations and/or remediations, adaptability in light of evolving needs, nor minimization and localization of necessary changes within systems. Furthermore, known techniques do not provide robust and flexible frameworks for applying different taxonomy/vocabulary/logic at each of a plurality of layers, generating output data and explainability outputs at each of the plurality of layers, and allowing localized modification of any of the layers without modification of other layers.
Accordingly, there is a need for improved systems and methods for tracking the reasoning and adjudication processes by which data is processed through a plurality of interconnected layers, wherein said improved systems address one or more of the above-identified shortcomings.
Disclosed herein are systems and methods for tracking the reasoning and adjudication processes by which data is processed through a plurality of interconnected layers. In some embodiments, the techniques disclosed herein may be applied to tracking the reasoning and adjudication processes by which evidence data and information are processed through a plurality of layers during an audit process. The fine-grained explainability of the reasoning process is critical for the human auditor to understand and believe in the conclusion and recommendation. The decoupling of the tracking at each stage of the deliberation enable maximal flexibility while minimize the necessary modification when frequent changes of the required summary, conclusion, and recommendation are required.
In some embodiments, the systems and methods disclosed herein provide full explainability for digitizing evidence that is not yet in digital form, for example by applying one or more document-understanding techniques to ingested documents for which the included data is not yet in digital form. In some embodiments, documents may be processed via one or more document understanding operations comprising one or more of structural, semantic, and/or linguistic analysis of the documents. In some embodiments, the systems may apply one or more natural-language understanding (NLU) techniques to extract information from received documents and to generate structured data based on said ingested documents, such that said structured data can be further processed by one or more data processing models, including by being processed through a plurality of layers and by being used to generate explainable outputs at one or more of the plurality of layers, as described herein. (Any data processing operation referenced herein may include application of one or more models trained by machine-learning.)
In some embodiments, the systems and methods disclosed herein provide full explainability for interpretation of processes and results during cross-validation between and/or among evidence.
In some embodiments, the systems and methods disclosed herein provide full explainability for reporting results from one or more final adjudications (e.g., assessments, classifications, quantifications, characterizations, and/or scores made on the basis of evidence provided) based on input data regarding available evidence. For example, adjudications based on available evidence may include adjudications as to whether one or more accounting policies and/or auditing standards are satisfied.
In some embodiments, the systems and methods disclosed herein provide full explainability for describing patterns, trends, and/or insights gained from evaluations of integrity of evidence, including evaluations made based on an individual piece of evidence and/or evaluations made based on multiple pieces of evidence.
In some embodiments, the systems and methods disclosed herein provide full explainability for describing final recommendations (e.g., outputs provided to one or more systems or users wherein the output indicates a recommended action) based on input data regarding available evidence, including recommendations generated based on an individual piece of evidence and/or based on multiple pieces of evidence.
In some embodiments, a system for providing explainability for data processing may define a plurality of layers through which data may be processed. Each layer may be provided by one or more processors, including embodiments in which layers are provided by separate processors and in which layers are provided by the same processors. Each layer may receive input data to the layer, apply one or more data processing operations to the received input data to generate output data, and may generate explainability data to be presented to a user to explain the input data, processing operations, and/or output data of the layer. The layers may be communicatively interconnected with one another such that output data from one layer may form all or part of the input data for another layer.
The plurality of layers may include an input layer which is communicatively coupled with one or more data sources for providing raw evidence data as input to the input layer. The input layer may apply one or more data processing operations to the received evidence data to generate output data of the input layer. The input layer may then provide the generated output data to one or more subsequent layers to which it is communicatively connected, such that the generated output data may be used by the one or more subsequent layers as input data. The input layer may additionally generate an explainability output, such as a user-facing visualization, report, or other human-readable information that may be displayed, transmitted, and/or presented to one or more users. The explainability output for the input layer may indicate information regarding the manner in which the input layer received evidence data, processed said evidence data, generated output data based on said evidence data, and/or provided said generated output data to one or more subsequent layers. In some embodiments, the input layer may be configured for providing fine-grained precision interpretation of evidence data.
The plurality of layers may include one or more intermediate layers that may be directly and/or indirectly communicatively coupled with and downstream of the input layer. The intermediate layers may be disposed in series and/or in parallel with one another. An intermediate layer may receive, as input data, the output data that is generated by the input layer. Alternatively or additionally, the intermediate layer may receive, as input data, output data that was generated by another intermediate layer. An intermediate layer may apply one or more data processing operations to the received data to generate output data of the intermediate layer. An intermediate layer may then provide the generated output data to one or more subsequent layers to which it is communicatively connected, such that the generated output data may be used by the one or more subsequent layers (e.g., another intermediate layer and/or a presentation layer, as described below) as input data. An intermediate layer may additionally generate an explainability output for the intermediate layer, such as a user-facing visualization, report, or other human-readable information that may be displayed, transmitted, and/or presented to one or more users. The explainability output for the intermediate layer may indicate information regarding the manner in which the intermediate layer received data, processed said received data, generated output data based on said received data, and/or provided said generated output data to one or more subsequent layers. In some embodiments, an intermediate layer may be configured for providing visibility and traceability of intermediate reasoning and adjudication of multiple related pieces of evidence.
The plurality of layers may include a presentation layer that may be directly and/or indirectly communicatively coupled with and downstream of one or more of the intermediate layers and/or the input layer. The presentation layer may receive, as input data, the output data that is generated by the input layer and/or by one or more of the input layers. The presentation layer may apply one or more data processing operations to the received data to generate output data of the presentation layer, wherein the output data of the presentation layer may include actionable insights, observations, interpretations, recommendations, and/or remediations. The output data of the presentation layer may be displayed, transmitted, and/or presented to one or more users. The output data of the presentation layer may be stored, transmitted to one or more other systems, and/or used to selectively trigger one or more automated actions by the system. The presentation layer may additionally generate an explainability output for the presentation layer, such as a user-facing visualization, report, or other human-readable information that may be displayed, transmitted, and/or presented to one or more users. The explainability output for the presentation layer may indicate information regarding the manner in which the presentation layer received data, processed said received data, generated output data based on said received data, and/or presented or provided said generated output data to one or more users and/or one or more other systems. In some embodiments, the presentation layer may be configured for presenting actionable insight, observation, interpretation, recommendation, and remediation.
The system may be configured such that a respective ontology (e.g., taxonomy and/or vocabulary) is established for each layer. The system may store data representing the respective ontology for each layer. In some embodiments, the system may store definition ontology data for a respective layer to enable explanations within the environment are standardized. In some embodiments, the ontology for a respective layer may be codified into reason code; codification of ontological data into reason code may simplify the tracing of deduction and reasoning processes.
In some embodiments, data stored for a respective layer (including, e.g., the ontology for a respective layer) may define a manner in which the layer is interconnected (e.g., configured to receive data from and/or provide data to) one or more other layers of the system.
In some embodiments, data stored for a respective layer (including, e.g., the ontology for a respective layer) may define deductive reasoning processes able to be applied by the layer, including one or more deterministic reasoning frameworks and/or one or more probabilistic reasoning frameworks. In some embodiments, data stored for a respective layer (including, e.g., the ontology for a respective layer) may define inductive reasoning processes able to be applied by the layer, including one or more machine learning models and/or one or more deep learning models. In some embodiments, data stored for a respective layer (including, e.g., the ontology for a respective layer) may define abductive reasoning processes able to be applied by the layer. In some embodiments, data stored for a respective layer (including, e.g., the ontology for a respective layer) may define general-purpose logic processes able to be applied by the layer.
In some embodiments, the system may be configured to be able to be modified/reconfigured in accordance with instructions received from a user, in accordance with performance data regarding the performance of the system, and/or in accordance with data received from one or more other systems. In some embodiments, the system may be configured such that modification of codification may be made with localized impact. The system may be configured such that information defining the ontology, data processing operations, explainability output generation, and/or layer interconnectedness of any one layer is able to be modified without modifying any of the other layers in the system. In one example, modification of the presentation layer may be achieved by modifying logic rules that connect the presentation layer to one or more upstream layers (e.g., intermediate layer(s) and/or input layer) without modifying the upstream layers themselves. In another example, modification of the input layer may be achieved by modifying logic rules that connect the input layer one or more downstream layers (e.g., intermediate layer(s) and/or presentation layer) without modifying the downstream layers themselves. In another example, modification of an intermediate layer may be achieved by modifying logic rules that connect the intermediate layer to one or more upstream or downstream layers (e.g., input layer, other intermediate layer(s), and/or presentation layer) without modifying the upstream or downstream layers themselves.
In some embodiments, the presentation layer may be configured in accordance with results of one or more supervised or unsupervised machine learning models, such as one or more clustering models and/or classification models. The presentation layer may be initialized, in some embodiments, based at least in part on historical data (e.g., data taken from one or more prior projects and/or prior analyses). The system may apply a supervised or unsupervised learning model to cluster/classify output data from one or more prior analyses, for example by applying the model(s) to data representing commentary, observations, summaries, conclusions, and/or recommendations from one or more prior analyses. The resulting output data from applying the model(s) to said data may be used to initialize the presentation layer.
In some embodiments, the system may be configured to continuously/iteratively learn via utilization of one or more of the layers. For example, the system may be configured to monitor user engagement with output data and/or with explainability data generated by one or more of the layers; user utilization of output data and/or of explainability data generated by one or more of the layers; and/or explicit user feedback regarding presentation layer output data and/or explainability data generate by one or more of the layers. For example, the system may track which portions of output data and/or explainability data are selected by the user to cause the system to “drill down” to display additional information. On the basis of the monitored engagement, utilization, and/or feedback data, the system may automatically update configurations of one or more of the layers (e.g., to adjust future explainability outputs and/or to adjust future recommendations). Updates made on this basis may include updates as described elsewhere herein, for example updates to individual layers and/or updates to the manner in which layers are interconnected with one another. In this way, the system may be configured to apply a feedback loop such that output data and explainability outputs generated by the system may be continuously/iteratively improved over time.
Some embodiments of the systems and methods described herein may be applied in the audit of financial data, such as data regarding revenue and receivable for an entity. While the figures below may describe the systems and methods disclosed herein as they relate to an audit of financial data, a person of ordinary skill in the art will understand, in light of the disclosure herein, that the systems and methods described herein may be applicable to any financial statement line item (FSLI)—including expense and payable, property, plant and equipment (PPE) (including lease accounting), cash and cash equivalent, JE, etc.—and may also be applicable to automatically processing any kind of data and providing explainability therefor, including processing and providing explainability for data that is not financial data.
Herein, the disclosed embodiments demonstrate how the systems and methods disclosed herein may be applied to an audit of financial data.
Account receivable and revenue accounts within the chart of accounts capture the revenue generated through the order to cash process, as shown in
During the order to cash process, various information systems may participate in the process. As shown in
An audit of revenue account, as a result, may include one or more data processing operations that validate data values associated with the account by tracing the transaction through the system in conjunction with corroborating evidence to ensure that each transaction has been posted correctly according to one or more criteria, for example criteria associated with accounting standards such as accounting policy ASC 606 (IFRS 15).
As shown in
As further shown in
In some embodiments, system 400 may include a plurality of intermediate layers arranged in series and/or in parallel, and system 400 may apply one or more data processing operations to the data processed through the intermediate layers. In some embodiments, multiple intermediate layers may receive the same input data. In some embodiments, one intermediate layer may forward received input data to another intermediate layer in the same format that it was received. In some embodiments, an upstream intermediate layer (e.g., layer 406) may generate output data (e.g., 408) that may itself be used as input data for a downstream intermediate layer (e.g., layer 410) to produce downstream intermediate output data (e.g., 412).
As further shown in
In some embodiments, a multilayer data processing such as system 400 may be configured to process data through one or more an input layers, a plurality of intermediate layers, and one or more presentation layers.
An input layer may be configured to receive input data from one or more external data sources, such as one or more databases, data stores, data repositories, live data feeds, or the like. The data may be received in structured, unstructured, and/or partially-structured (e.g., semi-structured) format. The data may be received by the input layer on a scheduled basis, in response to a user input, in response to one or more trigger conditions being met, and/or in response to the data being manually sent.
The input layer may comprise one or more processors configured to apply one or more data processing operations to some or all of the received input data to thereby generate output data. The output data may comprise, for example, data extracted from raw input data and/or normalized data generated based on input data. The generated output data may be stored locally, transmitted to one or more other system components, and/or forwarded to one or more downstream layers in the multilayer system for further processing.
In addition to the output data, the one or more processors of the input layer may additionally generate explainability output data, which may include metadata indicating a manner in which the output data was generated, for example including an indication of the input data, an indication of the data processing operation(s) applied, an indicating of a time and/or place of data processing, and/or an indication of one or more configurations associated with said data processing. The generated explainability output data may be stored locally or remotely and/or transmitted to one or more other system components.
An intermediate layer may be configured to receive input data from an input layer and/or from another intermediate layer that is upstream of the intermediate layer. In some embodiments, an intermediate layer may also receive input data from one or more external data sources, in a same or similar manner as described above with respect to an input layer. The data may be received in structured, unstructured, and/or partially-structured (e.g., semi-structured) format. The data may be received by the intermediate layer on a scheduled basis, in response to a user input, in response to one or more trigger conditions being met, in response to the data being manually sent, and/or in response to the data being generated by an upstream layer including an input layer and/or another intermediate layer.
The intermediate layer may comprise one or more processors configured to apply one or more data processing operations to some or all of the received input data to thereby generate output data. The output data may comprise, for example, one or more adjudications, scores, calculations, classifications, identified events, and/or identified content generated based on the input data. The generated output data may be stored locally, transmitted to one or more other system components, and/or forwarded to one or more downstream layers in the multilayer system for further processing.
In addition to the output data, the one or more processors of the intermediate layer may additionally generate explainability output data, which may include metadata indicating a manner in which the output data was generated, for example including an indication of the input data, an indication of the data processing operation(s) applied, an indicating of a time and/or place of data processing, and/or an indication of one or more configurations associated with said data processing. The generated explainability output data may be stored locally or remotely and/or transmitted to one or more other system components.
A presentation layer may be configured to receive input data from an input layer and/or from an intermediate layer upstream of the presentation layer. In some embodiments, a presentation layer may also receive input data from one or more external data sources, in a same or similar manner as described above with respect to an input layer. The data may be received in structured, unstructured, and/or partially-structured (e.g., semi-structured) format. The data may be received by the presentation layer on a scheduled basis, in response to a user input, in response to one or more trigger conditions being met, in response to the data being manually sent, and/or in response to the data being generated by an upstream layer including an input layer and/or an intermediate layer.
The presentation layer may comprise one or more processors configured to apply one or more data processing operations to some or all of the received input data to thereby generate output data. The output data may comprise, for example, one or more adjudications, scores, calculations, classifications, identified events, identified content, alerts, user interface objects, and/or visualizations generated based on the input data. In some embodiments, output data from a presentation layer may comprise an indication of classification, such as a risk classification. In some embodiments, output data from a presentation layer may indicate a risk classification for a transaction represented by the original input data received by the system. In some embodiments, output data from a presentation layer may indicate an irregularity, anomaly, and/or inconsistency for data that was received by and processed by the system. In some embodiments, output data from a presentation layer may indicate whether one or more criteria (e.g., vouching criteria, tracing criteria, related-party transaction criteria, data integrity criteria, process integrity criteria, and/or policy integrity criteria) are satisfied by the data that was received by and processed by the system. In some embodiments, output data from a presentation layer may comprise a confidence score associated with one or more indications (e.g., labels or classifications) of the output data. The generated output data may be stored locally, transmitted to one or more other system components, and/or presented (e.g., displayed on a display of a user system) to a user.
In addition to the output data, the one or more processors of the presentation layer may additionally generate explainability output data, which may include metadata indicating a manner in which the output data was generated, for example including an indication of the input data, an indication of the data processing operation(s) applied, an indicating of a time and/or place of data processing, and/or an indication of one or more configurations associated with said data processing. The generated explainability output data may be stored locally or remotely and/or transmitted to one or more other system components.
In some embodiments, different layers in a multilayer data processing system may leverage one or more of the same processors for data processing operations performed at different layers; in some embodiments, one or more separate processors may be used for data processing operations performed at different layers.
In some embodiments, a multilayer data processing system may only present (e.g., display) output data generated by a presentation layer to a user. In some embodiments, the system may additionally present some or all output data generated by one or more upstream layers to the user. In some embodiments, the system may present some or all of the original input data received by the system.
In some embodiments, in addition to output data, the system may present some or all of the explainability data generated by one or more of the layers (including input layers, intermediate layers, and/or presentation layers). In some embodiments, explainability data may be presented (e.g., displayed, for example as part of a visualization) to a user upon being generated, upon presentation to the user of output data associated with the explainability data, and/or in response to one or more requests from the user requesting the display of explainability data. In some embodiments, a user interface may initially display output data generated by one or more presentation layers, and the interface may afford the user the opportunity to execute one or more user inputs requesting display of explainability data associated with the displayed output data. For example, a user execute an input by clicking on a visualization, selecting one or more affordances, and/or entering one or more input strings into one or more fields. The user input may indicate a request for explainability data to be presented, for example by indicating (e.g., clicking on) a piece of output data for which associated explainability data should be requested. The user input may indicate a request for explainability data to be presented, for example by indicating a type of explainability data, an explainability visualization type, and/or one or more layer of the system for which to display explainability data. In some embodiments, a user may click on output data in order to “drill down” into associated explainability data, for example causing the system to display explainability data for the layer that generated the output data that was clicked, and/or causing the system to display output data for one or more upstream layers that was used as input to generate the output data that was clicked. In some embodiment, a user may execute a plurality of successive drill-down inputs, clicking on newly-displayed output data and/or newly-displayed explainability data to drill down further into the data that was processed through the multilayer system. By execute a plurality of successive drill-down inputs, a user may move backwards from the final output data (initially displayed to the user) back through all of the layers of the system, such that the system displays all intermediate data and the original input data that was used to arrive at the selected output data and/or such that the system displays explainability data for all system layers leveraged to arrive at the output data.
In some embodiments, the one or more processors of a layer of the system may be configured to apply one or more data processing operations that may include deductive reasoning data processing operations, inductive reasoning data processing operations, and/or abductive reasoning data processing operations.
In some embodiments, input data originally provided to an input layer of the system may include data indicating time information, and the data-processing operations performed by one or more layers of the system may include generating time-dependent output data. Time dependent information can be presented to the input either sequentially (e.g., for information that belong to the same point in time) or concurrently (e.g., for information that belongs to more than one point in time).
In some embodiments, (alternatively to or in addition to time-dependency as described above) input data originally provided to an input layer of the system may data indicating location information, and the data-processing operations performed by one or more layers of the system may include generating location-dependent output data. In some embodiments, location may be expressed by a two-dimensional description (x, y), which may for example represent a latitude and longitude; or location may be expressed by a three-dimensional description (x, y, z), where the third dimension indicates a vertical position, altitude, as measured from sea-level, and/or height off of the ground.
In some embodiments, when the system generates output data that is time-dependent and/or location-dependent, a user may execute one or more inputs to request display of output data (and/or associated explainability data) for period of time and/or for a location that is specified by the input. In some embodiments, the system may responsively update the displayed output data in order to display information relevant to the user-specified time and/or location. In some embodiments, the system may display output data that has already been generated; in some embodiments, the system may activate one or more of the data processing layers in order to process data to generate output data for the user-specified time and/or location.
In some embodiments, the one or more processors of a layer of the system may be configured to determine which of a plurality of data processing operations should be applied. The one or more processors may automatically select one or more data processing operations to apply based on system settings, user input, and/or based on input data generated by one or more upstream layers of the system. In some embodiments, the input data (e.g., data format, data type, data content, etc.) may dictate which one or more data processing operations are applied by a layer (and/or may dictate whether the layer applies any data processing operations or alternatively simply forwards the input data to another layer).
In some embodiments, the system may be configurable according to one or more user inputs, which may be provided to the system by an end user and/or administrator. In some embodiments, the system may be configurable according to one or more user inputs that specify a manner in which one or more layers processes data. For example some embodiments, a user input may indicate a type or identity of analysis, a type or identity of test, a type or identity of model, one or more weight values, and/or one or more threshold values, and the system may configure a specified layer in accordance with the user input. In some embodiments, reconfiguring a layer of the multilayer system may be done without reconfiguring any other layers of the system, including upstream layers that provide input data to the reconfigured layer and including downstream layers that accept the output data of the reconfigured layer. In some embodiments, a layer may be reconfigured (e.g., by adjusting data processing operations applied by the layer) without changing the format of data accepted as input by the layer and without changing the format of data generated as output by the layer. In this manner, a layer may be adjusted and may continue to interact with upstream and downstream layers in the same manner as before the adjustment was made.
In some embodiments, a layer of the system may be reconfigurable to change the amount, type, and/or content of explainability data generated by a layer. In some embodiments, the explainability data generated by a layer may be changed without changing the output data itself that is generated by the layer, such that downstream layers that receive the output data from that layer are not affected by the reconfiguration.
In some embodiments, a layer may be reconfigured in accordance with a change to an ontology that is leveraged by a data processing operation applied by the layer. For example, if an additional class is identified, then the additional class may be added to an ontology used by a classifier of the layer. In some embodiments, the addition of the new class cause the layer to generate output data in a same format (e.g., using a same language and/or a same set of symbols) as before the addition, such that downstream layers do not need to be adjusted to interpret output data that is generated in accordance with the new class. In some embodiments, the layer to which an additional class is added may be configured such that it applies one or more translation operations, normalization operations, and/or other data processing operations to convert data generated in accordance with the new class into a format that can be interpreted by downstream layers without requiring adjustments to the downstream layers. (In some embodiments, a downstream layer may be adjusted in order to ensure that the downstream layer can process all possible upstream output data when an upstream layer is reconfigured to generate output data comprising one or more new classes, new symbols, or the like.)
In some embodiments, the system may be configurable according to one or more user inputs that specify a manner in which one or more layers are interconnected with one or more other layers. For example, a user input may indicate: (a) from which upstream layers and/or other data sources a certain layer should accept input; and/or (b) to which downstream layers a certain layer should provide output data for further processing by the downstream layers. In some embodiments, a user input may specify whether a given layer should (or should not) forward any received data (without processing) to one or more subsequent layers. In some embodiments, a user input may indicate whether receipt of input data from a given source and/or provision of output data to a given source is optional or required. In some embodiments, a user may configure the interconnections between the layers by selecting one or more layers to be linked downstream or upstream to a given layer from a menu, and/or by using a drag-and-drop user interface to rearrange layers and/or to place, rearrange, and/or remove connecting links between visualizations representing respective layers.
In some embodiments of a multilayer data-processing schema such as schema 500, raw evidence data may be ingested and may be processed through one or more normalization operations with respect to context (such as customer/vendor/product master data) for being evaluated as evidence. Data and process integrity validation (see Disclosure on Composable Data-Process-Policy integrity framework for FSLI audit) data processing operations may be applied in order to perform validation and produce integrity observation data. This observation data may include report mismatches of various fields, such as customer name, address, shipping term, payment term, line items, unit price, and/or total amount. The interpretation of evidence evaluation data that belongs to the same transaction may then combined through deductive, inductive, and/or abductive reasoning data processing operations. Further aggregation data processing operations may be applied based on customer, location, and/or product, and said data processing operations may generate insight data. Insight data, such as data indicating one or more distinct out-of-ordinary behaviors or events, may be subject to one or more data processing operations to generate actionable insight data, which itself may be processed by one or more data processing operations to generate recommendation data indicating one or more potential solutions for such behaviors and/or events.
As shown in
As further shown in
As further shown in
As further shown in
As further shown in
As further shown in
In some embodiments, a system for tracking reasoning and adjudication processes for processing data through a plurality of interconnected layers may process each of the different classes of data described with respect to
In some embodiments, a system for tracking reasoning and adjudication processes for processing data through a plurality of interconnected layers, such as those described herein, may identify one or more of the anomalous classifications shown in the bottom portion of
In some embodiments of a system architecture 700, evidence data such as purchase order data 708, bill of lading data 706, and bank statement data 704 may be processed through a plurality of data processing operations (e.g., document-understanding models, AI models, machine learning models, etc.) to generate one or more instances of intermediate data (e.g., data regarding vouching, invoices, revenue, contracts, and/or prices). The intermediate data may additionally be generated on the basis of ERP data. In some embodiments, the intermediate data may be used to generate recommendation data 716, such as a recommendation to perform a revenue audit.
In the arrangement shown in
ERP data from ERP data source 702 may additionally be received by process integrity system 712, which may include one or more processors configured to perform one or more process integrity data processing operations. Process integrity system 712 may validate assertions (e.g., accuracy, cutoff, and/or classification) by tracing transactions based on ERP data from data source 702, for example by following a process (e.g., an order-to-cash transaction process, or a procure-to-pay transaction process) end to end. The one or more process integrity data processing operations may include any of those performed by process integrity system 110 as described in U.S. Patent Application titled “AI-AUGMENTED AUDITING PLATFORM INCLUDING TECHNIQUES FOR APPLYING A COMPOSABLE ASSURANCE INTEGRITY FRAMEWORK,” filed Jun. 30, 2022, Attorney Docket No. 13574-20070.00.
The output data generated by data integrity system 710 and/or process integrity system 712 may be received by policy integrity system 714, which may include one or more processors configured to perform one or more policy integrity data processing operations. Policy integrity system 714 may validates and adjudicate whether the input data (e.g., representing an overall transaction) is consistent with one or more predefined criteria, for example including accounting standards criteria and/or policy standards criteria. The one or more policy integrity data processing operations may include any of those performed by process integrity system 110 as described in U.S. Patent Application titled “AI-AUGMENTED AUDITING PLATFORM INCLUDING TECHNIQUES FOR APPLYING A COMPOSABLE ASSURANCE INTEGRITY FRAMEWORK,” filed Jun. 30, 2022, Attorney Docket No. 13574-20070.00. The output data generated by policy integrity system 714 may include, or may be used to in turn generate, recommendation data 716.
Below are steps for, in some embodiments, establishing summary/recommendation data to be presented to a human user:
Below are steps for, in some embodiments, tracing a reasoning process. For a specific order to cash transaction, we could potentially go through the following evaluation:
Computer 800 can be a host computer connected to a network. Computer 800 can be a client computer or a server. As shown in
Input device 820 can be any suitable device that provides input, such as a touch screen or monitor, keyboard, mouse, or voice-recognition device. Output device 830 can be any suitable device that provides an output, such as a touch screen, monitor, printer, disk drive, or speaker.
Storage 840 can be any suitable device that provides storage, such as an electrical, magnetic, or optical memory, including a random access memory (RAM), cache, hard drive, CD-ROM drive, tape drive, or removable storage disk. Communication device 860 can include any suitable device capable of transmitting and receiving signals over a network, such as a network interface chip or card. The components of the computer can be connected in any suitable manner, such as via a physical bus or wirelessly. Storage 840 can be a non-transitory computer-readable storage medium comprising one or more programs, which, when executed by one or more processors, such as processor 810, cause the one or more processors to execute methods described herein.
Software 850, which can be stored in storage 840 and executed by processor 810, can include, for example, the programming that embodies the functionality of the present disclosure (e.g., as embodied in the systems, computers, servers, and/or devices as described above). In some embodiments, software 850 can include a combination of servers such as application servers and database servers.
Software 850 can also be stored and/or transported within any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device. In the context of this disclosure, a computer-readable storage medium can be any medium, such as storage 840, that can contain or store programming for use by or in connection with an instruction execution system, apparatus, or device.
Software 850 can also be propagated within any transport medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device. In the context of this disclosure, a transport medium can be any medium that can communicate, propagate, or transport programming for use by or in connection with an instruction execution system, apparatus, or device. The transport-readable medium can include but is not limited to, an electronic, magnetic, optical, electromagnetic, or infrared wired or wireless propagation medium.
Computer 800 may be connected to a network, which can be any suitable type of interconnected communication system. The network can implement any suitable communications protocol and can be secured by any suitable security protocol. The network can comprise network links of any suitable arrangement that can implement the transmission and reception of network signals, such as wireless network connections, T1 or T3 lines, cable networks, DSL, or telephone lines.
Computer 800 can implement any operating system suitable for operating on the network. Software 850 can be written in any suitable programming language, such as C, C++, Java, or Python. In various embodiments, application software embodying the functionality of the present disclosure can be deployed in different configurations, such as in a client/server arrangement or through a Web browser as a Web-based application or Web service, for example.
Following is a list of embodiments:
This application incorporates by reference the entire contents of the U.S. Patent Application titled “AI-AUGMENTED AUDITING PLATFORM INCLUDING TECHNIQUES FOR AUTOMATED ASSESSMENT OF VOUCHING EVIDENCE”, filed Jun. 30, 2022, Attorney Docket no. 13574-20068.00.
This application incorporates by reference the entire contents of the U.S. Patent Application titled “AI-AUGMENTED AUDITING PLATFORM INCLUDING TECHNIQUES FOR AUTOMATED ADJUDICATION OF COMMERCIAL SUBSTANCE, RELATED PARTIES, AND COLLECTABILITY”, filed Jun. 30, 2022, Attorney Docket no. 13574-20069.00.
This application incorporates by reference the entire contents of the U.S. Patent Application titled “AI-AUGMENTED AUDITING PLATFORM INCLUDING TECHNIQUES FOR APPLYING A COMPOSABLE ASSURANCE INTEGRITY FRAMEWORK”, filed Jun. 30, 2022, Attorney Docket no. 13574-20070.00.
This application incorporates by reference the entire contents of the U.S. Patent Application titled “AI-AUGMENTED AUDITING PLATFORM INCLUDING TECHNIQUES FOR AUTOMATED DOCUMENT PROCESSING”, filed Jun. 30, 2022, Attorney Docket no. 13574-20071.00.
This application claims the benefit of U.S. Provisional Application No. 63/217,119 filed Jun. 30, 2021; U.S. Provisional Application No. 63/217,123 filed Jun. 30, 2021; U.S. Provisional Application No. 63/217,127 filed Jun. 30, 2021; U.S. Provisional Application No. 63/217,131 filed Jun. 30, 2021; and U.S. Provisional Application No. 63/217,134, filed Jun. 30, 2021, the entire contents of each of which are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 63217119 | Jun 2021 | US | |
| 63217123 | Jun 2021 | US | |
| 63217127 | Jun 2021 | US | |
| 63217131 | Jun 2021 | US | |
| 63217134 | Jun 2021 | US |
