Providing full data provenance visualization for versioned datasets

Abstract

Systems and methods for providing full data provenance visualization for versioned datasets. A method includes receiving selection of a versioned dataset that is within a data pipeline system. The method also includes determining the full data provenance of the selected versioned dataset. The full data provenance may comprise a set of versioned datasets. The method further includes providing for display of a visualization of the full data provenance of the selected versioned dataset. The visualization comprises a graph. The graph comprises a compound node for the selected versioned dataset and for each versioned dataset in the set of versioned datasets. The graph further comprises edges connecting the compounds nodes. Each edge represents a derivation dependency between versions of the versioned datasets represented by the compound nodes connected by the edge.

Description

CROSS-REFERENCE TO RELATED APPLICATION

This application is related to U.S. patent application Ser. No. 14/533,433, entitled “HISTORY PRESERVING DATA PIPELINE SYSTEM AND METHOD,” and filed Nov. 5, 2014, the entire contents of which is hereby incorporated by reference as if fully set forth herein.

TECHNICAL FIELD

The subject innovations relate to graphical user interfaces for computer systems and, in particular, relates to providing full data provenance visualization for versioned datasets.

BACKGROUND

Computers are very powerful tools for processing data. A computerized data pipeline is a useful mechanism for processing large amounts of data. A typical data pipeline is an ad-hoc collection of computer software scripts and programs for processing data extracted from “data sources” and for providing the processed data to “data sinks”. As an example, a data pipeline for a large insurance company that has recently acquired a number of smaller insurance companies might extract policy and claim data from the individual database systems of the smaller insurance companies, transform and validate the insurance data in some way, and provide validated and transformed data to various analytical platforms for assessing risk management, compliance with regulations, fraud, etc.

Between the data sources and the data sinks, a data pipeline system is typically provided as a software platform to automate the movement and transformation of data from the data sources to the data sinks. In essence, the data pipeline system shields the data sinks from having to interface with the data sources or even being configured to process data in the particular formats provided by the data sources. Typically, data from the data sources received by the data sinks is processed by the data pipeline system in some way. For example, a data sink may receive data from the data pipeline system that is a combination (e.g., a join) of data of from multiple data sources, all without the data sink being configured to process the individual constituent data formats.

One purpose of a data pipeline system is to execute data transformation steps on data obtained from data sources to provide the data in format expected by the data sinks. A data transformation step may be defined as a set of computer commands or instructions (e.g., a database query) which, when executed by the data pipeline system, transforms one or more input datasets to produce one or more output or “target” datasets. Data that passes through the data pipeline system may undergo multiple data transformation steps. Such a step can have dependencies on the step or steps that precede it. One example of a computer system for carrying out data transformation steps in a data pipeline is the well-known MapReduce system. See, e.g., Dean, Jeffrey, et al., “MapReduce: Simplified Data Processing on Large Clusters”, Google, Inc., 2004. Another more recent example of a computer system for carrying out data transformation steps in a data pipeline is the Spark system. See, e.g., Zaharia, et al., “Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing”, 9^th USENIX Symposium on Networked Systems Design and Implementation, 2012.

An important issue for users of data pipeline systems is provenance. In the context of data pipeline systems, provenance is metadata that describes the origins and history of datasets in their life cycles. Such metadata (sometimes also called “lineage”) is important for many data pipeline tasks. In particular, provenance is important to users to help them judge whether a given dataset produced by the data pipeline system is trustworthy.

Given the increasing amount of data collected by businesses and other organizations, processing data of all sorts through data pipeline systems can only be expected to increase. This trend is coupled with a need for users to be able to visualize the provenance of datasets produced by data pipeline systems.

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

SUMMARY

In one aspect, the subject innovations are embodied in a method for providing full data provenance visualization of versioned datasets. The method is performed at one or more computing devices having one or more processors and memory storing one or more programs executed by the one or more processors to perform the method. The method includes receiving selection of a versioned dataset that is within a data pipeline system. The method also includes determining full data provenance of the selected versioned dataset. The full data provenance comprises a set of versioned datasets. The method also includes providing for display of a visualization of the full data provenance of the selected versioned dataset. The visualization comprises a graph. The graph comprises a compound node for the selected versioned dataset and for each versioned dataset in the set of versioned datasets. The graph further comprises edges connecting the compounds nodes. Each edge represents a derivation dependency between versions of the versioned datasets represented by the compound nodes connected by the edge.

These and other embodiments of the subject innovations include one or more of the following features: The compound node of the selected versioned dataset may indicate a name or identifier of the selected version dataset. The compound node for each versioned dataset in the set of versioned datasets may indicate a name or identifier of the each versioned dataset. The compound node of the selected versioned dataset may comprise a sub-entry representing a particular version of the selected versioned dataset. The compound node for each versioned dataset in the set of versioned datasets may comprise at least one sub-entry representing a version of the each versioned dataset in the full data provenance of the selected versioned dataset. A sub-entry of the compound node for a particular versioned dataset in the set of versioned datasets may be visually distinguished in the graphical user interface from other sub-entries of compound nodes of the graph to indicate that a version of the particular versioned dataset represented by the sub-entry has been flagged in a database as containing invalid data. An edge in the graph representing a derivation dependency of a first version of a first versioned dataset in the set of versioned datasets on a second version of a second versioned dataset in the set of versioned datasets may be visually distinguished from other edges in the graph to indicate that the first version of the first versioned dataset potentially contains invalid data as a result of the derivation dependency. At least one version of a versioned dataset in the set of versioned datasets may contain data generated as a result of a Spark system executing a derivation program taking at least one version of another versioned dataset as input. At least one version of a versioned dataset in the set of versioned datasets may contain data generated as a result of a MapReduce system executing a derivation program taking at least one version of another versioned dataset as input.

In one aspect, the subject innovations are embodied in one or more non-transitory computer-readable media storing one or more programs. The one or more programs comprise instructions for receiving selection of a versioned dataset that is within a data pipeline system. The one or more programs further comprise instructions for determining full data provenance of the selected versioned dataset. The full data provenance comprises a set of versioned datasets. The one or more programs further comprise instructions for providing for display of a visualization of the full data provenance of the selected versioned dataset. The visualization comprises a graph. The graph comprises a compound node for the selected versioned dataset and for each versioned dataset in the set of versioned datasets. The graph further comprises edges connecting the compounds nodes. Each edge represents a derivation dependency between versions of the versioned datasets represented by the compound nodes connected by the edge.

In one aspect, the subject innovations are embodied in a system comprising memory, one or more processors, and one or more programs stored in the memory and configured for execution by the one or more processors. The one or more programs comprise instructions for receiving selection of a versioned dataset that is within a data pipeline system. The one or more programs further comprise instructions for determining full data provenance of the selected versioned dataset. The full data provenance comprises a set of versioned datasets. The one or more programs further comprise instructions for providing for display of a visualization of the full data provenance of the selected versioned dataset. The visualization comprises a graph. The graph comprises a compound node for the selected versioned dataset and for each versioned dataset in the set of versioned datasets. The graph further comprises edges connecting the compounds nodes. Each edge represents a derivation dependency between versions of the versioned datasets represented by the compound nodes connected by the edge.

It is understood that other configurations of the subject innovations will become readily apparent to those skilled in the art from the following detailed description, wherein various configurations of the subject innovations are shown and described by way of illustration. As will be realized, the subject innovations are capable of other and different configurations and its several details are capable of modification in various other respects, all without departing from the scope of the subject innovations. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not as restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

The features of the subject innovations are set forth in the appended claims. However, for purpose of explanation, several aspects of the disclosed subject matter are set forth in the following figures.

FIG. 1 illustrates an example of a computer system configured to provide full data provenance visualization of versioned datasets.

FIG. 2 illustrates an example graphical user interface configured to provide full data provenance visualization of versioned datasets.

FIG. 3 illustrates an example graphical user interface configured to provide full data provenance visualization of versioned datasets.

FIG. 4 illustrates an example process by which full data provenance visualization for versioned datasets is provided.

FIG. 5 is a very general block diagram of a computing device in which software-implemented processes of the subject innovations may be embodied.

FIG. 6 is a block diagram of a basic software system for controlling the operation of the computing device.

DETAILED DESCRIPTION

The detailed description set forth below is intended as a description of various configurations of the subject innovations and is not intended to represent the only configurations in which the subject innovations may be practiced. The appended drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details for the purpose of providing a thorough understanding of the subject innovations. However, the subject innovations are not limited to the specific details set forth herein and may be practiced without these specific details. In some instances, some structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject innovations.

Glossary

The following definitions are offered for purposes of illustration, not limitation, in order to assist with understanding the discussion that follows.

MapReduce: MapReduce is a programming model and an associated implementation for processing and generating large datasets with a parallel, distributed algorithm on a cluster. See, e.g., Dean, Jeffrey, et al., “MapReduce: Simplified Data Processing on Large Clusters”, Google, Inc., 2004, the entire contents of which is hereby incorporated by reference as if fully set forth herein. APACHE HADOOP is a well-known open source implementation of MapReduce.

Spark: Like MapReduce, Spark is a programming model and an associated implementation for processing and generating large datasets with a parallel, distributed algorithm on a cluster. However, Spark is optimized for data-intensive applications that reuse a working set across multiple parallel operations including iterative jobs and interactive analytics. See, e.g., Zaharia, et al., “Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing”, 9^th USENIX Symposium on Networked Systems Design and Implementation, 2012, the entire contents of which is hereby incorporated by reference as if fully set forth herein. APACHE SPARK is a well-known open source implementation of Spark.

General Overview

As noted above, it may be useful to a user of a data pipeline system to visualize the full data provenance of a versioned dataset. As used herein, the term “full data provenance” of a given versioned dataset encompasses at least all other versioned datasets from which the given versioned dataset is derived and may also include any further versioned datasets in the full data provenance of the other versioned datasets from which the given versioned dataset is derived. For example, if versioned dataset A is derived from versioned datasets B and C, versioned dataset C is derived from versioned dataset D, and versioned dataset D is derived from versioned dataset E, then the full data provenance of versioned dataset A encompasses at least versioned datasets B and C but may also include versioned dataset D and/or versioned dataset E.

As a practical matter, a version of a dataset in a typical data pipeline system may be derived from versions of one or more other datasets, and those datasets each derived from versions of one or more further datasets, and so forth creating a situation where the validity of the dataset version depends on the validity of tens, hundreds, or more other dataset versions. It may be difficult within existing data pipeline systems to discover the full data provenance of a given dataset version and determine whether the dataset version is based on an invalid dataset version. For example, user Alice may flag dataset version D1 as invalid and user Bob may want to know if dataset version X1 is based on dataset version D1. A data pipeline system vendor or other software vendor may wish to assist Bob in discovering the full data provenance of dataset version X1 and help Bob determine if dataset version X1 is based on an invalid dataset version D1. As the foregoing illustrates, an approach for providing a visualization of the full data provenance of a dataset version may be desirable.

The subject innovations relate to providing a visualization of the full data provenance of a dataset version that is within a data pipeline system. In one implementation, a server may receive selection of a dataset within the data pipeline system. For example, a user may direct user input to a graphical user interface at a client computing device that selects the dataset from among other possible selectable datasets, and the selection may be transmitted to the server. The server may determine the full data provenance of the selected dataset. The full data provenance of the selected dataset may include a set of zero or more other datasets. The set may include at least any other datasets from which a version of the selected dataset is derived from in addition to any datasets from which those datasets are derived from and so on. The server may provide for display of a visualization of the full data provenance of the version of the selected dataset. The visualization may be displayed in a graphical user interface at the client computing device. For example, the visualization may be displayed within a web browser window or within an application window.

The visualization may include a graph. The graph may include a compound node for the selected dataset and a compound node for each dataset in the set of datasets of the full data provenance of the selected dataset. For example, the compound node for a dataset may include a sub-entry for each version of the dataset involved in the full data provenance of the selected dataset. For example, version one of selected dataset E might be derived from version one of dataset C and version twenty-nine of dataset D. Version one of dataset C might be derived, at least in part, from version five of dataset B. And version twenty-nine of dataset D might be derived, at least in part, from version twelve of dataset B. In this case, the compound node for dataset B may have at least two sub-entries: one sub-entry representing version five of dataset B and another sub-entry representing version twelve of dataset B.

The graph further comprises edges connecting the compounds nodes. Each edge represents a derivation dependency between versions of the versioned datasets represented by the compound nodes connected by the edge. For example, an edge connecting a sub-entry of the compound node for dataset B and a sub-entry of the compound node for dataset D may represent a derivation dependency between version twenty-nine of dataset D and version twelve of dataset B.

The sub-entry of a compound node corresponding to a dataset version that has been flagged or marked invalid may be highlighted or visually distinguished in the visualization. For example, assume version twelve of dataset B fails a dataset validation process and as a result is flagged or marked invalid in a database. In this case, the sub-entry of the compound node representing version twelve of dataset B, the edge connecting that sub-entry to the sub-entry of the compound node representing version twenty nine of dataset D, the sub-entry of the compound node representing version twenty nine of dataset D, the edge connecting the sub-entry of the compound node representing version twenty nine of dataset D to version one of dataset E, and the compound node representing dataset E all may be colored red or visually distinguished in some way to indicate that version twelve of dataset B contains invalid data and that version twenty nine of dataset D and version one of dataset E may contain invalid data as a result of version twelve of dataset B containing invalid data.

Advantageously, in some implementations of the subject innovations, a user is informed of the full data provenance of a selected dataset version. As a result, the user may more easily identify derivation dependencies between dataset versions including derivation dependencies involving invalid dataset versions.

Example Computer System

FIG. 1 illustrates an example of a computer system 100 configured to provide full data provenance visualization of versioned datasets. As shown, the computer system 100 includes a client computing device 112 used by a human user 110, a server 130, a database 140, a data pipeline system 150, and a distributed file system 160. The client computing device 112 and the server 130 may be configured to communicate with one another via a network 120. The network 120 may include the Internet, an intranet, a local area network, a wide area network, a wired network, a wireless network, a virtual private network (VPN).

The client computing device 112 may be a laptop computer, a desktop computer, a mobile phone, a personal digital assistant (PDA), a tablet computer, a netbook, a television with one or more processors, embedded therein or coupled thereto, a physical machine or a virtual machine. The client computing device 112 may include one or more of a keyboard, a mouse, a display 114, or a touch screen (of which display 114 may be a part of). For example, the client computing device 112 may be composed of hardware components like those of basic computing device 500 described below with respect to FIG. 5 and configured with a basic software system like software system 600 described below with respect to FIG. 6. The client computing device 112 may also include a web browser or a client application configured to display, in a graphical user interface 116 of the client computing device 112 display 114, a visualization of the full provenance of a selected dataset version in accordance with this disclosure of the subject innovations. The graphical user interface 116 may be a web browser window, a client application window, an operating system window, or other computer graphical user interface window. While only one user 110 and one client computing device 112 are illustrated in FIG. 1, the subject innovations may be implemented in conjunction with one or more users 110 and one or more client computing devices 112.

The server 130 may include a full data provenance visualization module to provide a visualization of the full data provenance of a selected dataset version, based on provenance metadata 142 stored in database 140. The server 130 may be implemented as a single server computing device or as multiple server computing devices arranged in a distributed or clustered computing arrangement. Each such server computing device may be composed of hardware components like those of basic computing device 500 described below with respect to FIG. 5 and configured with a basic software system like software system 600 described below with respect to FIG. 6.

The server 130 may include one or more processors (e.g., CPUs), a network interface, and memory. The processor(s) may be configured to execute computer instructions that are stored in one or more computer-readable media, for example, the memory of the server 130. The server 130 may include a network interface that is configured to allow the server 130 to transmit and receive data in a network, e.g., network 120 of FIG. 1. The network interface may include one or more network interface cards (NICs). The memory of the server 130 may store data or instructions. The instructions stored in the memory may include the full data provenance visualization module.

The database 140 may include a database server module for storing and retrieving database data including provenance metadata 142 and derivation programs 144. The database 140 may be implemented as a single server computing device or as multiple server computing devices arranged in a distributed or clustered computing arrangement. Each such server computing device may be composed of hardware components like those of basic computing device 500 described below with respect to FIG. 5 and configured with a basic software system like software system 600 described below with respect to FIG. 6.

The database 140 may include one or more processors (e.g., CPUs), a network interface, and memory. The processor(s) may be configured to execute computer instructions that are stored in one or more computer-readable media, for example, the memory of the database 140. The database 140 may include a network interface that is configured to allow the database 140 to transmit and receive data in one or more networks, e.g., a network connecting the server 130 and the database 140 and a network connecting the data pipeline system 150 to the database 140, which may be the same or different network as the network that connects the server 130 and the database 140. The network interface may include one or more network interface cards (NICs). The memory of the database 140 may store data or instructions. The instructions stored in the memory may include the database server module.

The data pipeline system 150 may include a dataset derivation module to derive dataset 162 versions from other dataset 162 versions by executing derivation programs 144. The data pipeline system 150 may also include a provenance metadata update module for updating provenance metadata 142 in database 140 when new dataset 162 versions are derived. The data pipeline system 150 may be implemented as a single server computing device or as multiple server computing devices arranged in a distributed or clustered computing arrangement. Each such server computing device may be composed of hardware components like those of basic computing device 500 described below with respect to FIG. 5 and configured with a basic software system like software system 600 described below with respect to FIG. 6.

The data pipeline system 150 may include one or more processors (e.g., CPUs), a network interface, and memory. The processor(s) may be configured to execute computer instructions that are stored in one or more computer-readable media, for example, the memory of the data pipeline system 150. The data pipeline system 150 may include a network interface that is configured to allow the data pipeline system 150 to transmit and receive data in a network, e.g., a network connecting the data pipeline system 150 and the database 140 and a network connecting the data pipeline system 150 to the distributed file system 160, which may be the same or different network as the network that connects the data pipeline system 150 and the database 140. The network interface may include one or more network interface cards (NICs). The memory of the database 140 may store data or instructions. The instructions stored in the memory may include the dataset derivation module and the provenance metadata update module. In an exemplary non-limiting embodiment, the dataset derivation module is implemented at least in part by an implementation of the MapReduce system, for example, APACHE HADOOP. In an exemplary non-limiting embodiment, the dataset derivation module is implemented at least in part by an implementation of the Spark system, for example, APACHE SPARK.

The distributed file system 160 may include a distributed file system module to provide distributed file system services to the data pipeline system 150 over a network that connects the distributed file system 160 and the data pipeline system 150. The distributed file system 160 may be implemented as a single server computing device or as multiple server computing devices arranged in a distributed or clustered computing arrangement. Each such server computing device may be composed of hardware components like those of basic computing device 500 described below with respect to FIG. 5 and configured with a basic software system like software system 600 described below with respect to FIG. 6.

The distributed file system 160 may include one or more processors (e.g., CPUs), a network interface, and memory. The processor(s) may be configured to execute computer instructions that are stored in one or more computer-readable media, for example, the memory of the distributed file system 160. The distributed file system 160 may include a network interface that is configured to allow the distributed file system 160 to transmit and receive data in a network, e.g., a network connecting distributed file system 160 and the data pipeline system 150. The network interface may include one or more network interface cards (NICs). The memory of the distributed file system 160 may store data or instructions. The instructions stored in the memory may include the distributed file system module. In an exemplary non-limiting embodiment, the distributed file system module is implemented by the APACHE HADOOP Distributed File System (HDFS) configured on a cluster of commodity server computing devices.

The full data provenance visualization module of the server 130 is configured to provide a visualization of the full data provenance of a selected versioned dataset 162, based on provenance metadata 142 stored in database 140. The selected version dataset 162 may be stored within the data pipeline system 150. The data pipeline system 150 includes the distributed file system 160. The visualization may include a graph. The graph may comprise compound nodes and edges connecting the compound nodes in the graph. Each of the compound nodes may represent the selected dataset 162 or a dataset 162 in the full data provenance of the selected dataset 162. Each edge represents a derivation dependency between a version of a dataset 162 and a version of another dataset 162.

A dataset 162 is a logical collection of highly structured, semi-structured, or unstructured data. A non-limiting example of highly structured data is data that conforms to a standardized or well-known data model, for example, a relational model or other table-based data model. A non-limiting example of semi-structured data is data that has self-describing structure, for example, eXtensible Markup Language (XML) data or Javascript Object Notation (JSON) data. A non-limiting example of unstructured data is data that is not highly structured or semi-structured data, for example, some text data or log data. Each dataset 162 version may be stored in one or more files in the distributed file system 160.

A derivation dependency may exist between two versions of two datasets 162 if one of the two versions was derived from the other of the two versions within the data pipeline system 150. In particular, a version of a “target” dataset 162 may be derived by the data pipeline system 150 from one or more versions of one or more “input” datasets 162. In doing so, the data pipeline system 150 may provide the version(s) of the input dataset(s) 162 as input to the derivation program 144. The derivation program 144, in conjunction with the data pipeline system 150, may produce the version of the target dataset 162 as output. In this case, the version of the target dataset 162 has a derivation dependency on each of the version(s) of the input dataset(s) 162. Such derivation dependencies may be stored by the data pipeline system 150 in database 140 as part of the provenance metadata 142.

Database 140 may store one or more derivation programs 144. A derivation program 144 may include instructions for extracting (e.g., selecting) and transforming data from version(s) of one or more datasets 162 input to derivation program 144. The extracted and transformed data may be stored as a new dataset 162 version in the distributed file system 160. The derivation program 144 itself may specify the versions of the dataset(s) 162 that are to be the input to the derivation program 144 when executed. Alternatively, a user may specify the versions of the dataset(s) 162 that are to be the input to an execution of the derivation program 144. The derivation program 144 may be executed by the dataset derivation module of the data pipeline system 150. The derivation program 144 may include a variety of different high-level query language instructions depending on whether the dataset derivation module is a MapReduce-based or Spark-based. For example, if the dataset derivation module is MapReduce-based, then the derivation program 144 may include, for example, MapReduce instructions that invoke an APACHE HADOOP MapReduce Application Programming Interface (API), APACHE PIG instructions, APACHE HIVE instructions, Jaql instructions, or other instructions for carrying out MapReduce operations on datasets 162. If the dataset derivation module is Spark-based, then the derivation program 144 may include, for example, Scala, Java, Python, Clojure, or R instructions for carrying out Spark transformations on datasets 162. While derivation programs 144 are shown in FIG. 1 as being stored in database 140, derivation programs 144 may be stored in another location, for example, in the distributed file system 160 or in a different database.

Provenance metadata 142 comprises information about the full data provenance of dataset 162 versions. For a given dataset 162 within the data pipeline system 150, provenance metadata 142 may include all of the following information about the given dataset 162, or a subset or a superset thereof:

• • A name or unique identifier of the given dataset 162.
  • An identifier of each version of the given dataset 162 within the data pipeline system 150.
  • An identifier of the current version of the given dataset 162 within the data pipeline system 150.

For each version of the given dataset 162 within the data pipeline system 150, the provenance metadata 142 may include all of the following information about the given dataset 162 version, or a subset or a superset thereof:

• • The identifier of the version of the given dataset 162 version.
  • If the given dataset 162 version was derived from one or more other dataset 162 versions, then, for each such other dataset 162, the name or unique identifier of the other dataset 162 and the identifier of the version of the other dataset 162.
  • If the given dataset 162 version was derived from one or more other dataset 162 versions, the name or identifier of the derivation program 144 executed by the data pipeline system 150 to derive the given dataset 162 version. In some implementations, derivation programs 144 are versioned and the provenance metadata 142 includes the identifier of the version of the derivation program 144 executed by the data pipeline system 150 to derive the given dataset 162 version.
  • A flag (e.g., a dirty bit) that indicates that the given dataset 162 version contains invalid data. The flag may be set as a result of the given dataset 162 version failing a data validation process, for example.

In one example, the full data provenance visualization module of the server 130 is implemented in software. The full data provenance visualization module may include code for receiving selection of a versioned dataset 162 within the data pipeline system 150. The data pipeline system 150 includes the distributed file system 160. The selection may be received by the full data provenance visualization module of the server 130 over network 120 from client computing device 112 (e.g., in a HTTP or HTTPS request) as a result of the user 110 interacting with a graphical user interface 116 presented on the display 114. The selection may be for just a dataset 162 or for a particular version of a dataset 162. If the selection is for just a dataset 162, then a particular version of the dataset 162 may be selected by the full data provenance visualization module based on the selection. For example, the full data provenance visualization module may select, as the particular version of the dataset 162 to provide a full data provenance visualization of, the current version of the selected dataset 162 or the most recent version of the selected dataset 162 as indicated in the provenance metadata 142.

The full data provenance visualization module may further include code for determining full data provenance of the particular version of the selected versioned dataset 162. The full data provenance may include a set of zero or more versioned datasets 162. The set may include no versioned datasets 162 if the particular version of the selected dataset 162 is not derived from any other datasets 162. For example, the particular version of the selected dataset 162 may have been stored in the distributed file system 160 by an external data source and not generated by the data pipeline 150 as a result of executing a derivation program 144. As another example, the particular version of the selected dataset 162 may have been generated by the data pipeline 150 as a result of executing a derivation program 144 that did not accept any other dataset 162 versions as input.

To determine the full data provenance of the particular version of the selected dataset 162, the full data provenance visualization module may consult the provenance metadata 162 in the database 140. In particular, the full data provenance visualization module may start the determination with an empty set of dataset 162 versions representing the full data provenance of the particular version of the selected dataset 162. The determination may then include the full data provenance visualization module consulting the provenance metadata 162 to determine all dataset 162 versions from which the particular version of the selected dataset 162 was derived and adding those dataset 162 versions to the set of versioned datasets 162 representing the full data provenance of the particular version of the selected dataset 162. The full data provenance visualization module may then repeat this determination for each of those dataset 162 versions just added to the set and so on in a recursive or iterative manner, adding any dataset 162 versions from which a dataset 162 version in the full data provenance of the particular version of the selected dataset 162 was derived to the set of dataset 162 versions representing the full data provenance of the particular version of the selected dataset 162. The recursion or iteration may end when all dataset 162 versions, according to the provenance metadata 142, in the full data provenance of the particular version of the selected dataset 162 have been determined and added to the set, or when a stop condition is reached. The stop condition may be based on a threshold degree of derivation between the particular version of the selected dataset 162 and a dataset 162 version in the full data provenance of the particular version of the selected dataset 162. For example, if the threshold degree of derivation is ten, then only dataset 162 versions in the full data provenance of the particular version of the selected dataset 162 that are within ten degrees derivation of the particular version of the selected dataset 162 will be added to the set of versioned datasets 162 representing the full data provenance of the particular version of the selected dataset 162.

The full data provenance visualization module may further include code for providing for display (e.g., via a web browser on the client computing device 112) of a visualization of the determined full data provenance of the particular version of the selected dataset 162. The visualization may include a graph. The graph may include a compound node for the selected dataset 162 and for each dataset 162 in the set of dataset 162 versions representing the full data provenance of the particular version of the selected dataset 162. The graph may include directed edges connected the compound nodes. Each directed edge may represent a derivation dependency between versions of the versioned datasets 162 represented by the compound nodes connected by the edge. Each compound node may include a sub-entry for each version of the dataset 162 represented by the compound node in the set of dataset 162 versions representing the full data provenance of the particular version of the selected dataset 162.

Example Graphical User Interfaces

FIG. 2 illustrates an example graphical user interface window 200 (e.g., a web browser window) configured to provide full data provenance visualization of versioned datasets.

The window 200 may be displayed via the display 114 (e.g., a screen) of the client computing device 112. As shown, the window 200 includes a graph 202.

The graph 202 represents the determined full data provenance of version one of selected dataset E. As indicated by the graph 202, version one of selected dataset E has a derivation dependency on version one of dataset C and version twenty nine of dataset D. Version one of dataset C has a derivation dependency on version one of dataset A and version five of dataset B. Version twenty nine of dataset D has a derivation dependency on versions seven and twelve of dataset B. Versions five, seven, and twelve of dataset B each have a derivation dependency on version two of dataset A. Since there are three versions of dataset B in the full data provenance of version one of dataset E, there are three sub-entries of the compound node representing dataset B in the graph 202. The three sub-entries represent the three versions, respectively. The remaining compound nodes have only one sub-entry as only one version of each of the remaining datasets is in the full data provenance of version one of selected dataset E.

A compound node in a graph representing a dataset may indicate the name or identifier of the dataset. For example, the compound node representing dataset B in graph 202 is labeled with the text “B”.

The compound node in a graph representing the selected dataset may indicate the selected dataset. For example, the compound node representing selected dataset E in graph 202 is labeled with the text “(TARGETED NODE)” to indicate that dataset E is the selected dataset for which the full data provenance is visualized in GUI 200. The compound node representing the selected dataset may be colored differently (or otherwise visually distinguished) from the other compound nodes in the graph to indicate the selected dataset.

A sub-entry of a compound node representing a version of a dataset may include metadata about the version of the dataset. For example, the sub-entry of the compound node representing version one of dataset C in graph 202 indicates that version number (e.g., “V 1”), the name of a user of a data pipeline system that caused the data pipeline system to create version one of dataset C (e.g., “Jane Smith”), and the date version of dataset C was created (e.g., “Monday”).

FIG. 3 illustrates an example graphical user interface window 300 (e.g., a web browser window) configured to provide full data provenance visualization of versioned datasets.

The window 200 may be displayed via the display 114 (e.g., a screen) of the client computing device 112. As shown, the window 300 includes a graph 302.

The graph 302 represents the determined full data provenance of version one of selected dataset E. As indicated by the graph 302, version one of selected dataset E has a derivation dependency on version one of dataset C and version twenty nine of dataset D. Version one of dataset C has a derivation dependency on version one of dataset A and version five of dataset B. Version twenty nine of dataset D has a derivation dependency on versions seven and twelve of dataset B. Versions five, seven, and twelve of dataset B each have a derivation dependency on version two of dataset A. Since there are three versions of dataset B in the full data provenance of version one of dataset E, there are three sub-entries of the compound node representing dataset B in the graph 302. The three sub-entries represent the three versions, respectively. The remaining compound nodes have only one sub-entry as only one version of each of the remaining datasets is in the full data provenance of version one of selected dataset E.

In this example, version five of dataset B has been flagged as invalid. For example, version five of dataset B may have failed a data validation process. As a result, the sub-entry representing version five of dataset B is colored differently (or otherwise visually distinguished) from other sub-entries in the graph 302 to indicate that the dataset version contains invalid data.

Other sub-entries representing “downstream” dataset versions may also be colored differently (or otherwise visually distinguished) to indicate that they may also contain invalid data as a result of invalid data in a dataset version. For example, since version one of dataset C has a derivation dependency on invalid version five of dataset B and version one of dataset E has a derivation dependency on potentially invalid version one of dataset C, the sub-entries representing version one of dataset C and version one of dataset E may be colored differently (or otherwise visually distinguished) to indicate that they may potentially contain invalid data as a result of the invalid data in version five of dataset B.

An edge connecting a sub-entry representing a potentially invalid dataset version to a sub-entry representing an invalid dataset version may be colored differently (or otherwise visually distinguished) from other edges in a graph to indicate that the potentially invalid dataset version has a derivation dependency on an invalid dataset version. For example, the edge in graph 302 connecting the sub-entry representing version one of dataset C with the sub-entry representing version five of dataset B may be colored differently (or otherwise visually distinguished) from other edges in the graph 302 to indicate that the potentially invalid version one of dataset C has a derivation dependency on an invalid version five of dataset B.

An edge connecting a sub-entry representing a potentially invalid dataset version to a sub-entry representing another potentially invalid dataset version may be colored differently (or otherwise visually distinguished) from other edges in a graph to indicate that the potentially invalid dataset version has a derivation dependency on another potentially invalid dataset version. For example, the edge in graph 302 between the sub-entry representing version one of dataset E and the sub-entry representing version one of dataset C may be colored differently (or otherwise visually distinguished) from other edges in the graph 302 to indicate that the potentially invalid version one of dataset E has a derivation dependency on potentially invalid version one of dataset C.

While in some embodiments as exemplified in FIG. 3, an edge in a graph representing a derivation dependency between dataset versions is directed from a sub-entry of a compound node representing an input dataset version to a derivation program to a sub-entry of a compound representing the output dataset version from the derivation program, the edge is directed from the sub-entry representing the output dataset version to the sub-entry representing the input dataset version in other embodiments.

Example Process

FIG. 4 illustrates an example process 400 by which full data provenance visualization for versioned datasets is provided. Process 400 may be performed by software when executed by one or more computing devices. For example, process 400 may be performed by one or more applications 602 executing on one or more computing devices 500, each configured with a software system like software system 600. (See FIGS. 5 and 6 and associated description below). The one or more computing devices on which process 400 executes can be, for example, client 112, server 130, or a combination of client 112 and server 130.

The process 400 beings at step 410, where a server (e.g., server 130) receives selection of a versioned dataset (e.g., 162-1). That versioned dataset may be within a data pipeline system (e.g., data pipeline system 150 and distributed file system 160). In fact, a number of versioned datasets may be within the data pipeline system. Some, but not necessarily all, of the dataset versions within the data pipeline system may be “derived” datasets in that the dataset version is generated by the data pipeline system executing a derivation program (e.g., 144), or a version of a derivation program. When executed, the derivation program may accept one or more other dataset versions as input. In this way, the generated dataset version is derived from the input dataset version(s). In some instances, a dataset version within the data pipeline system is generated as a result of a Spark system executing a derivation program taking a version of at least one other dataset as input to the derivation program. In some instances, a dataset version within the data pipeline system is generated as a result of a MapReduce system executing a derivation program taking a version of at least one other dataset as input to the derivation program.

The server may receive selection of the versioned dataset over a network (e.g., 120). The server may receive the selection from a client computing device (e.g., 112). The received selection may identify the versioned dataset selected. In addition, the received selection may identify a particular version of the selected dataset for which to determine the full data provenance of. If the selection does not identify a particular version of the selected dataset, then the server may assume a default version of the selected dataset. The default version can be selected by the server based on provenance metadata (e.g. 142) for the selected dataset. In some instances, the server selects a current version of the selected dataset as the default version. In some instances, the server selects the latest (more recent) version of the selected dataset as the default version.

In step 420, the server determines the full data provenance of the particular version of the selected dataset. The full data provenance may comprise a set of zero or more other datasets within the data pipeline system. For example, GUI 300 of FIG. 3 shows that datasets A, B, C, and D are in the full data provenance of version one of dataset E. In particular, version twenty nine of dataset D, version one of dataset C, versions five, seven, and twelve of dataset B, and versions one and two dataset A are in the full data provenance of version one of dataset E. The determination of the full data provenance of the particular version of the selected dataset may be based on the provenance metadata for datasets as maintained by the data pipeline system.

In step 430, the server provides for display (e.g., via a web browser window on the client computing device) of a visualization of the full data provenance of the particular version of the selected dataset. The visualization comprises a graph (e.g., 202 and 302). The graph may comprise a compound node for the selected dataset and a compound node for each versioned dataset in the set of versioned datasets determined in step 420. The graph may further comprise edges connecting the compounds nodes where each edge represents a derivation dependency between versions of the versioned datasets represented by the compound nodes connected by the edge.

In some instances, if a particular version of a particular dataset in the full data provenance of the particular version of the selected dataset is flagged or marked as invalid in the provenance metadata, then the sub-entry of the compound node representing the particular version of the particular dataset may be colored differently or otherwise visually distinguished in the graph from other sub-entries to indicate that the particular version of the particular dataset contains invalid data.

In some instances, if a particular version of a first dataset in the full data provenance of the particular version of the selected dataset has a derivation dependency on a particular version of a second dataset in the full data provenance of the particular version of the selected dataset and the particular version of the second dataset is flagged or marked as invalid in the provenance metadata, then the edge in the graph connecting the sub-entry of the compound node for the particular version of the first dataset to the sub-entry of the compound node for the particular version of the second dataset may be colored differently or otherwise visually distinguished in the graph from other edges to indicate that the particular version of the first dataset potentially contains invalid data as a result of the derivation dependency on the particular version of the second dataset.

In some instances, if a particular version of a first dataset in the full data provenance of the particular version of the selected dataset has a derivation dependency on a particular version of a second dataset in the full data provenance of the particular version of the selected dataset and the particular version of the second dataset potentially contains invalid data as a result of a derivation dependency on a version of a dataset that contains or potentially contains invalid data, then the edge in the graph connecting the sub-entry of the compound node for the particular version of the first dataset to the sub-entry of the compound node for the particular version of the second dataset may be colored differently or otherwise visually distinguished in the graph from other edges to indicate that the particular version of the first dataset potentially contains invalid data as a result of the derivation dependency on the particular version of the second dataset.

Basic Computing Device

Referring now to FIG. 5, it is a block diagram that illustrates a basic computing device 500 in which software-implemented processes of the subject innovations may be embodied. Computing device 500 and its components, including their connections, relationships, and functions, is meant to be exemplary only, and not meant to limit implementations of the subject innovations. Other computing devices suitable for implementing the subject innovations may have different components, including components with different connections, relationships, and functions.

Computing device 500 may include a bus 502 or other communication mechanism for addressing main memory 506 and for transferring data between and among the various components of device 500.

Computing device 500 may also include one or more hardware processors 504 coupled with bus 502 for processing information. A hardware processor 504 may be a general purpose microprocessor, a system on a chip (SoC), or other processor suitable for implementing the subject innovations.

Main memory 506, such as a random access memory (RAM) or other dynamic storage device, also may be coupled to bus 502 for storing information and instructions to be executed by processor(s) 504. Main memory 506 also may be used for storing temporary variables or other intermediate information during execution of software instructions to be executed by processor(s) 504.

Such software instructions, when stored in non-transitory storage media accessible to processor(s) 504, render computing device 500 into a special-purpose computing device that is customized to perform the operations specified in the instructions. The terms “instructions”, “software”, “software instructions”, “program”, “computer program”, “computer-executable instructions”, and “processor-executable instructions” are to be broadly construed to cover any machine-readable information, whether or not human-readable, for instructing a computing device to perform specific operations, and including, but not limited to, application software, desktop applications, scripts, binaries, operating systems, device drivers, boot loaders, shells, utilities, system software, JAVASCRIPT, web pages, web applications, plugins, embedded software, microcode, compilers, debuggers, interpreters, virtual machines, linkers, and text editors.

Computing device 500 also may include read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor(s) 504.

One or more mass storage devices 510 may be coupled to bus 502 for persistently storing information and instructions on fixed or removable media, such as magnetic, optical, solid-state, magnetic-optical, flash memory, or any other available mass storage technology. The mass storage may be shared on a network, or it may be dedicated mass storage. Typically, at least one of the mass storage devices 510 (e.g., the main hard disk for the device) stores a body of program and data for directing operation of the computing device, including an operating system, user application programs, driver and other support files, as well as other data files of all sorts.

Computing device 500 may be coupled via bus 502 to display 512, such as a liquid crystal display (LCD) or other electronic visual display, for displaying information to a computer user. In some configurations, a touch sensitive surface incorporating touch detection technology (e.g., resistive, capacitive, etc.) may be overlaid on display 512 to form a touch sensitive display for communicating touch gesture (e.g., finger or stylus) input to processor(s) 504.

An input device 514, including alphanumeric and other keys, may be coupled to bus 502 for communicating information and command selections to processor 504. In addition to or instead of alphanumeric and other keys, input device 514 may include one or more physical buttons or switches such as, for example, a power (on/off) button, a “home” button, volume control buttons, or the like.

Another type of user input device may be a cursor control 516, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 504 and for controlling cursor movement on display 512. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.

While in some configurations, such as the configuration depicted in FIG. 5, one or more of display 512, input device 514, and cursor control 516 are external components (i.e., peripheral devices) of computing device 500, some or all of display 512, input device 514, and cursor control 516 are integrated as part of the form factor of computing device 500 in other configurations.

Functions of the disclosed systems, methods, and modules may be performed by computing device 500 in response to processor(s) 504 executing one or more programs of software instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device(s) 510. Execution of the software program instructions contained in main memory 506 cause processor(s) 504 to perform the functions of the disclosed systems, methods, and modules.

While in some implementations, functions of the disclosed systems and methods are implemented entirely with software instructions, hard-wired or programmable circuitry of computing device 500 (e.g., an ASIC, a FPGA, or the like) may be used in place of or in combination with software instructions to perform the functions, according to the requirements of the particular implementation at hand.

The term “storage media” as used herein refers to any non-transitory media that store data and/or instructions that cause a computing device to operate in a specific fashion. Such storage media may comprise non-volatile media and/or volatile media. Non-volatile media includes, for example, non-volatile random access memory (NVRAM), flash memory, optical disks, magnetic disks, or solid-state drives, such as storage device 510. Volatile media includes dynamic memory, such as main memory 506. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid-state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, flash memory, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.

Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor(s) 504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid-state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computing device 500 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 502. Bus 502 carries the data to main memory 506, from which processor(s) 504 retrieves and executes the instructions. The instructions received by main memory 506 may optionally be stored on storage device(s) 510 either before or after execution by processor(s) 504.

Computing device 500 also may include one or more communication interface(s) 518 coupled to bus 502. A communication interface 518 provides a two-way data communication coupling to a wired or wireless network link 520 that is connected to a local network 522 (e.g., Ethernet network, Wireless Local Area Network, cellular phone network, Bluetooth wireless network, or the like). Communication interface 518 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information. For example, communication interface 518 may be a wired network interface card, a wireless network interface card with an integrated radio antenna, or a modem (e.g., ISDN, DSL, or cable modem).

Network link(s) 520 typically provide data communication through one or more networks to other data devices. For example, a network link 520 may provide a connection through a local network 522 to a host computer 524 or to data equipment operated by an Internet Service Provider (ISP) 526. ISP 526 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet” 528. Local network(s) 522 and Internet 528 use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link(s) 520 and through communication interface(s) 518, which carry the digital data to and from computing device 500, are example forms of transmission media.

Computing device 500 can send messages and receive data, including program code, through the network(s), network link(s) 520 and communication interface(s) 518. In the Internet example, a server 530 might transmit a requested code for an application program through Internet 528, ISP 526, local network(s) 522 and communication interface(s) 518.

The received code may be executed by processor 504 as it is received, and/or stored in storage device 510, or other non-volatile storage for later execution.

Basic Software System

FIG. 6 is a block diagram of a basic software system 600 that may be employed for controlling the operation of computing device 500. Software system 600 and its components, including their connections, relationships, and functions, is meant to be exemplary only, and not meant to limit implementations of the subject innovations. Other software systems suitable for implementing the subject innovations may have different components, including components with different connections, relationships, and functions.

In various embodiments, software system 600 is provided for directing the operation of computing device 500. Software system 600, which may be stored in system memory (RAM) 506 and on fixed storage (e.g., hard disk or flash memory) 510, includes a kernel or operating system (OS) 610. The OS 610 manages low-level aspects of computer operation, including managing execution of processes, memory allocation, file input and output (I/O), and device I/O. One or more application programs, represented as 602A, 602B, 602C . . . 602N in FIG. 6, may be “loaded” (e.g., transferred from fixed storage 510 into memory 506) for execution by the system 600. The applications or other software intended for use on device 600 may also be stored as a set of downloadable computer-executable instructions, for example, for downloading and installation from an Internet location (e.g., a Web server).

Software system 600 may include a graphical user interface (GUI) 615, for receiving user commands and data in a graphical (e.g., “point-and-click” or “touch gesture”) fashion. These inputs, in turn, may be acted upon by the system 600 in accordance with instructions from operating system 610 and/or application(s) 602. The GUI 615 also serves to display the results of operation from the OS 610 and application(s) 602, whereupon the user may supply additional inputs or terminate the session (e.g., log off).

OS 610 can execute directly on the bare hardware 620 (e.g., processor(s) 504) of device 500. Alternatively, a hypervisor or virtual machine monitor (VMM) 630 may be interposed between the bare hardware 620 and the OS 610. In this configuration, VMM 630 acts as a software “cushion” or virtualization layer between the OS 610 and the bare hardware 620 of the device 500.

VMM 630 instantiates and runs one or more virtual machine instances (“guest machines”). Each guest machine comprises a “guest” operating system, such as OS 610, and one or more applications, such as application(s) 602, designed to execute on the guest operating system. The VMM 630 presents the guest operating systems with a virtual operating platform and manages the execution of the guest operating systems.

In some instances, the VMM 630 may allow a guest operating system to run as if it is running on the bare hardware 620 of device 500 directly. In these instances, the same version of the guest operating system configured to execute on the bare hardware 620 directly may also execute on VMM 630 without modification or reconfiguration. In other words, VMM 630 may provide full hardware and CPU virtualization to a guest operating system in some instances.

In other instances, a guest operating system may be specially designed or configured to execute on VMM 630 for efficiency. In these instances, the guest operating system is “aware” that it executes on a virtual machine monitor. In other words, VMM 630 may provide para-virtualization to a guest operating system in some instances.

The above-described basic computer hardware and software is presented for purpose of illustrating the basic underlying computer components that may be employed for implementing the subject innovations. The subject innovations, however, are not necessarily limited to any particular computing environment or computing device configuration. Instead, the subject innovations may be implemented in any type of system architecture or processing environment that one skilled in the art, in light of this disclosure, would understand as capable of supporting the features and functions of the subject innovations as presented herein.

Extensions and Alternatives

It is understood that any specific order or hierarchy of steps in the processes disclosed is an illustration of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged, or that all illustrated steps be performed. Some of the steps may be performed simultaneously. For example, in certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components illustrated above should not be understood as requiring such separation, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Various modifications to these aspects will be readily apparent, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but is to be accorded the full scope consistent with the language claims, where reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” Unless specifically stated otherwise, the term “some” refers to one or more. Unless specifically stated otherwise, the term “may” is used to express one or more non-limiting possibilities. Headings and subheadings, if any, are used for convenience only and do not limit the subject innovations.

A phrase, for example, an “aspect”, an “embodiment”, a “configuration”, or an “implementation” does not imply that the aspect, the embodiment, the configuration, or the implementation is essential to the subject innovations or that the aspect, the embodiment, the configuration, or the implementation applies to all aspects, embodiments, configurations, or implementations of the subject innovations. A disclosure relating to an aspect, an embodiment, a configuration, or an implementation may apply to all aspects, embodiments, configurations, or implementations, or one or more aspects, embodiments, configurations, or implementations. A phrase, for example, an aspect, an embodiment, a configuration, or an implementation may refer to one or more aspects, embodiments, configurations, or implementations and vice versa.

Claims

1. A method, comprising: at one or more computing devices having one or more processors and memory storing one or more programs executed by the one or more processors to perform the method, performing the operations of:storing an input dataset and provenance metadata identifying one or more previous versions of the input dataset;using a derivation program, transforming data in the input dataset and storing the transformed data as a versioned dataset;updating the provenance metadata to identify the input dataset in addition to the one or more previous versions of the input dataset;receiving selection of the versioned dataset that is within a data pipeline system;determining full data provenance of the selected versioned dataset, the full data provenance comprising a set of versioned datasets, by identifying, in the provenance metadata, at least the input dataset and the one or more previous versions of the input dataset;providing for display of a visualization of the full data provenance of the selected versioned dataset, the visualization comprising a graph, the graph comprising a compound node for the selected versioned dataset and a compound node for each versioned dataset in the set of versioned datasets, the graph further comprising edges connecting the compounds nodes, each edge representing a derivation dependency between versions of the versioned datasets represented by the compound nodes connected by the edge;wherein a sub-entry of the compound node for a particular versioned dataset in the set of versioned datasets is visually distinguished in the graphical user interface from other compound node sub-entries of the graph to indicate that a version, of the particular version dataset represented by the sub-entry has been flagged in a database as containing invalid data;wherein an edge in the graph representing a derivation dependency of a first version of a first versioned dataset in the set of versioned datasets on a second version of a second versioned dataset in the set of versioned datasets is visually distinguished from other edges in the graph to indicate that the first version of the first versioned dataset potentially contains invalid data as a result of the derivation dependency.

2. The method of claim 1, wherein the compound node of the selected versioned dataset indicates a name or identifier of the selected version dataset; and wherein the compound node for each versioned dataset in the set of versioned datasets indicates a name or identifier of the each versioned dataset.

3. The method of claim 1, wherein the compound node for each versioned dataset in the set of versioned datasets comprises at least one sub-entry representing a version of the each versioned dataset in the full data provenance of the selected versioned dataset.

4. The method of claim 1, wherein at least one version of a versioned dataset in the set of versioned datasets contains data generated as a result of one or more Spark systems executing a derivation program taking at least one version of another versioned dataset as input to the derivation program.

5. The method of claim 1, wherein at least one version of a versioned dataset in the set of versioned datasets contains data generated as a result of one or more MapReduce systems executing a derivation program taking at least one version of another versioned dataset as input as input to the derivation program.

6. One or more non-transitory computer-readable media storing one or more programs, the one or more programs comprising instructions for: storing an input dataset and provenance metadata identifying one or more previous versions of the input dataset;using a derivation program, transforming data in the input dataset and storing the transformed data as a versioned dataset;updating the provenance metadata to identify the input dataset in addition to the one or more previous versions of the input dataset;receiving selection of a versioned dataset that is within a data pipeline system;determining full data provenance of the selected versioned dataset, the full data provenance comprising a set of versioned datasets, by identifying, in the provenance metadata, at least the input dataset and the one or more previous versions of the input dataset;providing for display of a visualization of the full data provenance of the selected versioned dataset, the visualization comprising a graph, the graph comprising a compound node for the selected versioned dataset and for each versioned dataset in the set of versioned datasets, the graph further comprising edges connecting the compounds nodes, each edge representing a derivation dependency between versions of the versioned datasets represented by the compound nodes connected by the edge;wherein a sub-entry of the compound node for a particular versioned dataset in the set of versioned datasets is visually distinguished in the graphical user interface from other compound node sub-entries of the graph to indicate that a version, of the particular version dataset represented by the sub-entry has been flagged in a database as containing invalid data;wherein an edge in the graph representing a derivation dependency of a first version of a first versioned dataset in the set of versioned datasets on a second version of a second versioned dataset in the set of versioned datasets is visually distinguished from other edges in the graph to indicate that the first version of the first versioned dataset potentially contains invalid data as a result of the derivation dependency.

7. The one or more non-transitory computer-readable media of claim 6, wherein the compound node of the selected versioned dataset indicates a name or identifier of the selected version dataset; and wherein the compound node for each versioned dataset in the set of versioned datasets indicates a name or identifier of the each versioned dataset.

8. The one or more non-transitory computer-readable media of claim 6, wherein the compound node for each versioned dataset in the set of versioned datasets comprises at least one sub-entry representing a version of the each versioned dataset in the full data provenance of the selected versioned dataset.

9. The one or more non-transitory computer-readable media of claim 6, wherein at least one version of a versioned dataset in the set of versioned datasets contains data generated as a result of one or more Spark systems executing a derivation program taking at least one version of another versioned dataset as input to the derivation program.

10. The one or more non-transitory computer-readable media of claim 6, wherein at least one version of a versioned dataset in the set of versioned datasets contains data generated as a result of one or more MapReduce systems executing a derivation program taking at least one version of another versioned dataset as input as input to the derivation program.

11. A system comprising: memory;one or more processors;one or more programs stored in the memory and configured for execution by the one or more processors, the one or more programs comprising instructions for:storing an input dataset and provenance metadata identifying one or more previous versions of the input dataset;using a derivation program, transforming data in the input dataset and storing the transformed data as a versioned dataset;updating the provenance metadata to identify the input dataset in addition to the one or more previous versions of the input dataset;receiving selection of a versioned dataset that is within a data pipeline system;determining full data provenance of the selected versioned dataset, the full data provenance comprising a set of versioned datasets, by identifying, in the provenance metadata, at least the input dataset and the one or more previous versions of the input dataset;providing for display of a visualization of the full data provenance of the selected versioned dataset, the visualization comprising a graph, the graph comprising a compound node for the selected versioned dataset and for each versioned dataset in the set of versioned datasets, the graph further comprising edges connecting the compounds nodes, each edge representing a derivation dependency between versions of the versioned datasets represented by the compound nodes connected by the edge;wherein a sub-entry of the compound node for a particular versioned dataset in the set of versioned datasets is visually distinguished in the graphical user interface from other compound node sub-entries of the graph to indicate that a version, of the particular version dataset represented by the sub-entry has been flagged in a database as containing invalid data;wherein an edge in the graph representing a derivation dependency of a first version of a first versioned dataset in the set of versioned datasets on a second version of a second versioned dataset in the set of versioned datasets is visually distinguished from other edges in the graph to indicate that the first version of the first versioned dataset potentially contains invalid data as a result of the derivation dependency.

12. The system of claim 11, wherein the compound node of the selected versioned dataset indicates a name or identifier of the selected version dataset; and wherein the compound node for each versioned dataset in the set of versioned datasets indicates a name or identifier of the each versioned dataset.

13. The system of claim 11, wherein the compound node for each versioned dataset in the set of versioned datasets comprises at least one sub-entry representing a version of the each versioned dataset in the full data provenance of the selected versioned dataset.