Patent Application
20240045886
This application claims priority under 35 U.S.C. ยง 119 to patent application no. DE 10 2022 208 099.8, filed on Aug. 4, 2022 in Germany, the disclosure of which is incorporated herein by reference in its entirety.
The disclosure relates to a method for efficiently validating a data set in a distributed database system or file system (e.g., HDFS), a computer program, and a device configured to perform the method and a machine-readable storage medium.
Data is available in a variety of different formats, and processing it consumes large amounts of resources. It is therefore desirable to have the data in the correct format prior to initiating processing in order to work as resource-efficiently as possible (so-called fail fast, fail cheap). In order to process the data in a resource-efficient manner, erroneous data sets are validated before processing. In a validation step, for example, the data set is checked for the expected scheme (data type of the columns, nullability), the expected properties (numeric range, length of character strings, etc.) and the expected key structure (which combination of columns is to give a unique identifier).
Validation is also very useful before and after transformations and prior to persistence of results to ensure that no undesirable changes have been made.
There are several existing Python frameworks that cover this need, but they all suffer from one or more shortcomings. Pyspark data sets are saved in a column-based format (parquet), wherein the above-mentioned frameworks perform a row by row validation, which affects the performance for validation. The validation process should be a fast test and therefore kept as short as possible.
The disclosure has the advantage that a large data set in a distributed database system can be validated in a particularly resource-efficient manner, in particular under a low utilization of a communication channel between the individual databases of the distributed database system. This also results in validation being able to be carried out up to 20 times faster.
In a first aspect, the disclosure relates to a particularly computer-implemented method for validating a data set in a distributed database system. With validating it can be understood to mean data validation, in particular a check for plausibility of data of the data set. That is to say, a validation can in particular be understood to mean that specific data, or the value of the data, are checked to determine whether they belong to a specific data type, are within a given range of values and/or a given amount of values, etc.
The distributed database system comprises a plurality of computers and a central computer, wherein the data set is stored distributedly on the computers. That is to say, the data record is distributed across several computers, in particular parts of the data set are stored on the computers. It is conceivable that a plurality of partial data sets is saved on one of the computers. The central computer can store how the data set is distributed over the plurality of computers, in particular, which partial data set is stored on which computer. It is conceivable that the central computer has access to further information that characterizes the data set, for example, which data types are stored in the data set, in particular which data type of the partial data record is available on the respective computer. Preferably, the further information comprises the range of values, etc., in which the entries of the respective partial data set may move.
The method begins with the central computer sending a request to one of the computers. The request has a command to check a partial data set of the data set on the computer to which the request was sent with respect to a given (validation) criterion. The criterion may be determined from the further information characterizing the data set.
Thereafter, execution of the command to be tested by the computer to which the request was sent may follow. Subsequently, a response by the computer may be sent to the central computer, comprising a result of the step of executing the command, in particular whether the partial data set satisfies the given criterion.
Due to the targeted request, the method has the advantage over known methods that only one request is necessary for the partial data set, whereas known methods check the entries of the data set individually. Usually the central computer checks the data individually, which is why the data set must then be effectively transferred. Thus, less data is exchanged within the distributed database system, resulting in an improved system throughput and thus an improved response time for validation.
It is proposed that the data set is in the form of a table having a plurality of columns, each associated with a data type, wherein the rows of the data set each have an entry, wherein the data set is stored in terms of columns distributed across the plurality of computers. Preferably, the command for checking is a command in which the entries are individually checked to see if they correspond to the data type associated with the corresponding column.
It is further proposed that the central computer send a request to a plurality, in particular to each, of the computers, wherein the computers execute the request and return a result of the request to the central computer via a message, and the central computer determines a validation of the data set, depending on the received messages.
It is further proposed that the central computer sends the requests simultaneously. This makes better use of the distributed structure of the database and generates even further performance gains for faster validation.
In further aspects, the disclosure relates to an apparatus and to a computer program, which are each configured to perform the aforementioned methods, and to a machine-readable storage medium on which said computer program is stored.
Embodiments of the disclosure are explained in greater detail below with reference to the accompanying drawings. In the drawings:
A large data set may be stored on the distributed database system (10). Here, the data set may be dissected and the individual dissected parts are stored on individual computers (11-14). If the computers are in geographically different locations, the data is stored at multiple physical locations.
In one embodiment, the data set may have x values, y values, z values, and optionally a timestamp. The data set can be given by a table in which the columns are assigned to one of the values and the rows each have the corresponding values for the columns. In a preferred embodiment of the data set, so-called distinct_keys may be defined, which combine a combination of columns into a unique identifier. In a further embodiment, the data set may also have nested entries, which may not contain an entry but may contain further data sets.
It is conceivable that the x values are stored on computer 11, the y values are stored on computer 12, the z values are stored on computer 13, and the timestamps are stored on computer 14.
To now validate the values of the data set, known methods would be performed according to the following pseudo-algorithm, which is performed on the central computer 15:
That is, a row by row validation of the data set is performed. For a data set with, for example, 10,000,000 data points per column, such a loop including the transfer of the values results in a high resource consumption of the database system.
In contrast, in a preferred embodiment, it is proposed that validation should now be done column by column. An exemplary pseudo-algorithm would be:
Based on this pseudo-algorithm, it can be seen that the loop over the individual values is omitted and furthermore, the validation takes place locally on the respective computer (11-14). In the above exemplary embodiment, for example, only one request is made to computer 11 to validate the x values and no individual transmission of the values to the central computer for validation.
The method starts with step S21. In this step, a loop is started via the computers (11-14). That is to say, for the first loop pass, a first computer is selected from the plurality of computers (11-14).
In the subsequent step S22, a request is sent to the selected computer from step S21 via the central computer (15).
On the selected computer, the request is then executed in the following step S23. In particular, the column of the data set on the computer is validated with respect to a validation criterion from the request.
The selected computer can then send a message to the central computer (15) and confirm whether the validation was successful, i.e., the entries of the column have satisfied the criterion or not.
If the criterion is satisfied, then step S21 is performed again and another computer is selected and steps S22 and S23 are performed for that newly selected computer.
After all distributed columns have been checked on the computer by means of the loop via steps S21 to S23, the data set can be released in a subsequent, optional step S24, for example if the central computer has received the message from all computers that the columns meet the respective criteria.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10 2022 208 099.8 | Aug 2022 | DE | national |
