Patent Application
20220391582
Software applications may process a variety of online forms, such as compliance forms designed to comply with government regulations and into which users enter data. When compliance and other forms are modified, the online computer based forms are updated to reflect the changes. Often, the updates are performed by a human identifying the changes and coding the changes in the revised form. A capability to automatically process a form embodied in a document would be advantageous.
This summary is provided to introduce a selection of concepts that are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in limiting the scope of the claimed subject matter.
In general, in one aspect, one or more embodiments relate to a method including clustering form elements into line objects and columns of a table of a structured representation by applying a trained multi-dimensional clustering model to spatial coordinates of the form elements, and assigning a table header line type to a table header line object of the line objects based on a spatial coordinate of the table header line object relative to a spatial coordinate of a topmost table data line object of the line objects, and a determination that a number of columns of the table header line object is within a threshold of a number of columns of the topmost table data line object. The topmost table data line object is assigned a table data line type. The method further includes presenting the structured representation to a user.
In general, in one aspect, one or more embodiments relate to a system including a computer processor and a data repository configured to store a structured representation including line objects and columns of a table. The system further includes a structured representation generator executing on the computer processor and configured to cluster form elements into the line objects and the columns of the table by applying a trained multi-dimensional clustering model to spatial coordinates of the form elements, and assign a table header line type to a table header line object of the line objects based on a spatial coordinate of the table header line object relative to a spatial coordinate of a topmost table data line object of the line objects, and a determination that a number of columns of the table header line object is within a threshold of a number of columns of the topmost table data line object. The topmost table data line object is assigned a table data line type. The structured representation generator is further configured to present the structured representation to a user.
In general, in one aspect, one or more embodiments relate to a method including clustering initial form elements into initial line objects and initial columns of an initial table of an initial structured representation by applying a trained multi-dimensional clustering model to initial spatial coordinates of the initial form elements. The initial structured representation has a structured representation type. The method further includes assigning a table header line type to an initial table header line object of the initial line objects based on a spatial coordinate of the initial table header line object relative to a spatial coordinate of an initial topmost table data line object of the initial line objects, and a determination that a number of columns of the initial table header line object is within a threshold of a number of columns of the initial topmost table data line object. The initial topmost table data line object is assigned a table data line type. The method further includes adding the initial structured representation to a data repository, and clustering next form elements into next line objects and next columns of a next table of a next structured representation by applying a trained multi-dimensional clustering model to next spatial coordinates of the next form elements. The next structured representation has the structured representation type. The method further includes assigning a table header line type to a next table header line object of the next line objects based on a spatial coordinate of the next table header line object relative to a spatial coordinate of a next topmost table data line object of the next line objects, and a determination that a number of columns of the next table header line object is within a threshold of a number of columns of the next topmost table data line object. The next topmost table data line object is assigned a table data line type. The method further includes determining that the initial structured representation and the next structured representation are different, and in response to determining that the initial structured representation and the next structured representation are different, replacing, in the data repository, the initial structured representation with the next structured representation.
Other aspects of the invention will be apparent from the following description and the appended claims.
Specific embodiments of the invention will now be described in detail with reference to the accompanying figures. Like elements in the various figures are denoted by like reference numerals for consistency.
In the following detailed description of embodiments of the invention, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.
Throughout the application, ordinal numbers (e.g., first, second, third, etc.)
may be used as an adjective for an element (i.e., any noun in the application). The use of ordinal numbers is not to imply or create any particular ordering of the elements nor to limit any element to being only a single element unless expressly disclosed, such as by the use of the terms “before”, “after”, “single”, and other such terminology. Rather, the use of ordinal numbers is to distinguish between the elements. By way of an example, a first element is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.
Forms represented as machine-readable documents may have a variety of elements with visual relationships among the form elements. Providing a capability to systematically and reliably capture the visual relationships among the form elements may be challenging due to variations in the spatial layout of the machine-readable documents. Such relationships may include: 1) the spatial layout of form elements (e.g., fields) into rows and columns of tables, 2) the classification of particular rows into header rows vs. data rows, and 3) the alignment of form elements in different portions of a document into columns.
The disclosed invention combines machine learning algorithms and rules to provide the capability to systematically and reliably capture visual relationships among elements of forms represented as machine-readable documents. A trained multi-dimensional clustering model is applied to spatial coordinates of document elements to capture the layout of tabular data as rows and columns of a structured representation. A combination of rules may be used to classify table rows into header rows vs. data rows. The rules may be based on the spatial coordinates of candidate header rows relative to table data rows, as well as the number of columns in candidate header rows relative to table data rows. A trained classifier may be used to determine the alignment of form elements in different portions of a document. The trained multi-dimensional clustering model may be re-applied to adjust the structured representation in response to corrections received from a user. For example, the corrections may be received via a graphical user interface (GUI) that permits the user to edit the structured representation.
The aforementioned capability enables efficient, scalable processing and/or updating of forms represented as machine-readable documents. This capability is especially useful for forms that are periodically updated with new versions. The resulting structured representation may be used to drive a forms-based GUI usable by both end-users entering data into forms and/or by expert users editing the structured representation to provide added value.
In one or more embodiments, the user computing system (102) provides, to a user, a variety of computing functionality. For example, the computing functionality may include word processing, multimedia processing, financial management, business management, social network connectivity, network management, and/or various other functions that a computing device performs for a user. The user may be a small business owner. Alternatively, the user may be a company employee that acts as a sender, a potential sender, or a requestor of services performed by a company (e.g., a client, a customer, etc.) of the user computing system. The user computing system (102) may be a mobile device (e.g., phone, tablet, digital assistant, laptop, etc.) or any other computing device (e.g., desktop, terminal, workstation, etc.) with a computer processor (not shown) and memory (not shown) capable of running computer software. The user computing system (102) may take the form of the computing system (500) shown in
The user computing system (102) includes a structured representation editor (148) in accordance with one or more embodiments. The structured representation editor (148) is a user interface (UI) (not shown) for receiving input from a user and transmitting output to the user. For example, the UI may be a graphical user interface or other user interface. The UI may be rendered and displayed within a local desktop software application or the UI may be generated by a remote web server and transmitted to a user's web browser executing locally on a desktop or mobile device.
Continuing with
In one or more embodiments, the data repository (106) includes functionality to store a document (110) and a structured representation (120). A document (110) is any type of written matter that captures information. The document (110) may be represented as a file using the Portable Document Format (PDF), HyperText Markup Language (HTML), eXtensible Markup Language (XML), JavaScript Object Notation (JSON), or any other file format. For example, a document (110) may be or include one or more of a form, a spreadsheet, a presentation, a word processing application document, or other such document. By way of an example, the document (110) may be a compliance form (e.g., audit form, data security form, tax form, medical form, privacy policy, etc.) to be completed by a user, and designed to comply with the regulations of a government agency. For example, the compliance form may be specific to a jurisdiction (e.g., a geographic region such as a state, country, region, municipality, reinvestment zone, etc.).
The document (110) includes document elements (112A, 112N). A document element (112A) is a discrete visual component of the document (110) that is displayed when the document is displayed in a user interface. For example, a document element (112A) may be a chunk of text. Alternatively, a document element (112A) may be a shape (e.g., a line or rectangle), an image (e.g., a bitmap), etc. Continuing this example, a document element (112A) may be a rectangle or box that represents an input field. In one or more embodiments, a document element (112A) includes spatial coordinates (114) indicating the placement of the document element (112A) within the document (110). The placement may be expressed in terms of a region (e.g., a rectangle) in a coordinate system (e.g., Cartesian coordinates within the document (110)), where the region encompasses the document element (112A). In one or more embodiments, the placement may be used to calculate distances between document elements (112A, 112N). The document element (112A) may include additional attributes such as font, font size, number of characters (e.g., broken down into the number of numeric characters and the number of alphabetic characters), number of words, etc.
The document (110) may include a document type. The document type is a category that describes the document (110). For example, the document type may be a general category, such as tax document, payroll document, or legal document. Alternatively, the document type (114) may be a specific category, such as Schedule 1 of a Federal Tax Form, etc.
In one or more embodiments, the structured representation (120) is a schema for a form. The structured representation (120) is the output of embodiments of the invention. For example, the schema may be represented using JavaScript Object Notation (JSON) or eXtended Markup Language (XML). The structured representation (120) may include one or more tables (121T, 121Y). A table (121T) is a representation of information in terms of rows and columns. For example, a row (e.g., a record) may include one or more values corresponding to one or more columns. A table (121T) may include line objects (122L, 122Q) and columns (130). A line object (122L) is an instance of a row in a table (121T). A line object (122L) includes form elements (124F, 124J) and a line type (128). A form element (124F) is a component of the structured representation (120). The form element (124F) is a representation of a particular document element (112A) in a structural format. Thus, each form element corresponds to a particular document element. Examples of form elements (124F, 124J) may include line descriptions, line numbers, fields, field numbers, field descriptions, etc.
In one or more embodiments, a line object (122L) corresponds to a placement (e.g., a location) within the document (110). For example, a line object (122L) may correspond to a placement within the document (110) that is within a threshold distance of the placements (e.g., regions) of the document elements corresponding to the form elements included in the line object (122L). Continuing this example, the placement of a line object (122L) may be represented as a spatial coordinate (e.g., a y-coordinate) of the vertical axis of the document (110). Further continuing this example, the spatial coordinate may be represented as a horizontal line (e.g., the line defined by the equation “y=12”) within a coordinate system of the document (110).
In one or more embodiments, columns (130) are vertical lines to which form elements in different line objects of the structured representation (120) are aligned. For example, the different line objects in the same column may be in the same table or in different tables. Each of the columns (130) may be a vertical line defined using a spatial coordinate (114) (e.g., an x-coordinate) of the horizontal axis of the document (110) corresponding to the structured representation (120). For example, the vertical line may be defined by the equation “x=3” within a coordinate system of the document. In one or more embodiments, one or more form elements of a line object (122L) (e.g., form elements (124F, 124J)) are assigned column IDs (126F, 126J) each identifying one of the columns (130). For example, a line object may include a first form element that is assigned column ID “column1” and a second form element that is assigned column ID “column4”. The column IDs (126F, 126J) may correspond to the spatial coordinates (114) of the columns (130). For example, if N is the number of columns, then the column ID corresponding to the column with the largest x-coordinate may be “column0”, the column ID corresponding to the column with the next largest x-coordinate may be “column1”, etc., and the column ID corresponding to the column with the smallest x-coordinate may be “columnN”. For example, see the columns (352A, 352B, 352C) in
The line type (128) is a category that describes the corresponding line object (122L). For example, the line type (128) may be “table header” when the line object corresponds to a line in a header of a table (121T). Continuing this example, a line object with line type (128) “table header” may describe the type of data contained in the table (121T). Alternatively, the line type (128) may be “table data” when the line object (122L) corresponds to a line of data (e.g., a record) in the table (121T). As another example, the line type (128) may be “form header” when the line object corresponds to a line in a header of a form represented in the document (110) corresponding to the structured representation (120). The structured representation generator (140) includes functionality to assign a line type (128) to a line object (122L).
The structured representation (120) may include a structured representation type. The structured representation type is a category that describes the structured representation (120). The structured representation type may correspond to the document type of the document (110) corresponding to the structured representation (120).
Continuing with
The multi-dimensional clustering model may perform the clustering into line objects (122L, 122Q) using a distance measure based on spatial coordinates (114) of the document elements corresponding to the form elements. For example, the distance measure may be a Cartesian distance between the y-coordinates of the document elements corresponding to the form elements. Continuing this example, form elements (124F, 124J), whose corresponding document elements are within a threshold distance of each other, may be clustered into a single line object (122L). Further continuing this example, the single line object (122L) may correspond to specific spatial coordinates (e.g., a y-coordinate). Still further continuing this example, the document elements may also be within a threshold distance of a placement within the document (110) that corresponds to the line object (122A). The multi-dimensional clustering model may be trained to cluster form elements (124F, 124J) into a line object (122A) with a high degree of accuracy despite variations in the spatial coordinates and/or sizes of documents elements. In other words, the rows of document elements may not be horizontally aligned, as shown in the example of un-aligned rows of document elements (370) in
Similarly, the multi-dimensional clustering model includes functionality to cluster form elements into columns (130). The multi-dimensional clustering model may perform the clustering into columns (130) using a distance measure based on spatial coordinates (114) of document elements corresponding to the form elements. For example, the distance measure may be based on a Cartesian distance between the x-coordinates of the document elements corresponding to the form elements. Continuing this example, form elements whose corresponding document elements are within a threshold distance of each other may be clustered in the same column (130). Further continuing this example, the document elements may also be within a threshold distance of a placement within the document (110) that corresponds to the column (130). Still further continuing this example, the placement may be defined as a vertical line (e.g., a x-coordinate) within the document (110).
In one or more embodiments, the form element/column classifier (144) includes functionality to assign column IDs (126F, 126J) to form elements (124F, 124J). In one or more embodiments, the form element/column classifier (144) is implemented using a decision tree classifier that performs multi-class classification, where the multiple classes are different column IDs. In one or more embodiments, the internal nodes of the decision tree are spatial coordinates (e.g., x-coordinates) of a training data set of document elements, and the leaf nodes are column IDs. As a default, the decision tree classifier may return, for a given document element, the column ID with the highest probability. The decision tree classifier may be primarily used to classify document elements into column IDs that are typically not part of a table. For example, the leftmost columns of a document are typically not part of a table. Alternatively, the form element/column classifier (144) may be implemented by any other type of classifier, such as k-nearest neighbors. The decision tree classifier may be trained using documents whose document elements are labeled with spatial coordinates and column IDs. The column IDs may identify columns that correspond to vertical lines in the document. The form element/column classifier (144) may perform the assignment of column IDs (126F, 126J) to form elements (124F, 124J) using a distance measure based on the spatial coordinates of the document elements corresponding to the form elements, as well as the spatial coordinates of the columns.
The structured representation generator (140) may include one or more element classification models (e.g., supervised machine learning models) with functionality to assign an element type to a document element (112A). The form element corresponding to the document element (112A) may inherit the assigned element type. Examples of element types may include: descriptive text, header, input field, line number, etc. For example, a first element classification model may classify a first document element as “descriptive text”, a second element classification model may classify a second document element as a “line number”, and a third element classification model may classify a third document element as an “input field”. The element classification models may be trained using document elements labeled as various types of elements. An element classification model may be implemented as a classifier using XGBoost (available at https://github.com/dmlc/xgboost). Alternatively, an element classification models may be implemented as a k-nearest neighbor (k-NN) classifier. Still alternatively, an element classification model may be implemented as a deep learning classifier, such as a neural network classifier (e.g., based on convolutional neural networks (CNNs)), random forest classifier, SGD classifier, lasso classifier, gradient boosting classifier, bagging classifier, ada boost classifier, ridge classifier, elastic net classifier, or NuSVR classifier. Deep learning, also known as deep structured learning or hierarchical learning, is part of a broader family of machine learning methods based on learning data representations, as opposed to task-specific algorithms.
Continuing with
In one or more embodiments, the computer processor(s) (146) takes the form of the computer processor(s) (502) described with respect to
While
Initially, in Step 202, form elements are clustered into line objects and columns of a table of a structured representation by applying a trained multi-dimensional clustering model to spatial coordinates of the form elements. The structured representation generator may obtain the form elements as follows. First, the structured representation generator may obtain a document (e.g., from a data repository). The structured representation generator may obtain, from the document, document elements and spatial coordinates indicating the placement of each document element within the document. For example, the structured representation generator may obtain the document elements and the spatial coordinates from the document using a software tool. Continuing this example, the structured representation generator may use a PDF mining tool to extract, from a document represented in PDF, the document elements and the spatial coordinates. Alternatively, the structured representation generator may obtain the document elements and the spatial coordinates from a marked up version of the document. For example, the marked up version of the document may be represented in a machine-readable format, such as JavaScript Object Notation (JSON). The structured representation generator may convert the document elements to form elements using the spatial coordinates of the document elements. For example, the structured representation generator may assign the spatial coordinates of each form element to be the spatial coordinates of the corresponding document element.
The multi-dimensional clustering model may cluster form elements into line objects of the table using a distance measure based on the y-coordinates of the form elements. The multi-dimensional clustering model may cluster form elements into columns using distances based on the x-coordinates of the form elements. The multi-dimensional clustering may correctly identify the line objects and columns even in tables with holes in the line objects and/or columns. For example, a line object with a hole lacks a value for one or more columns. Similarly, a column with a hole lacks a value for one or more line objects.
In addition to applying the trained multi-dimensional clustering model, the structured representation generator may use one or more rules (e.g., heuristics) to cluster line objects into tables. For example, a rule may specify that a line object is part of a table when the number of user-enterable form elements of the line object exceeds a threshold.
In Step 204, a table header line type is assigned to a table header line object. In one or more embodiments, the structured representation generator first identifies line objects of the table that are candidate table header line objects (i.e., line objects that are candidates to be assigned the line type “table header”). The structured representation generator may identify a set of candidate table header line objects using one or more rules. For example, one rule may be: a candidate table header line object contains text only (e.g., a candidate table header line object does not accept user-entered input). In contrast, the structured representation generator may identify line objects of the table that are candidate table data line objects (i.e., candidate line objects to be assigned the line type “table data”) using the following rule: a line object that accepts user-entered input may be a table data line object. As another example, the structured representation generator may identify a candidate form header line object using the following rule: a line object containing text whose size exceeds the size of text in any other line object in the structured representation may be assigned the line type “form header”.
Once the candidate table header line objects are identified, the structured representation generator may assign the line type “table header” to a subset of the candidate table header line objects based on the following:
1) a spatial coordinate of the candidate table header line object relative to a spatial coordinate of a topmost table data line object, where the topmost table data line object is assigned the line type “table data”. For example, the spatial coordinate of the candidate table header line object may be within a threshold distance of the spatial coordinate of the topmost table data line object. Continuing this example, the spatial coordinate of the candidate table header line object may be a y-coordinate corresponding to the table header line object and the spatial coordinate of the topmost table data line object may be a y-coordinate corresponding to the topmost table data line object. The topmost table data line object may be a line object whose y-coordinate is the smallest in the cluster of line objects of the table.
2) a determination that a number of columns of the candidate table header line object is within a threshold of a number of columns of the topmost table data line object. For example, the candidate table header line object may have the same number of columns as the topmost table data line object. As another example, the difference between the number of columns in the candidate table header line object and the number of columns in the topmost table data line object may be within a threshold of two. As an example, in
In Step 206, a same column ID is assigned to a first form element in a first line object and a second form element in a second line object based on applying a trained classifier to the spatial coordinates of the form elements. The same column ID identifies a column in the table. The trained classifier may align form elements in different line objects to the same column despite variations in the placement (e.g., variations in the x-coordinates) of document elements corresponding to the form elements. For example, the x-coordinates of document elements on different lines of a document may not be aligned, even though the document elements may be intended to be aligned in the same column. In addition, the x-coordinates of document elements of different documents may vary. In contrast, rule-based approaches to assigning column IDs to form elements in the presence of such variations may not be as reliable as using a trained classifier.
In one or more embodiments, the different line objects are in different tables. For example, the document elements of the document corresponding to the form elements of the structured representation may be arranged such that the columns to which the document elements are aligned are “global” columns spanning multiple tables.
In Step 208, the structured representation is presented to a user. The structured representation may be presented to the user via a structured representation editor. The structured representation editor may receive the structured representation from the structured representation generator. The structured representation editor may receive, from the user, one or more corrections to the structured representation. That is, the structured representation generated by Step 202, Step 204, and Step 206 above may contain errors, which may be corrected by a user. A correction may modify a line type of a line object of the structured representation. For example, a correction may change a line type of a line object from “table data” to “table header”. Alternatively, a correction may modify an assignment of a column ID to a form element of a line object. Still alternatively, a correction may modify an element type assigned to a form element.
The table below shows an example of a structured representation as presented to a user via a structured representation editor with a graphical user interface (GUI). The structured representation includes the line type assigned to each line object. For example, the line type of the line objects on line numbers 6, 7, 8, and 13 is “table header”, and the line type of the line objects on line numbers 14, 15, 22, and 23 is “table data”. The user may correct the line type assigned to any of the line objects by selecting a line type from a drop-down menu that includes a list
of valid line types.
In response to receiving the correction, the structured representation generator may repeat Step 202 above to re-cluster the form elements into line objects and columns by repeating the application of the trained multi-dimensional clustering model to the spatial coordinates of the form elements. In response to receiving the correction, the structured representation generator may repeat Step 204 above to modify the line type of a line object. For example, the correction may change the topmost table data line object, which may change the set of candidate table header line objects. Continuing this example, repeating Step 204 may re-assign the form elements in the new table header line objects to different columns. In addition, the correction may be used as additional training data to retrain the trained multi-dimensional clustering model.
Initially, in Step 252, initial form elements are clustered into initial line objects and initial columns of a table of an initial structured representation by applying a trained multi-dimensional clustering model to initial spatial coordinates of the initial form elements (see description of Step 202 above). The initial structured representation may have a structured representation type.
In Step 254, a table header line type is assigned to an initial table header line object. In one or more embodiments, the structured representation generator first identifies line objects of the table that are candidate initial table header line objects (see description of Step 204 above). Once the candidate initial table header line objects are identified, the structured representation generator may assign the line type “table header” to a subset of the candidate initial table header line objects based on 1) and 2) as described in Step 204 above.
In Step 256, the initial structured representation is added to a data repository. In one or more embodiments, the data repository serves as a knowledge base of structured representations. In one or more embodiments, the structured representations included in the repository may be accessed (e.g., queried) using the structured representation type.
In Step 258, next form elements are clustered into next line objects and next columns of a table of a next structured representation by applying the trained multi-dimensional clustering model to next spatial coordinates of the next form elements (see description of Step 202 above). The next structured representation may have the structured representation type. For example, the next structured representation may be a later version of the initial structured representation.
In Step 260, a table header line type is assigned to a next table header line object. In one or more embodiments, the structured representation generator first identifies line objects of the table that are candidate next table header line objects (see description of Step 204 above). Once the candidate next table header line objects are identified, the structured representation generator may assign the line type “table header” to a subset of the candidate next table header line objects based on 1) and 2) as described in Step 204 above.
If, in Step 262, a determination is made that the initial structured representation and the next structured representation are different, then Step 264 below is executed. In one or more embodiments, the initial structured representation and the next structured representation are different when at least one of the following is true:
In Step 264, the initial structured representation is replaced with the next structured representation in the data repository. For example, the next structured representation may representation an updated version of the initial structured representation. Alternatively, the next structured representation may be a version of the initial structured representation that is adapted to a jurisdiction different from a jurisdiction corresponding to the initial structured representation.
The following examples are for explanatory purposes only and not intended to limit the scope of the invention.
Then, the structured representation editor presents the structured representation to a user via a graphical user interface (GUI). The structured representation editor receives corrections (410) ((150) in
Embodiments of the invention may be implemented on a computing system. Any combination of mobile, desktop, server, router, switch, embedded device, or other types of hardware may be used. For example, as shown in
The computer processor(s) (502) may be an integrated circuit for processing instructions. For example, the computer processor(s) may be one or more cores or micro-cores of a processor. The computing system (500) may also include one or more input devices (510), such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device.
The communication interface (512) may include an integrated circuit for connecting the computing system (500) to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) and/or to another device, such as another computing device.
Further, the computing system (500) may include one or more output devices (508), such as a screen (e.g., a liquid crystal display (LCD), a plasma display, touchscreen, cathode ray tube (CRT) monitor, projector, or other display device), a printer, external storage, or any other output device. One or more of the output devices may be the same or different from the input device(s). The input and output device(s) may be locally or remotely connected to the computer processor(s) (502), non-persistent storage (504), and persistent storage (506). Many different types of computing systems exist, and the aforementioned input and output device(s) may take other forms.
Software instructions in the form of computer readable program code to perform embodiments of the invention may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer readable medium such as a CD, DVD, storage device, a diskette, a tape, flash memory, physical memory, or any other computer readable storage medium. Specifically, the software instructions may correspond to computer readable program code that, when executed by a processor(s), is configured to perform one or more embodiments of the invention.
The computing system (500) in
Although not shown in
The nodes (e.g., node X (522), node Y (524)) in the network (520) may be configured to provide services for a client device (526). For example, the nodes may be part of a cloud computing system. The nodes may include functionality to receive requests from the client device (526) and transmit responses to the client device (526). The client device (526) may be a computing system, such as the computing system shown in
The computing system or group of computing systems described in
Based on the client-server networking model, sockets may serve as interfaces or communication channel end-points enabling bidirectional data transfer between processes on the same device. Foremost, following the client-server networking model, a server process (e.g., a process that provides data) may create a first socket object. Next, the server process binds the first socket object, thereby associating the first socket object with a unique name and/or address. After creating and binding the first socket object, the server process then waits and listens for incoming connection requests from one or more client processes (e.g., processes that seek data). At this point, when a client process wishes to obtain data from a server process, the client process starts by creating a second socket object. The client process then proceeds to generate a connection request that includes at least the second socket object and the unique name and/or address associated with the first socket object. The client process then transmits the connection request to the server process. Depending on availability, the server process may accept the connection request, establishing a communication channel with the client process, or the server process, busy in handling other operations, may queue the connection request in a buffer until server process is ready. An established connection informs the client process that communications may commence. In response, the client process may generate a data request specifying the data that the client process wishes to obtain. The data request is subsequently transmitted to the server process. Upon receiving the data request, the server process analyzes the request and gathers the requested data. Finally, the server process then generates a reply including at least the requested data and transmits the reply to the client process. The data may be transferred, more commonly, as datagrams or a stream of characters (e.g., bytes).
Shared memory refers to the allocation of virtual memory space in order to substantiate a mechanism for which data may be communicated and/or accessed by multiple processes. In implementing shared memory, an initializing process first creates a shareable segment in persistent or non-persistent storage. Post creation, the initializing process then mounts the shareable segment, subsequently mapping the shareable segment into the address space associated with the initializing process. Following the mounting, the initializing process proceeds to identify and grant access permission to one or more authorized processes that may also write and read data to and from the shareable segment. Changes made to the data in the shareable segment by one process may immediately affect other processes, which are also linked to the shareable segment. Further, when one of the authorized processes accesses the shareable segment, the shareable segment maps to the address space of that authorized process. Often, only one authorized process may mount the shareable segment, other than the initializing process, at any given time.
Other techniques may be used to share data, such as the various data described in the present application, between processes without departing from the scope of the invention. The processes may be part of the same or different application and may execute on the same or different computing system.
Rather than or in addition to sharing data between processes, the computing system performing one or more embodiments of the invention may include functionality to receive data from a user. For example, in one or more embodiments, a user may submit data via a graphical user interface (GUI) on the user device. Data may be submitted via the graphical user interface by a user selecting one or more graphical user interface widgets or inserting text and other data into graphical user interface widgets using a touchpad, a keyboard, a mouse, or any other input device. In response to selecting a particular item, information regarding the particular item may be obtained from persistent or non-persistent storage by the computer processor. Upon selection of the item by the user, the contents of the obtained data regarding the particular item may be displayed on the user device in response to the user's selection.
By way of another example, a request to obtain data regarding the particular item may be sent to a server operatively connected to the user device through a network. For example, the user may select a uniform resource locator (URL) link within a web client of the user device, thereby initiating a Hypertext Transfer Protocol (HTTP) or other protocol request being sent to the network host associated with the URL. In response to the request, the server may extract the data regarding the particular selected item and send the data to the device that initiated the request. Once the user device has received the data regarding the particular item, the contents of the received data regarding the particular item may be displayed on the user device in response to the user's selection. Further to the above example, the data received from the server after selecting the URL link may provide a web page in Hyper Text Markup Language (HTML) that may be rendered by the web client and displayed on the user device.
Once data is obtained, such as by using techniques described above or from storage, the computing system, in performing one or more embodiments of the invention, may extract one or more data items from the obtained data. For example, the extraction may be performed as follows by the computing system in
Next, extraction criteria are used to extract one or more data items from the token stream or structure, where the extraction criteria are processed according to the organizing pattern to extract one or more tokens (or nodes from a layered structure). For position-based data, the token(s) at the position(s) identified by the extraction criteria are extracted. For attribute/value-based data, the token(s) and/or node(s) associated with the attribute(s) satisfying the extraction criteria are extracted. For hierarchical/layered data, the token(s) associated with the node(s) matching the extraction criteria are extracted. The extraction criteria may be as simple as an identifier string or may be a query presented to a structured data repository (where the data repository may be organized according to a database schema or data format, such as XML).
The computing system in
The user, or software application, may submit a statement or query into the
DBMS. Then the DBMS interprets the statement. The statement may be a select statement to request information, update statement, create statement, delete statement, etc. Moreover, the statement may include parameters that specify data, or data container (database, table, record, column, view, etc.), identifier(s), conditions (comparison operators), functions (e.g. join, full join, count, average, etc.), sort (e.g. ascending, descending), or others. The DBMS may execute the statement. For example, the DBMS may access a memory buffer, a reference or index a file for read, write, deletion, or any combination thereof, for responding to the statement. The DBMS may load the data from persistent or non-persistent storage and perform computations to respond to the query. The DBMS may return the result(s) to the user or software application.
The computing system of
For example, a GUI may first obtain a notification from a software application requesting that a particular data object be presented within the GUI. Next, the GUI may determine a data object type associated with the particular data object, e.g., by obtaining data from a data attribute within the data object that identifies the data object type. Then, the GUI may determine any rules designated for displaying that data object type, e.g., rules specified by a software framework for a data object class or according to any local parameters defined by the GUI for presenting that data object type. Finally, the GUI may obtain data values from the particular data object and render a visual representation of the data values within a display device according to the designated rules for that data object type.
Data may also be presented through various audio methods. In particular, data may be rendered into an audio format and presented as sound through one or more speakers operably connected to a computing device.
Data may also be presented to a user through haptic methods. For example, haptic methods may include vibrations or other physical signals generated by the computing system. For example, data may be presented to a user using a vibration generated by a handheld computer device with a predefined duration and intensity of the vibration to communicate the data.
The above description of functions presents only a few examples of functions performed by the computing system of
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this disclosure, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as disclosed herein. Accordingly, the scope of the invention should be limited only by the attached claims.
This application is a continuation application of U.S. application Ser. No. 16/940,604, filed Jul. 28, 2020, the entirety of which is hereby incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16940604 | Jul 2020 | US |
| Child | 17876768 | US |
