Patent Application
20220229841
Digitizing a workflow process can often increase the efficiency of performing the process. For example, converting a series of tasks to a digital and automated process can save a significant amount of time and resources. This can be especially true when the workflow process includes complex, repetitive, and/or many individual steps. Examples of these workflow processes can include approvals, notifications, and record operations, among others. When moved to a cloud service, the automated process can be managed from a single design environment. In many cases, the digitizing and automating of tasks in a single design environment provides a more consistent user experience.
Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.
The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.
Integration of database streaming for automated workflow processes is disclosed. Using the techniques disclosed herein, a process automation environment can support the streaming of data from external datastores, such as external databases. In various embodiments, as part of creating an automated process using a cloud-based process design environment, a data source step utilizing an external data source can be created with little to no programming experience. The interested data source can be specified, for example, by specifying a data source request such as a database SQL query along with data source connection parameters. When the automated process is executed, the requested data is retrieved as part of a step of the automated process as streaming data that can be iterated over. For example, each iteration step of the streamed data can process a converted object that corresponds to a transformed record or database query result element retrieved from an external database. In some embodiments, the objects are serialized into a human-readable text format that can be consumed by iterating over the data stream. In some embodiments, as part of the iteration step, the streamed data can further be stored in a cloud-based datastore, for example, as part of a migration process for migrating the data from the external database to a cloud-based datastore.
In some embodiments, the database streaming request is delegated to an intermediate server with the appropriate access to the external data source. The intermediate server requests the data by executing the data request and processes the request results as chunks of data. In some embodiments, each chunk is processed to serialize and convert the included database records or database query result elements into a converted object format that can be consumed by the automated process. The processed objects can be paginated into attachments that are written to a datastore of the cloud instance by the intermediate server. In some embodiments, the datastore is a memory, such as a scratch memory, of the cloud instance, a database of the cloud instance, or another appropriate datastore accessible by the cloud instance. Once the requested data is accessible at the cloud instance, the attachments are processed and provided to the automated process. The automated process can consume the streamed data by iterating over the transformed objects. In some embodiments, the transformation from records stored using an external database schema to the converted object format utilized by an automated process is performed by an automated transformation process and does not require a user to create or program a migration script. The process automation environment of the cloud instance facilitates the automatic conversion of external data to a format consumable by the automated process.
In some embodiments, a specification of data to be obtained from an external database during execution of an automated process configured using the process automation environment is received via a user interface of a process automation environment. For example, a user configures an external data source step for utilizing streaming data as an input for an automated process. The user interface is a graphical user interface that allows the automated process to be configured with little to no programming experience. In some embodiments, the specification of the data can include a database query, database configuration parameters such as database credentials, pre-processing steps for manipulating the requested data, and a transformation specification to automate the conversion of a database query result element to a converted object format used by the automated process. An indication of the data to be obtained from the external database is provided to an intermediary server. For example, a data stream request message is created and transmitted to an intermediary server to retrieve the requested data. The intermediary server requests and obtains the data from the external database and sends a stream of the obtained data of the external database. For example, in response to the data stream request message, the intermediary server executes the corresponding data request such as an SQL database query and provides the query results as a stream. In some embodiments, the intermediary server streams the data as chunks of paginated attachments after processing the database query results. The stream of the obtained data of the external database is received from the intermediary server. For example, the paginated attachments are processed and provided to an automated process as a stream of objects. The obtained data of the external database is used in the automated process of the process automation environment. For example, by iterating over the provided object data, the automated process can utilize the obtained data of the external database.
In some embodiments, remote server 111 is a customer server located within the customer's private network and is shielded from certain connections of network 151 via a firewall or another appropriate technique. Remote server 111 can be configured to disallow incoming network connections but maintains a private and secure connection with cloud service 101, for example, by initiating one or more outgoing connections from remote server 111 to cloud service 101. In some embodiments, external database 119 is an external customer database and can be used to store data using a custom schema. External database 119 can be located within a customer's private network and may even be located on the same private network as remote server 111 (not shown). In various embodiments, external database 119 does not allow database connections directly from cloud service 101 but does allow connections from remote server 111. In the example shown, an automation workflow design tool provided by cloud service 101 is utilized by client 131 to configure an automated process to process data stored in external database 119.
In some embodiments, remote server 111 is utilized by cloud service 101 as an intermediary server to access data from external database 119. For example, external database 119 can be shielded from cloud service 101 whereas remote server 111 can be configured with the required network permissions to create database connections to and execute queries on external database 119. To support data streaming of data from external database 119, cloud service 101 can utilize a private connection with remote server 111 to retrieve the interested data from external database 119, for example, for executing automated processes. In some embodiments, cloud datastore 109 is used as an intermediate storage for processing the streamed data and/or for migrating the data from external database 119. In some embodiments, cloud datastore 109 is also a configuration management database (CMDB) used by cloud service 101 for providing CMDB services related to automated process workflow solutions. Among other information, cloud datastore 109 can store configuration information related to configured automated processes and managed assets, such as related hardware and/or software configurations.
In some embodiments, network clients such as client 131 can access cloud service 101 to create a custom workflow that utilizes an external data source such as external database 119. The external data source can be configured as a stream data source utilized in one of the steps of the automated workflow process. Cloud service 101 supplies to a client, such as client 131, an interactive visual design tool. Utilizing the visual design tool, the external data source can be configured as a step of an automated workflow. In various embodiments, the visual design tool allows a user to create the automated workflow without requiring the user to write any software or programming code. In some embodiments, the created automated process can be executed via cloud service 101, and when executed, retrieves and processes data from the configured external data source. In some embodiments, client 131 is an example network client and can run on one of many different computing devices, including laptops, desktops, mobile devices, tablets, kiosks, smart televisions, etc. In various embodiments, client 131 is a network client such as a web browser for accessing the cloud-based services of cloud service 101.
Although single instances of some components have been shown to simplify the diagram, additional instances of any of the components shown in
In some embodiments, the data streaming components of a process automation environment work together to provide data from an external database to an automated process via a streaming interface. For example, an automated process evaluated by application engine 203 includes a data streaming step to retrieve data from external database 219. When the data stream action step of a workflow is encountered, application engine 203 creates a data stream request message that is transmitted to remote server 211 over network connection 251 via message engines 205 and 213. In some embodiments, remote server 211 is configured to disallow incoming connections from cloud instance 201 and network connection 251 is established by remote server 211. For example, message engine 213 of remote server 211 establishes network connection 251 with message engine 205 of cloud instance 201 and listens for incoming messages, such as incoming data request messages. At remote server 211 and message engine 213, a data stream request message is received and includes a data request specification. The message is processed by process and transform engine 215. Process and transform engine 215 initiates the opening of a database connection to the requested external database, in this case external database 219, using database connection engine 217. A database query associated with the data stream request is executed by database connection engine 217 and the results are passed back to process and transform engine 215.
In some embodiments, the database query results from external database 219 are provided in chunks that are paginated into attachments by process and transform engine 215 for transmission to cloud instance 201. For example, process and transform engine 215 transforms the chunk results into serialized objects based on the received data request specification. The converted objects are compatible with the automated process being evaluated at application engine 203. In some embodiments, once an attachment of converted data is prepared, an attachment is written to cloud datastore 209 by process and transform engine 215 (shown by the dashed line). At cloud instance 201, application engine 203 processes the received attachments as streaming data. In various embodiments, application engine 203, in evaluating an automated process, iterates over the streaming data. For example, application engine 203 can iterate over each object in the streaming data. In some embodiments, a pre-processing step can be performed, such as a pre-processing script for transforming the received objects.
In some embodiments, as part of an automated process configuration process, the data streaming components of a process automation environment are used to automatically generate a transformation specification for converting records stored in external database 219 into objects compatible with an automated process executed at application engine 203. For example, in advance of executing an automated process, a test database query is executed using the request path described above. When the result of the sample query is received, the fields of the database query result are evaluated and a transformation specification from a database record or query result element to a converted object format is automatically generated and provided to application engine 203. A user can be presented with the automatically determined converted object format to confirm the validity of the format and is provided with the option to make customized modifications to the transformation specification. For example, a user can transform one or more fields as appropriate to convert the data into a more manageable or appropriate form. In some embodiments, an object schema may also be automatically created, for example, for writing the received objects into a cloud-base database such as cloud datastore 209.
Although specific components are shown in
At 301, an automated process with an external data source is configured. For example, a user using an interactive visual design tool configures an automated process. The configuration process requires little to no programming experience and the user is not required to write software or programming code. In some embodiments, the automated process is created by specifying a sequence of action and trigger steps. The actions and triggers can operate on input data such as data retrieved from an external database.
At 303, the automated process is activated. For example, the user activates the newly configured automated process that relies on data streamed from an external database. This allows the automated process to be evaluated by the cloud instance. In some embodiments, the process is scheduled to run at a predefined time and/or over pre-defined intervals.
At 305, data from the external data source is streamed. For example, as part of evaluating the automated process, data is retrieved from the external data source, such as an external database located in a customer's or a third party's internal network. The data can be retrieved via an intermediary server and is transformed from the database schema to a converted object format compatible with the automated process. In various embodiments, the streamed data is iterated over by one or more steps of the automated process, for example, by iterating over each transformed object in the streamed data. The workflow steps for streaming the data are typically just one of many action steps of the automated workflow. Once the data streaming step is complete, access to the external data source is no longer required.
At 401, an external data source connection is configured. For example, a user of a network client accesses an interactive visual design tool to configure an external data source connection to be used as the database source for a streaming data step of an automated process. The connection information can include the hostname, IP address, and/or another identifier of the external database along with additional connection information such as a database network port and database access credentials. For example, database credentials can include a username and login password. In some embodiments, the credentials utilize one of several supported credential frameworks. In various embodiments, the external data source connection is configured using a predefined connection referenced by a connection alias or name.
At 403, a data request is configured. For example, using an interactive visual design tool, a user configures a data request such as a database query. In some embodiments, the database query is specified as an SQL database query. The query can reference variables that are available in the context of the workflow process. For example, a global variable and/or a variable assigned from a previous step of the workflow process can be utilized in the specified SQL database query.
At 405, pre-processing actions are configured. For example, using an interactive visual design tool, a user configures optional pre-processing actions that can run before the streaming action sends the initial data request. For example, a pre-processing action can validate or set default values.
At 407, a data transformation specification is automatically determined. For example, a transformation specification to convert the data from the database format for database query result elements into a converted object format compatible with the automated process is performed. In some embodiments, the automated conversion process is performed by initiating a test database query and analyzing the returned query results. The results are mapped to a converted object format, for example, by identifying the fields of the query result and their corresponding data types. In some embodiments, an object schema is automatically generated along with a transformation specification. The object schema can be utilized for storing the streamed objects into a cloud datastore.
At 409, the data transformation specification is confirmed. For example, the data transformation specification automatically generated at 407 is presented to the user via an interactive visual design tool. The user can view the mapping of a database query result element to a converted object format and confirm that the transformation specification is correct. In some embodiments, the user can further modify the transformation specification, for example, to apply post-processing to specific fields and/or to modify a field's data type. The transformation can be applied to convert the retrieved data into a more manageable or appropriate form. For example, a user can modify the data transformation specification to concatenate two fields corresponding to a first name field and last name field into a single name field. As another example, a user can modify the data to convert binary data into a human-readable text format such as JavaScript Object Notation (JSON).
At 411, post-processing for iterative actions are configured. For example, using an interactive visual design tool, a user configures one or more post-processing actions to iterate over the retrieved object data. In some embodiments, the post-processing actions are subsequent steps that follow the streaming data step and are used to consume and process the streaming data. For example, an iterative action can be configured to write each retrieved object from the streaming data into a cloud database table. As another example, additional iterative actions can be used to create a new notification based on each retrieved object.
At 501, a data stream action step of a workflow is received. For example, an application engine such as application engine 203 receives a data stream action step. In some embodiments, the data stream action step specifies an external database connection, the data to request from the external database, pre-processing actions, and a data transformation specification, among other potential configuration settings. In some embodiments, the pre-processing actions are performed at 501. For example, pre-processing actions such as validation and initialization actions can be performed.
At 503, a data stream request message is created and transmitted. For example, a data stream request message is prepared that includes the required information for an intermediary server to request and process the requested data. In some embodiments, the message is prepared and transmitted using message engine 205 of
At 505, streamed data is received as an attachment. For example, streamed data corresponding to at least a portion of the requested data is provided to the cloud instance by the intermediary server. In some embodiments, the streamed data is written to a cloud datastore of the cloud instance, such as cloud datastore 109 of
At 507, the streamed data attachment is iterated over and data objects are processed. For example, an automated process is applied to consume the streamed database data as data objects. As the streamed data is iterated over by the steps of the automated process, the objects of the stream are consumed and can be processed one at a time. In some embodiments, the objects can be further transformed or modified as part of the workflow process. For example, the objects can be written to a cloud datastore as a migration step to duplicate (or transform) the data from the external database to a cloud-based database. In various embodiments, the automated process iterates over the third-party database data to create notifications, incident item reports, record operations, or perform other object-based actions.
At 509, a determination is made whether additional data exists. In the event additional data exists, processing loops back to step 505. In the event no additional data exists, processing completes.
At 601, a received data stream request message is processed. For example, a data stream request message from a cloud instance is received at a message engine of a remote server such as message engine 213 of
At 603, a data request to an external data source is performed. For example, a database query is performed to retrieve the requested data from the specified external database. In some embodiments, a connection to the external database is established and the requested data is retrieved by executing a specified database query. In some embodiments, the connection is established, and an SQL query is executed using a database connection engine such as database connection engine 217 of
At 605, a data request result chunk is received. For example, in response to the data request performed at 603, the database query result is received in one or more chunks of query result data. In various embodiments, the query results are processed as each chunk of results is made available by the external database since the total amount of data returned by the data request can be arbitrarily large and consist of multiple result data chunks. As additional chunks are prepared by the external database, step 605 can be repeated to receive the additional data results.
At 607, the data request chunk is processed into a data attachment. In some embodiments, the data is processed by applying a transformation specification. In various embodiments, each data chunk corresponds to the at least a portion of the results of a database query. A transform specification is utilized to convert the data from a database format or schema to a converted object format compatible with the automated process of the cloud instance. In some embodiments, additional preprocessing steps may be performed prior to and/or during the conversion process. For example, two fields can be merged, the data type of a field can be modified, the data can be converted to a human-readable object text format, or another customized pre-processing modification or transformation can be performed.
In various embodiments, the data request chunk is processed into a data attachment. Since the data results can be arbitrarily long, the data request chunk is paginated into a data attachment. In various embodiments, the number of data request chunks used to create an attachment can depend on the data request chunk and attachment sizes. In some embodiments, one or more data request chunks are used to create a single attachment.
At 609, the data attachment is written to a cloud data store. For example, the data attachment is written to a cloud datastore of the cloud instance such as cloud datastore 209 of
At 611, a determination is made whether additional data exists. In the event additional data exists, processing loops back to step 607. In the event no additional data exists, processing completes.
In some embodiments, the configuration settings shown in user interface 700 correspond to step label 701, a streaming data step of an automated workflow. As shown in user interface 700, step label 701 is labeled as step 1 and is a JDBC database streaming step. The JDBC database streaming step is used to configure an external database as a streaming data source using a Java Database Connectivity (JDBC) interface. Configuration settings of step 1 are shown in configuration section 705. As shown in the diagram, configuration section 705 includes a connection details section and a JDBC configuration. The connection details section allows the user to set up a database connection and can correspond to the settings configured at 401 of
As shown in the example, the right portion of user interface 700 is a data section. The data window includes target object format section 707. Target object format section 707 shows the converted object format described by the transformation specification. For example, target object format section 707 specifies the six fields and corresponding data types of the target object. As shown in the example, the target object has an integer Id field, an integer PostId field, an integer Score field, a Date/Time CreationDate field, an integer UserId field, and a String Test field. In various embodiments, the displayed converted object format of target object format section 707 corresponds to the data transformation specification automatically determined at 407 of
In some embodiments, the Process Engine and Script transform and for-each iteration components of the ServiceNow instance of diagram 800 correspond to application engine 203 of
In the example shown, the arrows between the components of diagram 800 correspond to interactions performed for implementing an external database as a streaming data source. The interactions performed by the left half of diagram 800 correspond to one or more steps of the process of
Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.
