Persistent query dispatch and execution architecture

Abstract

The disclosed subject matter includes systems, methods, and computer readable medium for improving performance of a computer data system. An electronic request for a remote query processor (RQP) can be sent from the persistent query controller to a remote query dispatcher (RQD) executing on a query server computer. The request can include parameters for configuring the RQP and an operating environment for the RQP. The RQD can automatically attempt to allocate an isolated operating environment for the RQP and to prepare the RQP on the query server computer. When the RQP is prepared, performing: providing the controller with an address assignment of the RQP; automatically connecting from the controller to the RQP via a network; transmitting a persistent database query electronically from the controller to the RQP; publishing persistent database query configuration information including a query state and the RQP address assignment; and connecting from a client to the RQP.

Description

Embodiments relate generally to computer data systems, and more particularly, to methods, systems and computer readable media for persistent query connection architecture.

Some graphical user interfaces may provide a display of information from a database query result. However, in the case of data that is changing over time and would cause a change in a query result over time, a typical static query result display may not provide an up-to-date visualization of the changed data. A need may exist to provide a dynamically updating display of a query result that is changing over time. Also, a need may exist to provide an access control mechanism for allowing a user to create dynamically updating, long running, repeated, and/or automatically started queries, share access to queries with other users, share query results between queries, provide real-time data to a GUI/console, provide real-time data to a query distributed across multiple jobs, enforce access controls based on user roles, and/or provide remote debugging of a running query.

Embodiments were conceived in light of the above mentioned needs, problems and/or limitations, among other things.

Some implementations can include a computer data system having a persistent query dispatch and execution architecture, the system can comprise one or more processors and computer readable storage coupled to the one or more processors, the computer readable storage having stored thereon instructions that, when executed by the one or more processors, cause the one or more processors to perform operations. The operations can include sending an electronic request for a remote query processor from the persistent query controller to a remote query dispatcher executing on a query server computer. The request can include parameters for configuring the remote query processor and an operating environment for the remote query processor. The operations can also include attempting, at the remote query dispatcher, to allocate an isolated operating environment for the remote query processor and to start execution of the remote query processor on the query server computer.

When the remote query processor is started, operations can be performed that include: providing the persistent query controller with an address assignment of the remote query processor or of a proxy machine in communication with the remote query processor, the address assignment identifying a specific address of the query server computer or of the proxy machine available to connect electronically over an electronic communications network; automatically connecting from the persistent query controller to the remote query processor via the electronic communications network; transmitting a persistent database query electronically from the persistent query controller to the remote query processor; publishing persistent database query configuration information including a state of the persistent database query and the address assignment of the remote query processor; and/or connecting from a client to the remote query processor via the electronic communications network.

The operations can further include, when the connection by the client is allowed, receiving at the client at least a portion of a current result of the persistent database query from the remote query processor.

The operations can further include, when the connection by the client is allowed, filtering, based on the access control information, a current result of the persistent database query requested by the client from the remote query processor, and sending at least a portion of the filtered current result of the persistent database query to the client.

The operations can further include determining whether to allow the connection by the client to the remote query processor based on access control information associated with the persistent database query, and, when the connection by the client is allowed, sending a request to perform an administrative operation with respect to the persistent database query from the client to the persistent query controller, and determining whether the client is authorized to perform the administrative operation based on the access control information.

The operations can further include sending, from a second client different than the client, an instruction to the persistent query controller to start, stop, restart, modify parameters, or modify code of the persistent database query.

The operations can further include, when the connection by the client is allowed, receiving a result of the persistent database query at the client, displaying at least a portion of the result at the client via a graphical user interface and/or a console, receiving at least a portion of an updated result of the persistent database query from the remote query processor; and responsive to the receiving the at least a portion of the updated result, updating the graphical user interface and/or console to display the at least a portion of the updated result.

The operations can further include, determining whether the remote query processor rejects the request for a remote query processor from the persistent query controller. When the remote query dispatcher rejects the request, the operations can include publishing an indication of the rejection. The operations can further include detecting, by the remote query processor or remote query dispatcher, an error in the execution of the persistent database query and, when the remote query processor or remote query dispatcher detects an error in the execution of the persistent database query, publishing an indication of the error.

The operations can further include, when the connection by the client is allowed, transmitting an additional query task electronically from the client to the remote query processor, executing, at the remote query processor, the additional query task, and receiving at least a portion of a result of the additional query task at the client.

The operations can further include periodically providing a liveness indication from the persistent query controller to the remote query dispatcher, and when the liveness indication is not received after a limited amount of time, stopping the remote query processor.

Some implementations can include a method for improving performance of a computer data system through control of a persistent query dispatch and execution architecture. The method can include sending an electronic request for a remote query processor from the persistent query controller to a remote query dispatcher executing on a query server computer. The request can include parameters for configuring the remote query processor and an operating environment for the remote query processor. The method can also include automatically attempting, at the remote query dispatcher, to allocate an isolated operating environment for the remote query processor and to prepare the remote query processor on the query server computer.

When the remote query processor is prepared, the method can include performing operations. The operations can include providing the persistent query controller with an address assignment of the remote query processor or of a proxy machine in communication with the remote query processor, the address assignment identifying a specific address of the query server computer or of the proxy machine available to connect electronically over an electronic communications network. The operations can also include automatically connecting from the persistent query controller to the remote query processor via the electronic communications network. The operations can further include transmitting a persistent database query electronically from the persistent query controller to the remote query processor. The operations can also include publishing persistent database query configuration information including a state of the persistent database query and the address assignment of the remote query processor. The operations can further include connecting from a client to the remote query processor via the electronic communications network.

The method can further include, when the connection by the client is allowed, receiving at the client at least a portion of a current result of the persistent database query from the remote query processor.

The method can further include, when the connection by the client is allowed, filtering, based on the access control information, a current result of the persistent database query requested by the client from the remote query processor, and sending at least a portion of the filtered current result of the persistent database query to the client.

The method can further include determining whether to allow the connection by the client to the remote query processor based on access control information associated with the persistent database query. The method can further include, when the connection by the client is allowed, sending a request to perform an administrative operation with respect to the persistent database query from the client to the persistent query controller, and determining whether the client is authorized to perform the administrative operation based on the access control information.

The method can further include, sending, from a second client different than the client, an instruction to the persistent query controller to start, stop, or restart the persistent database query.

The method can further include, when the connection by the client is allowed, receiving a result of the persistent database query at the client. The method can also include displaying at least a portion of the result at the client via a graphical user interface and/or a console. The method can further include receiving at least a portion of an updated result of the persistent database query from the remote query processor. The method can further include, responsive to the receiving the at least a portion of the updated result, updating the graphical user interface and/or console to display the at least a portion of the updated result.

The method can include determining whether the remote query processor rejects the request for a remote query processor from the persistent query controller. When the remote query dispatcher rejects the request, the method can include publishing an indication of the rejection. The method can include detecting, by the remote query processor or remote query dispatcher, an error in the execution of the persistent database query, and, when the remote query processor or remote query dispatcher detects an error in the execution of the persistent database query, the method can further include publishing an indication of the error.

The method can further include, when the connection by the client is allowed, transmitting an additional query task electronically from the client to the remote query processor, executing, at the remote query processor, the additional query task, and optionally receiving at least a portion of a result of the additional query task at the client.

The method can include periodically providing a liveness indication from the persistent query controller to the remote query dispatcher, and when the liveness indication is not received after a limited amount of time, stopping the remote query processor.

Some implementations can include a nontransitory computer readable medium having stored thereon software instructions that, when executed by one or more processors, cause the processors to perform operations. The operations can include sending an electronic request for a remote query processor from the persistent query controller to a remote query dispatcher executing on a query server computer, wherein the request includes parameters for configuring the remote query processor and an operating environment for the remote query processor. The operations can also include automatically attempting, at the remote query dispatcher, to allocate an isolated operating environment for the remote query processor and to run of the remote query processor on the query server computer.

When the remote query processor is running, the operations can include performing additional operations. The additional operations can include providing the persistent query controller with an address assignment of the remote query processor or of a proxy machine in communication with the remote query processor, the address assignment identifying a specific address of the query server computer or of the proxy machine available to connect electronically over an electronic communications network. The additional operations can also include automatically connecting from the persistent query controller to the remote query processor via the electronic communications network. The additional operations can further include transmitting a persistent database query electronically from the persistent query controller to the remote query processor. The additional operations can also include publishing persistent database query configuration information including a state of the persistent database query and the address assignment of the remote query processor. The additional operations can further include connecting from a client to the remote query processor via the electronic communications network.

In any of the above-mentioned implementations, the client can be another remote query processor.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an example computer data system showing an example data distribution configuration in accordance with some implementations.

FIG. 2 is a diagram of an example computer data system showing an example administration/process control arrangement in accordance with some implementations.

FIG. 3 is a diagram of an example computing device in accordance with some implementations.

FIG. 4 is a diagram of an example persistent query dispatch and execution architecture in accordance with some implementations.

FIG. 5 is a flowchart of an example method of starting a persistent query in accordance with some implementations.

FIG. 6 is a flowchart of an example method of connecting a secondary client to a persistent query in accordance with some implementations.

DETAILED DESCRIPTION

Reference may be made herein to the Java programming language, Java classes, Java bytecode and the Java Virtual Machine (JVM) for purposes of illustrating example implementations. It will be appreciated that implementations can include other programming languages (e.g., groovy, Scala, R, Go, etc.), other programming language structures as an alternative to or in addition to Java classes (e.g., other language classes, objects, data structures, program units, code portions, script portions, etc.), other types of bytecode, object code and/or executable code, and/or other virtual machines or hardware implemented machines configured to execute a data system query.

FIG. 1 is a diagram of an example computer data system and network 100 showing an example data distribution configuration in accordance with some implementations. In particular, the system 100 includes an application host 102, a periodic data import host 104, a query server host 106, a long-term file server 108, and a user data import host 110. While tables are used as an example data object in the description below, it will be appreciated that the data system described herein can also process other data objects such as mathematical objects (e.g., a singular value decomposition of values in a given range of one or more rows and columns of a table), TableMap objects, etc. A TableMap object provides the ability to lookup a Table by some key. This key represents a unique value (or unique tuple of values) from the columns aggregated on in a byExternal( ) statement execution, for example. A TableMap object is can be the result of a byExternal( ) statement executed as part of a query. It will also be appreciated that the configurations shown in FIGS. 1 and 2 are for illustration purposes and in a given implementation each data pool (or data store) may be directly attached or may be managed by a file server.

The application host 102 can include one or more application processes 112, one or more log files 114 (e.g., sequential, row-oriented log files), one or more data log tailers 116 and a multicast key-value publisher 118. The periodic data import host 104 can include a local table data server, direct or remote connection to a periodic table data store 122 (e.g., a column-oriented table data store) and a data import server 120. The query server host 106 can include a multicast key-value subscriber 126, a performance table logger 128, local table data store 130 and one or more remote query processors (132, 134) each accessing one or more respective tables (136, 138). The long-term file server 108 can include a long-term data store 140. The user data import host 110 can include a remote user table server 142 and a user table data store 144. Row-oriented log files and column-oriented table data stores are discussed herein for illustration purposes and are not intended to be limiting. It will be appreciated that log files and/or data stores may be configured in other ways. In general, any data stores discussed herein could be configured in a manner suitable for a contemplated implementation.

In operation, the input data application process 112 can be configured to receive input data from a source (e.g., a securities trading data source), apply schema-specified, generated code to format the logged data as it's being prepared for output to the log file 114 and store the received data in the sequential, row-oriented log file 114 via an optional data logging process. In some implementations, the data logging process can include a daemon, or background process task, that is configured to log raw input data received from the application process 112 to the sequential, row-oriented log files on disk and/or a shared memory queue (e.g., for sending data to the multicast publisher 118). Logging raw input data to log files can additionally serve to provide a backup copy of data that can be used in the event that downstream processing of the input data is halted or interrupted or otherwise becomes unreliable.

A data log tailer 116 can be configured to access the sequential, row-oriented log file(s) 114 to retrieve input data logged by the data logging process. In some implementations, the data log tailer 116 can be configured to perform strict byte reading and transmission (e.g., to the data import server 120). The data import server 120 can be configured to store the input data into one or more corresponding data stores such as the periodic table data store 122 in a column-oriented configuration. The periodic table data store 122 can be used to store data that is being received within a time period (e.g., a minute, an hour, a day, etc.) and which may be later processed and stored in a data store of the long-term file server 108. For example, the periodic table data store 122 can include a plurality of data servers configured to store periodic securities trading data according to one or more characteristics of the data (e.g., a data value such as security symbol, the data source such as a given trading exchange, etc.).

The data import server 120 can be configured to receive and store data into the periodic table data store 122 in such a way as to provide a consistent data presentation to other parts of the system. Providing/ensuring consistent data in this context can include, for example, recording logged data to a disk or memory, ensuring rows presented externally are available for consistent reading (e.g., to help ensure that if the system has part of a record, the system has all of the record without any errors), and preserving the order of records from a given data source. If data is presented to clients, such as a remote query processor (132, 134), then the data may be persisted in some fashion (e.g., written to disk).

The local table data server 124 can be configured to retrieve data stored in the periodic table data store 122 and provide the retrieved data to one or more remote query processors (132, 134) via an optional proxy.

The remote user table server (RUTS) 142 can include a centralized consistent data writer, as well as a data server that provides processors with consistent access to the data that it is responsible for managing. For example, users can provide input to the system by writing table data that is then consumed by query processors.

The remote query processors (132, 134) can use data from the data import server 120, local table data server 124 and/or from the long-term file server 108 to perform queries. The remote query processors (132, 134) can also receive data from the multicast key-value subscriber 126, which receives data from the multicast key-value publisher 118 in the application host 102. The performance table logger 128 can log performance information about each remote query processor and its respective queries into a local table data store 130. Further, the remote query processors can also read data from the RUTS, from local table data written by the performance logger, or from user table data read over NFS, for example.

It will be appreciated that the configuration shown in FIG. 1 is a typical example configuration that may be somewhat idealized for illustration purposes. An actual configuration may include one or more of each server and/or host type. The hosts/servers shown in FIG. 1 (e.g., 102-110, 120, 124 and 142) may each be separate or two or more servers may be combined into one or more combined server systems. Data stores can include local/remote, shared/isolated and/or redundant. Any table data may flow through optional proxies indicated by an asterisk on certain connections to the remote query processors. Also, it will be appreciated that the term “periodic” is being used for illustration purposes and can include, but is not limited to, data that has been received within a given time period (e.g., millisecond, second, minute, hour, day, week, month, year, etc.) and which has not yet been stored to a long-term data store (e.g., 140).

FIG. 2 is a diagram of an example computer data system 200 showing an example administration/process control arrangement in accordance with some implementations. The system 200 includes a production client host 202, a controller host 204, a GUI host or workstation 206, and query server hosts 208 and 210. It will be appreciated that there may be one or more of each of 202-210 in a given implementation.

The production client host 202 can include a batch query application 212 (e.g., a query that is executed from a command line interface or the like) and a real time query data consumer process 214 (e.g., an application that connects to and listens to tables created from the execution of a separate query). The batch query application 212 and the real time query data consumer 214 can connect to a remote query dispatcher 222 and one or more remote query processors (224, 226) within the query server host 1208.

The controller host 204 can include a persistent query controller 216 configured to connect to a remote query dispatcher 232 and one or more remote query processors 228-230. In some implementations, the persistent query controller 216 can serve as the “primary client” for persistent queries and can request remote query processors from dispatchers, and send instructions to start persistent queries. For example, a user can submit a query to 216, and 216 starts and runs the query every day. In another example, a securities trading strategy could be a persistent query. The persistent query controller can start the trading strategy query every morning before the market opened, for instance. It will be appreciated that 216 can work on times other than days. In some implementations, the controller may require its own clients to request that queries be started, stopped, etc. This can be done manually, or by scheduled (e.g., cron jobs). Some implementations can include “advanced scheduling” (e.g., auto-start/stop/restart, time-based repeat, etc.) within the controller.

The GUI/host workstation can include a user console 218 and a user query application 220. The user console 218 can be configured to connect to the persistent query controller 216. The user query application 220 can be configured to connect to one or more remote query dispatchers (e.g., 232) and one or more remote query processors (228, 230).

FIG. 3 is a diagram of an example computing device 300 in accordance with at least one implementation. The computing device 300 includes one or more processors 302, operating system 304, computer readable medium 306 and network interface 308. The memory 306 can include persistent query dispatch/execution application 310 (e.g., console 218 and/or user query application 220) and a data section 312 (e.g., for persistent query configuration information, index data structures, column source maps, etc.).

In operation, the processor 302 may execute the application 310 stored in the memory 306. The application 310 can include software instructions that, when executed by the processor, cause the processor to perform operations for persistent query dispatch/execution in accordance with the present disclosure (e.g., performing one or more of 502-512, or one or more of 602-610 described below). The application program 310 can operate in conjunction with the data section 312 and the operating system 304.

In some embodiments, application program 310 includes a graphical user interface (GUI) that includes a console (e.g., console 218) and/or a user query application (e.g., 220). The GUI can include a configuration panel that provides information on jobs running on a query server host (e.g., 106 and/or 210, as shown in FIG. 1 and FIG. 2, respectively) and/or queries being managed by a persistent query controller (e.g., 216 and/or 410, as shown in FIG. 2 and FIG. 4, respectively). The information provided by the configuration panel can include: job owner, type of job (script, trading strategy, etc.), state of the job (running, exception, stopped, etc.), job start time, etc. The GUI can allow a user to perform an administrative action (e.g., start, stop, restart, edit, or view) on one or more jobs (e.g., by allowing the user to select one or more jobs, right click on the selection, and choose the action to be performed for the selected jobs). In some embodiments, jobs can also include simulated jobs that can replay historical data.

In some embodiments, the GUI includes elements that allow users to add a new job and configure it with the appropriate parameters. For example, right clicking on a query can allow a user to edit the parameters for the query (e.g. change RAM usage). The editable parameters can include: memory usage (e.g., RAM usage), virtual machine parameters (e.g., Java Virtual Machine (JVM) parameters), operating system parameters (e.g. shell variables), code executed, access control parameters (e.g., access control 404), configuration type (e.g. script, trading strategy, or replay), query server (or load balancer) (e.g., query host 210 or 408), classpath, enabled, script location or actual script code, script language, start time, and/or stop time. In some embodiments, the GUI elements include start/stop/restart/edit/view, etc. admin actions.

In some embodiments, users have permission levels and resource allocations. Based on the user permissions and resources, the user can create a query. The user “owns” their query (i.e., the user is set as the owner of the query) and can set access control information (e.g., access control 404 as shown in FIG. 4) that limits access to the query and the remote query processor executing the query. A user's ability to limit access to their queries can be limited by, for example, system-level access controls (e.g., allow supervisor users to have access to all queries of the their respective subordinates) which the user cannot override. User setable access control information can include an admin group (e.g., admin groups 412) and a viewer group (e.g., viewer groups 414). The query runs as the owner and is limited in scope to what the owner can see. In some embodiments, the owner, zero or more administrators, and zero or more global super-users are able to view all of the results of the query; and also edit, view the code, stop it, start it, etc. The viewers can view the results of the query, but are not allowed to view the code or perform any administrative actions. In some embodiments, viewers can be limited to viewing only a limited set of the results of the query. In some embodiments, administrators, global super users, and/or owners may allow viewers to restart the query when it is already stopped.

FIG. 4 is a diagram of an example persistent query dispatch and execution architecture 400 in accordance with some implementations. Persistent query dispatch and execution architecture 400 includes a controller host 402, a query host 408, a client host 406, and an access control 404. Controller host 402 includes a persistent query controller (PQC) 410, which includes a primary client 416. Query host 408 includes a remote query dispatcher (420) and a remote query processor (RQP) 422. Client host 406 includes a secondary client 418. Access control 404 includes access control groups/roles (e.g., as discussed above) including admin groups 412 and viewer groups 414.

In operation, secondary client 418 transmits data to and receives data from PQC 410 to, for example, transmit an instruction to PQC 410 to start a persistent query (e.g., as shown in FIG. 5) and/or receive persistent query configuration information to connect to the persistent query after it has been started on RQP 422 (e.g., as shown in FIG. 6). Primary client 416 of PQC 410 transmits data to and receives data from RQD 420 to, for example, request a RQP (RQP 422) for executing the persistent query, and RQD 420 transmits data to and receives data from RQP 422 to, for example, configure RQP 422 for the persistent query. Primary client 416 of PQC 410 connects to RQP 422 to start the persistent query. PQC 410 stores persistent query configuration information including a definition of the persistent query and access control 404. In some embodiments, the persistent query configuration information is stored in a map by PQC 410 (e.g., a persistent map). Secondary client 418 transmits data to and receives data from RQP 422 to, for example, connect to the persistent query after it has started (e.g., as shown in FIG. 6), and RQP 422 receives access control 404 information to determine whether to allow secondary client 418 to connect to the persistent query and/or perform operations with respect to the persistent query (e.g., start/stop/etc.).

Once connected to RQP 422, secondary client 418 can, based on the access control set for the query, perform various operations. For example, secondary client 418 can connect a GUI/console to the persistent query, and displaying the data involves a query task which returns the table to display. Optional access control filters are applied to the tables before they are returned to the user. If the GUI user wants to filter the dataset or otherwise massage it (e.g. Sym=‘AAPL’, sort the data, etc.), the GUI can submit a query task to RQP 422 for only the desired data (i.e., a filter query task). The GUI can submit various query tasks to, for example, control how results of a persistent query are provided by RQP 422 (e.g., filter query tasks, sort tasks to sort the dataset, etc.) and/or select only a portion of the results (i.e., a subset including at least what is being displayed and/or what is used to generate what is being displayed by the GUI).

In another example, secondary client 418 can be a debugger GUI/console that connects to RQP 422 to debug running processes and execute on the remote process. In such embodiments, secondary client 418 can connect to RQP 422 as a debugger GUI/console to fetch variables/parameters of RQP 422 and execute queries or other commends (e.g., can execute on the RQP 422 from the console (i.e., command line) of the secondary client 418.

In another example, secondary client 418 can be a query task configured to retrieve consistent snapshots of an entire table, e.g., to save the current view to a CSV file.

In yet another example, secondary client 418 can be another RQP that connects to RQP 422 in order to retrieve a preemptive table, which allows query results to be shared among remote query processors. In such an example, a remote query processor with super-user privileges could provide an aggregated position table to one or more other remote query processors that may not have permission to access the source data required to generate the aggregated position table.

Access control information 404 is provided to RQP 422 by RQC 410 (e.g., on startup and optionally updated after startup) which indicates access control groups/roles for the persistent query (as discussed above) and RQP 422 then restricts access based on access control 404.

In some embodiments, PQC 410 acts as primary client 416 for persistent queries.

In some embodiments, if PQC 410 or primary client 416 goes down; then the query is stopped. In some such embodiments, a heartbeat is used to check whether PQC 410 or primary client 416 has gone down. The query definition remains in place in the stored persistent query configuration information (e.g., stored in the persisted map); and when PQC 410 or primary client 416 is restarted, the query can also be restarted. Any displayed results are recomputed on reinitialization (in general, it is possible to write things out and reload them; but that is not the common scheme).

Secondary client 418 can request a connection to a running RQP (e.g., RQP 422) (it is secondary because it is requesting and not starting). If the secondary client has sufficient permissions, it will be allowed to connect to the RQP, where it can see and interact with the state from the primary query. In some embodiments, secondary client 418 is a graphical user interface (GUI) for di splaying output for the persistent query. The GUI connects to the existing RQP executing the persistent query (i.e., RQP 422 is concurrently used for connections by the primary client and secondary clients of that persistent query).

In some embodiments, the persistent query configuration information stored by PQC 410 includes a state of each persistent query (e.g. running, stopped, exception, etc.). In some embodiments, PQC 410 monitors the state each persistent query to prevent accidental duplicates if, for example, multiple requests to start the same persistent query are received.

In some embodiments, PQC 410 publishes persistent query configuration information and secondary connection 418 can receive the persistent query configuration information and automatically reconnect when a persistent query is restarted. In some such embodiments, when a persistent query is restarted; a brand new RQP is created. PQC 410 publishes information about the state of persistent queries in addition to publishing the contents of the stored (e.g., persistent) map. Clients (e.g., secondary client 418) receive the notifications, and if they have any display (e.g., GUI) that corresponds to the persistent query configuration that changed, the displayed results are updated (e.g., with a message saying the query is down or with the new contents of the table/widget via a connection to the new RQP that was created when the persistent query was restarted).

It will be appreciated that hosts 402, 406, and 408 can be the same or different (actual or virtual) servers. For example, in some embodiments, host 402 and host 408 can be executed on the same server, such as, for example, query server host 106 as shown in FIG. 1.

Although not show, some embodiments include more than one remote query dispatcher 420 executing on the same or different hosts (actual or virtual hosts), and a load balancer (e.g., one or more load balancers and/or a distributed load balancer). The load balancer could be implemented as a sub-module within each remote query dispatcher of the multiple dispatchers. This configuration could support a distributed system with each remote query dispatcher participating in a distributed state exchange and a single “leader” remote query dispatcher making scheduling decisions for all participating remote query dispatchers. The load balancer could also include a distributed 100% uptime load balancer. It will be appreciated that if a load balancer is included in an implementation, clients (primary client 416) may connect to the remote query dispatchers through the load balancer. When a load balancer is not included or is integrated within each remote query dispatcher, the clients (primary client 416) may connect directly to respective remote query dispatchers.

Although not show, some embodiments include more than one persistent query controller 410 executing on the same or different hosts (actual or virtual hosts), and a load balancer (e.g., one or more load balancers and/or a distributed load balancer). The load balancer could be implemented for the persistent query controllers in configurations similar to those discussed above with respect to the load balancer for the multiple remote query dispatchers.

It will be appreciated that the persistent query dispatch/execution architecture 400 is a simplified configuration for purposes of illustrating the principles of the disclosed subject matter. An actual implementation may include one or more clients, zero or more load balancers, one or more persistent query controllers, one or more remote query dispatchers and zero or more remote query processors associated with each remote query dispatcher.

FIG. 5 is a flowchart of an example method 500 of starting a persistent query in accordance with some implementations. Processing begins at 502, where a persistent query controller (PQC) (e.g., PQC 410 as shown in FIG. 4) receives an instruction from a persistent query client (e.g., host 406 or secondary client 418) to start a persistent query. Processing continues to 504.

At 504, the PQC sends a request for a remote query processor (RQP) (e.g., RQP 422) to a remote query dispatcher (RQD) (e.g., RQD 420). The PQC can start a primary client (e.g. 416) for communicating with the RQD/RQP. The request can be sent to the RQD via an optional load balancer (as described above). In some embodiments, the PQC can use one primary client for connecting to multiple RQPs. Processing continues to 506.

At 506, it is determined whether the RQP request was granted. If it was granted, then processing continues to 508; otherwise processing continues to 512.

At 508, the PQC sends RQP access control information that can include admin/viewer groups (e.g., admin groups 412 and viewer groups 414 of access control 404) and/or other access control roles (e.g., superuser, supervisor, etc., as discussed above). Processing continues to 510.

At 510, the PQC publishes RQP configuration information and state of RQP, The state information can include: the current life cycle of the query (authenticating, initializing, running, stopped, error after initialize, failed to initialize); what RQP host/port was assigned; and also the sufficient data to display the query configuration panel in the GUI client (e.g., table names, widget names, and possibly the groups allowed to access each table/widget). Other summary information can be published to augment the query configuration panel and prevent the need to connect to individual RQP simply to display the configuration panel.

At 512, the PQC publishes query failure information. The query failure information can include an indication that the query failed. In addition, the query failure information can include error messages, exception messages, and/or other data (e.g., stack traces, variables, etc.) which may enable debugging the underlying problem. For example, the RQP and/or RQD can be configured to detect errors (e.g., catch exceptions) related to the execution of the persistent query and publish such errors.

It will also be appreciated that 502-512 may be repeated in whole or in part to, for example, automatically retry starting the query if the RQP request is not granted.

FIG. 6 is a flowchart of an example method 600 of connecting a secondary client to a persistent query in accordance with some implementations. Processing begins at 602, where a secondary client (e.g., secondary client 418 as shown in FIG. 4) requests persistent query configuration(s) from a persistent query controller (PQC) (e.g., PQC 410), In some embodiments, the secondary client can request configuration information from the PQC for one or more particular queries. Additionally or alternatively, the PQC can publish available persistent query configurations. Processing continues to 604.

At 604, the secondary client receives persistent query configuration(s) from the PQC. The secondary client can receive configurations for those requested at 602, and/or available configurations published by the PQC. In some embodiments, the PQC sends/publishes valid configurations that the secondary client has access to. A persistent query configuration for a persistent query of the PQC can include: unique serial ID, name, remote query processor (RQP) ID, host and port information.

In some embodiments, 602 is omitted and the secondary client receives persistent query configuration(s) without requesting the configuration(s) from the PQC at 602 (e.g., by receiving configuration(s) published by the PQC, by manual user input, or by some other method), and

In some embodiments in which the secondary client is a GUI, the name can be used display purposes and the unique serial ID can be used to retrieve information about the query and stored in a user's workspace for saving the appropriate views across runs of the client code. For example, persistent queries can run each day, and GUI users may want to view the same results each day with particular filtering and sort combinations applied, in a specific layout. To enable users to save the layout, a serial ID can be associated with each query, and when a user saves their workspace: for each view, the serial ID of the query and the name of the table/widget that the view contains is recorded. The name/owner of the query can then be changed for administrative reasons without disconnecting it from the user's workspaces. Processing continues to 606.

At 606, the secondary client requests connection to a RQP. Processing continues to 608.

At 608, the RQP checks the access control groups/roles for secondary client. The access control groups can be set in access control information (e.g., access control 404) and can include access control groups/roles including admin, viewer and other groups/roles (as discussed above). The RQP can check the access control groups for the presence of the user authenticated on the secondary client. Additionally or alternatively, the RQP can check the network address and/or other identifier of the secondary client (e.g., IP address, digital certificate or other encryption credentials, etc.) against those set in the access control groups/roles. If secondary client is in an access control group/role, processing continues to 610. Otherwise, processing continues to 612, where the RQP rejects the connection from the secondary client.

At 610, when the secondary client is in an access control group/role, the RQP grants the connection. The RQP can grant the connection when the user of the secondary client is in one of the access control groups/roles.

It will be appreciated that 602-610 may be repeated in whole or in part. For example, 606-610 may be repeated to connect to multiple RQPs. In another example, 602-610 and/or 604-610 may be repeated to connect to RQPs as new persistent query configuration(s) become available (e.g., are requested by the secondary client and/or published by the PQC).

It will be appreciated that the modules, processes, systems, and sections described above can be implemented in hardware, hardware programmed by software, software instructions stored on a nontransitory computer readable medium or a combination of the above. A system as described above, for example, can include a processor configured to execute a sequence of programmed instructions stored on a nontransitory computer readable medium. For example, the processor can include, but not be limited to, a personal computer or workstation or other such computing system that includes a processor, microprocessor, microcontroller device, or is comprised of control logic including integrated circuits such as, for example, an Application Specific Integrated Circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (e.g., GPGPU or GPU) or the like. The instructions can be compiled from source code instructions provided in accordance with a programming language such as Java, C, C++, C#.net, assembly or the like. The instructions can also comprise code and data objects provided in accordance with, for example, the Visual Basic™ language, a specialized database query language, or another structured or object-oriented programming language. The sequence of programmed instructions, or programmable logic device configuration software, and data associated therewith can be stored in a nontransitory computer-readable medium such as a computer memory or storage device which may be any suitable memory apparatus, such as, but not limited to ROM, PROM, EEPROM, RAM, flash memory, disk drive and the like.

Furthermore, the modules, processes systems, and sections can be implemented as a single processor or as a distributed processor. Further, it should be appreciated that the steps mentioned above may be performed on a single or distributed processor (single and/or multi-core, or cloud computing system). Also, the processes, system components, modules, and sub-modules described in the various figures of and for embodiments above may be distributed across multiple computers or systems or may be co-located in a single processor or system. Example structural embodiment alternatives suitable for implementing the modules, sections, systems, means, or processes described herein are provided below.

The modules, processors or systems described above can be implemented as a programmed general purpose computer, an electronic device programmed with microcode, a hard-wired analog logic circuit, software stored on a computer-readable medium or signal, an optical computing device, a networked system of electronic and/or optical devices, a special purpose computing device, an integrated circuit device, a semiconductor chip, and/or a software module or object stored on a computer-readable medium or signal, for example.

Embodiments of the method and system (or their sub-components or modules), may be implemented on a general-purpose computer, a special-purpose computer, a programmed microprocessor or microcontroller and peripheral integrated circuit element, an ASIC or other integrated circuit, a digital signal processor, a hardwired electronic or logic circuit such as a discrete element circuit, a programmed logic circuit such as a PLD, PLA, FPGA, PAL, GP, GPU, or the like. In general, any processor capable of implementing the functions or steps described herein can be used to implement embodiments of the method, system, or a computer program product (software program stored on a nontransitory computer readable medium).

Furthermore, embodiments of the disclosed method, system, and computer program product (or software instructions stored on a nontransitory computer readable medium) may be readily implemented, fully or partially, in software using, for example, object or object-oriented software development environments that provide portable source code that can be used on a variety of computer platforms. Alternatively, embodiments of the disclosed method, system, and computer program product can be implemented partially or fully in hardware using, for example, standard logic circuits or a VLSI design. Other hardware or software can be used to implement embodiments depending on the speed and/or efficiency requirements of the systems, the particular function, and/or particular software or hardware system, microprocessor, or microcomputer being utilized. Embodiments of the method, system, and computer program product can be implemented in hardware and/or software using any known or later developed systems or structures, devices and/or software by those of ordinary skill in the applicable art from the function description provided herein and with a general basic knowledge of the software engineering and computer networking arts.

Moreover, embodiments of the disclosed method, system, and computer readable media (or computer program product) can be implemented in software executed on a programmed general purpose computer, a special purpose computer, a microprocessor, or the like.

It is, therefore, apparent that there is provided, in accordance with the various embodiments disclosed herein, methods, systems and computer readable media for persistent query dispatch and execution architecture.

application Ser. No. 15/154,974, entitled “DATA PARTITIONING AND ORDERING” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,975, entitled “COMPUTER DATA SYSTEM DATA SOURCE REFRESHING USING AN UPDATE PROPAGATION GRAPH” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,979, entitled “COMPUTER DATA SYSTEM POSITION-INDEX MAPPING” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,980, entitled “SYSTEM PERFORMANCE LOGGING OF COMPLEX REMOTE QUERY PROCESSOR QUERY OPERATIONS” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,983, entitled “DISTRIBUTED AND OPTIMIZED GARBAGE COLLECTION OF REMOTE AND EXPORTED TABLE HANDLE LINKS TO UPDATE PROPAGATION GRAPH NODES” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,984, entitled “COMPUTER DATA SYSTEM CURRENT ROW POSITION QUERY LANGUAGE CONSTRUCT AND ARRAY PROCESSING QUERY LANGUAGE CONSTRUCTS” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,985, entitled “PARSING AND COMPILING DATA SYSTEM QUERIES” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,987, entitled “DYNAMIC FILTER PROCESSING” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,988, entitled “DYNAMIC JOIN PROCESSING USING REAL-TIME MERGED NOTIFICATION LISTENER” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,990, entitled “DYNAMIC TABLE INDEX MAPPING” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,991, entitled “QUERY TASK PROCESSING BASED ON MEMORY ALLOCATION AND PERFORMANCE CRITERIA” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,993, entitled “A MEMORY-EFFICIENT COMPUTER SYSTEM FOR DYNAMIC UPDATING OF JOIN PROCESSING” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,995, entitled “QUERY DISPATCH AND EXECUTION ARCHITECTURE” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,996, entitled “COMPUTER DATA DISTRIBUTION ARCHITECTURE” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,997, entitled “DYNAMIC UPDATING OF QUERY RESULT DISPLAYS” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,998, entitled “DYNAMIC CODE LOADING” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/154,999, entitled “IMPORTATION, PRESENTATION, AND PERSISTENT STORAGE OF DATA” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/155,001, entitled “COMPUTER DATA DISTRIBUTION ARCHITECTURE” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/155,005, entitled “PERSISTENT QUERY DISPATCH AND EXECUTION ARCHITECTURE” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/155,006, entitled “SINGLE INPUT GRAPHICAL USER INTERFACE CONTROL ELEMENT AND METHOD” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/155,007, entitled “GRAPHICAL USER INTERFACE DISPLAY EFFECTS FOR A COMPUTER DISPLAY SCREEN” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/155,009, entitled “COMPUTER ASSISTED COMPLETION OF HYPERLINK COMMAND SEGMENTS” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/155,010, entitled “HISTORICAL DATA REPLAY UTILIZING A COMPUTER SYSTEM” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/155,011, entitled “DATA STORE ACCESS PERMISSION SYSTEM WITH INTERLEAVED APPLICATION OF DEFERRED ACCESS CONTROL FILTERS” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

application Ser. No. 15/155,012, entitled “REMOTE DATA OBJECT PUBLISHING/SUBSCRIBING SYSTEM HAVING A MULTICAST KEY-VALUE PROTOCOL” and filed in the United States Patent and Trademark Office on May 14, 2016, is hereby incorporated by reference herein in its entirety as if fully set forth herein.

While the disclosed subject matter has been described in conjunction with a number of embodiments, it is evident that many alternatives, modifications and variations would be, or are, apparent to those of ordinary skill in the applicable arts. Accordingly, Applicants intend to embrace all such alternatives, modifications, equivalents and variations that are within the spirit and scope of the disclosed subject matter.

Claims

1. A computer data system having a persistent query dispatch and execution architecture, the system comprising: one or more processors each being a hardware processor;computer readable storage coupled to the one or more processors, the computer readable storage having stored thereon instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:sending an electronic request for a remote query processor from a persistent query controller to a remote query dispatcher executing on a query server computer, wherein the request includes parameters for configuring the remote query processor and an operating environment for the remote query processor, the parameters including a parameter selected from the group consisting of: a shell variable to be set on the operating environment for the remote query processor, anda classpath specifying a location of code to be used by the remote query processor;attempting, at the remote query dispatcher, to allocate an isolated operating environment for the remote query processor and to start execution of the remote query processor on the query server computer;when the remote query processor is started, performing operations including: providing the persistent query controller with an address assignment of the remote query processor or of a proxy machine in communication with the remote query processor, the address assignment identifying a specific address of the query server computer or of the proxy machine available to connect electronically over an electronic communications network;automatically connecting from the persistent query controller to the remote query processor via the electronic communications network;transmitting a persistent database query electronically from the persistent query controller to the remote query processor;publishing persistent database query configuration information including a state of the persistent database query and the address assignment of the remote query processor;connecting from a client to the remote query processor via the electronic communications network;executing the persistent database query;after the executing the persistent database query has started, receiving, from a second client different than the client, an instruction to the persistent query controller to modify code executed for the persistent database query;determining whether to allow the connection by the client to the remote query processor based on access control information specific to the persistent database query; andwhen the connection by the client is allowed:sending a request to perform an administrative operation with respect to the persistent database query from the client to the persistent query controller, the administrative operation being an instruction to modify one or more of the parameters for configuring the remote query processor and the operating environment for the remote query processor, anddetermining whether the client is authorized to perform the administrative operation based on the access control information.

2. The system of claim 1, wherein the operations further comprise: when the connection by the client is allowed: receiving at the client at least a portion of a current result of the persistent database query from the remote query processor.

3. The system of claim 1, wherein the operations further comprise: when the connection by the client is allowed: filtering, based on access control information, a current result of the persistent database query requested by the client from the remote query processor; andsending at least a portion of the filtered current result of the persistent database query to the client.

4. The system of claim 1, wherein the operations further comprise: sending, from the second client different than the client, an instruction to the persistent query controller to start, stop, restart, modify parameters, or modify code of the persistent database query.

5. The system of claim 1, wherein the operations further include: when the connection by the client is allowed: receiving a result of the persistent database query at the client;displaying at least a portion of the result at the client via a graphical user interface and/or a console;receiving at least a portion of an updated result of the persistent database query from the remote query processor; andresponsive to the receiving the at least a portion of the updated result, updating the graphical user interface and/or console to display the at least a portion of the updated result.

6. The system of claim 1, wherein the client is another remote query processor.

7. The system of claim 1, wherein the operations further include: determining whether the remote query processor rejects the request for a remote query processor from the persistent query controller;when the remote query dispatcher rejects the request, publishing an indication of the rejection;detecting, by the remote query processor or remote query dispatcher, an error in the execution of the persistent database query; andwhen the remote query processor or remote query dispatcher detects an error in the execution of the persistent database query, publishing an indication of the error.

8. The system of claim 1, wherein the operations further include: when the connection by the client is allowed: transmitting an additional query task electronically from the client to the remote query processor;executing, at the remote query processor, the additional query task; andreceiving at least a portion of a result of the additional query task at the client.

9. The system of claim 1, wherein the operations further include: periodically providing a liveness indication from the persistent query controller to the remote query dispatcher, andwhen the liveness indication is not received after a limited amount of time, stopping the remote query processor.

10. A method for improving performance of a computer data system through control of a persistent query dispatch and execution architecture, the method comprising: sending an electronic request for a remote query processor from a persistent query controller to a remote query dispatcher executing on a query server computer, wherein the request includes parameters for configuring the remote query processor and an operating environment for the remote query processor, the parameters including a parameter selected from the group consisting of: a shell variable to be set on the operating environment for the remote query processor, anda classpath specifying a location of code to be used by the remote query processor;automatically attempting, at the remote query dispatcher, to allocate an isolated operating environment for the remote query processor and to prepare the remote query processor on the query server computer;when the remote query processor is prepared, performing operations including: providing the persistent query controller with an address assignment of the remote query processor or of a proxy machine in communication with the remote query processor, the address assignment identifying a specific address of the query server computer or of the proxy machine available to connect electronically over an electronic communications network;automatically connecting from the persistent query controller to the remote query processor via the electronic communications network;transmitting a persistent database query electronically from the persistent query controller to the remote query processor;publishing persistent database query configuration information including a state of the persistent database query and the address assignment of the remote query processor;connecting from a client to the remote query processor via the electronic communications network;executing the persistent database query;after the executing the persistent database query has started, receiving, from a second client different than the client, an instruction to the persistent query controller to modify code executed for the persistent database query;determining whether to allow the connection by the client to the remote query processor based on access control information specific to the persistent database query; andwhen the connection by the client is allowed:sending a request to perform an administrative operation with respect to the persistent database query from the client to the persistent query controller, the administrative operation being an instruction to modify one or more of the parameters for configuring the remote query processor and the operating environment for the remote query processor, anddetermining whether the client is authorized to perform the administrative operation based on the access control information.

11. The method of claim 10, further comprising: when the connection by the client is allowed: receiving at the client at least a portion of a current result of the persistent database query from the remote query processor.

12. The method of claim 10, wherein the operations further comprise: when the connection by the client is allowed: filtering, based on access control information, a current result of the persistent database query requested by the client from the remote query processor; andsending at least a portion of the filtered current result of the persistent database query to the client.

13. The method of claim 10, further comprising: sending, from the second client different than the client, an instruction to the persistent query controller to start, stop, or restart the persistent database query.

14. The method of claim 10, further comprising: when the connection by the client is allowed: receiving a result of the persistent database query at the client;displaying at least a portion of the result at the client via a graphical user interface and/or a console;receiving at least a portion of an updated result of the persistent database query from the remote query processor; andresponsive to the receiving the at least a portion of the updated result, updating the graphical user interface and/or console to display the at least a portion of the updated result.

15. The method of claim 10, wherein the client is another remote query processor.

16. The method of claim 10, further comprising: determining whether the remote query processor rejects the request for a remote query processor from the persistent query controller;when the remote query dispatcher rejects the request, publishing an indication of the rejection;detecting, by the remote query processor or remote query dispatcher, an error in the execution of the persistent database query; andwhen the remote query processor or remote query dispatcher detects an error in the execution of the persistent database query, publishing an indication of the error.

17. The method of claim 10, further comprising: when the connection by the client is allowed: transmitting an additional query task electronically from the client to the remote query processor;executing, at the remote query processor, the additional query task; andreceiving at least a portion of a result of the additional query task at the client.

18. The method of claim 10, further comprising: periodically providing a liveness indication from the persistent query controller to the remote query dispatcher, andwhen the liveness indication is not received after a limited amount of time, stopping the remote query processor.

19. A nontransitory computer readable medium having stored thereon software instructions that, when executed by one or more processors, cause the processors to perform operations including: sending an electronic request for a remote query processor from a persistent query controller to a remote query dispatcher executing on a query server computer, wherein the request includes parameters for configuring the remote query processor and an operating environment for the remote query processor, the parameters including a parameter selected from the group consisting of: a shell variable to be set on the operating environment for the remote query processor, anda classpath specifying a location of code to be used by the remote query processor;automatically attempting, at the remote query dispatcher, to allocate an isolated operating environment for the remote query processor and to run-o€ the remote query processor on the query server computer;when the remote query processor is running, performing operations including: providing the persistent query controller with an address assignment of the remote query processor or of a proxy machine in communication with the remote query processor, the address assignment identifying a specific address of the query server computer or of the proxy machine available to connect electronically over an electronic communications network;automatically connecting from the persistent query controller to the remote query processor via the electronic communications network;transmitting a persistent database query electronically from the persistent query controller to the remote query processor;publishing persistent database query configuration information including a state of the persistent database query and the address assignment of the remote query processor;connecting from a client to the remote query processor via the electronic communications network;executing the persistent database query;after the executing the persistent database query has started, receiving, from a second client different than the client, an instruction to the persistent query controller to modify code executed for the persistent database query;determining whether to allow the connection by the client to the remote query processor based on access control information specific to the persistent database query; andwhen the connection by the client is allowed:sending a request to perform an administrative operation with respect to the persistent database query from the client to the persistent query controller, the administrative operation being an instruction to modify one or more of the parameters for configuring the remote query processor and the operating environment for the remote query processor, anddetermining whether the client is authorized to perform the administrative operation based on the access control information.

20. The computer readable medium of claim 19, wherein the operations further comprise: when the connection by the client is allowed: receiving at the client at least a portion of a current result of the persistent database query from the remote query processor.

21. The computer readable medium of claim 19, wherein the operations further comprise: when the connection by the client is allowed: filtering, based on access control information, a current result of the persistent database query requested by the client from the remote query processor; andsending at least a portion of the filtered current result of the persistent database query to the client.

22. The computer readable medium of claim 19, wherein the operations further comprise: sending, from the second client different than the client, an instruction to the persistent query controller to start, stop, or restart the persistent database query.

23. The computer readable medium of claim 19, wherein the operations further include: when the connection by the client is allowed: receiving a result of the persistent database query at the client;displaying at least a portion of the result at the client via a graphical user interface and/or a console;receiving at least a portion of an updated result of the persistent database query from the remote query processor; andresponsive to the receiving the at least a portion of the updated result, updating the graphical user interface and/or console to display the at least a portion of the updated result.

24. The computer readable medium of claim 19, wherein the client is another remote query processor.

25. The computer readable medium of claim 19, wherein the operations further include: determining whether the remote query processor rejects the request for a remote query processor from the persistent query controller;when the remote query dispatcher rejects the request, publishing an indication of the rejection;detecting, by the remote query processor or remote query dispatcher, an error in the execution of the persistent database query; andwhen the remote query processor or remote query dispatcher detects an error in the execution of the persistent database query, publishing an indication of the error.

26. The computer readable medium of claim 19, wherein the operations further include: determining whether the attempting to start a remote query processor was successful; andwhen the attempting was not successful, repeating the attempting a limited number of times.

27. The computer readable medium of claim 19, wherein the operations further include: periodically providing a liveness indication from the persistent query controller to the remote query dispatcher, andwhen the liveness indication is not received after a limited amount of time, stopping the remote query processor.