Patent Application
20180232406
The present invention relates to the field of databases, particularly to a big data database system created in a relational database of a structural query language (SQL) technology and a non-relational database (NoSQL database) technology, and an intermediate device is provided to organize and assign a connection task to reduce the connection load of a terminal host to improve the overall database operation performance.
In 2012, an article “The Age of Big Data” published in The New York Times officially declared the advent of the era of big data, and the term “Big Data” can be said to be familiar to everyone thereafter. When the volume of accumulated data becomes increasingly larger, databases are facing challenges of high read/write requirement, high storage efficiency, high access and high scalability. For example, a general notification relational database adopts synchronous ad symmetric connection structure to access data through a number of connections equal to the number of clients in order to achieve a powerful data query capability. However, such server-client distributed database is limited by the load capability of hardware equipments since the number of connections has to be equal to the number of clients. When there is a need of accessing big data, the original hardware equipments must be expanded, upgraded or replaced with a higher specification, and thus incurring a higher cost as well as a higher level of difficulty for the operation.
Although the non-relational database using distributed storage and computation to process data has the advantages of a low hardware equipment cost and a high expandability, yet the non-relational database also adopt the synchronous and symmetric connection structure. As a result, it is also necessary to expand or upgrade the original hardware equipment to access big data in the non-relational database. Since the non-relational database adopts a new programming language, therefore it has a very weak query capability, and also has a high error rate while writing and outputting a large quantity of data, and the programming language is still not well developed to maintain a low version upgrade risk, and thus reducing the safety and stability of the database, which is not conducive to the industrial development with a high risk requirement.
To take the real-time, accuracy and safety of the relational database and the high scalability and scalability of the non-relational database into consideration, a SQL+NoSQL database 1 as shown in
In view of the drawbacks of the prior art, it is an important subject for the present invention to improve the conventional database structure or provide a novel database structure in order to reduce the number of connections or load of the database, while ensuring the work efficiency of converting data into information.
Therefore, it is a primary objective of the present invention to overcome the drawbacks of the prior art by providing a big data database system with a high scalability to reduce the network load of a data host while maintaining a high capability of processing the data requests for the network connections of at least 400K clients, so as to achieve a high-quality operation of big data.
To achieve the aforementioned objective, the present invention discloses a big data database system created in a language structure of a SQL database and a language structure of a NoSQL database and provided for accessing data after a plurality of clients have requested for a network connection, characterized in that the big data database system comprises at least one transfer server and a data host, and the transfer server includes a plurality of queuing devices and an assigner, and the queuing devices are respectively and telecommunicatively coupled to the clients, and the assigner is telecommunicatively coupled to the queuing devices and the data host, so that when one of the clients sends out a data request, the respective queuing device receives and queues the data request, and the assigner sends the data request to the data host through a single connection between the transfer server and the data host.
Wherein, the assigner sends the data requests to the data host sequentially according to a first-in-first-out message mechanism after reading the data requests from the queuing devices, and the queuing devices are installed to be corresponsive to the clients respectively, so that each client is connected to the respective queuing device, and when the transfer server come with a plural quantity, and if the quantity of transfer servers is equal to a, then the quantity of network connections between the data host and the transfer servers will also be equal to a.
In addition, the transfer server and the clients adopt a language technology of a relational database structure, and the transfer server and the data host adopts a language technology of a non-relational database structure.
In summation, the present invention adopts the assigner to do the translation work of the new programming language to overcome the issue of a low data query performance of the conventional NoSQL database and also adopts the sequence of using the queuing devices and the assigner to process the data requests, so as to overcome the issue of having equal number of clients and network connections as required in the conventional SQL database, NoSQL database, or SQL+NoSQL database. In other words, the present invention no longer requires distributing a corresponding thread for each data request for data processing and responses. Even if the quantity of clients increases, or the data requests increases drastically, the data host still can process data or respond by a single connection with the transfer server through the coordination of the assigner, so that the overload of network connections will not occur anymore. Further, the big data database system of the present invention is used for the exchange of messages and implemented with different data processing methods.
The above and other objects, features and advantages of this disclosure will become apparent from the following detailed description taken with the accompanying drawings.
With reference to
S1: if the quantity of clients 20 is equal to N, the same quantity of queuing devices 210 will be installed, so that the transfer server 21 has N queuing devices 210, and each queuing device 210 is connected to each respective client 20. In other words, there are N network connections between the transfer server(s) 21 and the clients 20.
S2: When any one of the clients 20 sends out a data request, the respective queuing device 210 receives and queues the data requests. In other words, an instruction, transaction or request for use transmitted from each client 20 is written into each queuing device 210 by using a SQL language. In the meantime, each queuing device 210 sends a request notice to the assigner 211, and a large quantity of data requests are stored in the transfer server 21, so that the assigner 211 creates a thread for executing the data requests by stages.
S3: The assigner 211 reads the data requests from the queuing devices 210 according to a first-in-first-out message mechanism and translates the data request into a programming language recognizable by the data host 22, and the single connection between the transfer server 21 and the data host 22 sequentially sends the data requests to the data host 22, so that the thread of each respective queuing device 210 coordinated by such message mechanism have different nodes for work, so as to allow the data host 22 to receive and recover a large quantity of requests by using a single connection, so as to avoid repeatedly showing the data requests inputted by the clients 20 or affecting the system operation performance. It is noteworthy that the assigner 211 is used as an intermediate device between the transfer server 21 and the data host 22 for translating the programming language for the data processing, so as to improve the query capability of the big data database system 2 significantly, and the quantity of connections between the clients 20 and the transfer servers 21 is obviously not equal to the quantity of connections between the transfer servers 21 and the data host 22, so as to reduce the network load of the data host 22 significantly.
In addition, when the quantity of clients 20 has reached a predetermined number such that the transfer server 21 has reach an upper limit of network connections, another transfer server 21 may be added to the big data database system 2 for automatically receiving the overflow data requests of the clients 20, and a single connection between such transfer server 21 and the data host 22 is also used to sequentially send the data requests to the data host 22. In
In
