Patent Application
20210051211
This application claims priority of Chinese patent application No. 201810082188.7 filed Jan. 29, 2018, and entitled “MIRROR PULL METHOD AND SYSTEM THEREFOR”, the entire contents of which are incorporated herein by reference.
The present invention relates to the field of containerized platforms, and in particular, to a method and system for image pulling.
Container virtualization technology has become a widely accepted way of sharing server resources. It can provide system administrators with great flexibility in the process of building operating system instances on demand. The containers are completely using the sandbox mechanism, and there will not be any interface between each other.
Docker is a container engine for open source applications. It also allows developers to package their applications and dependent packages into a portable container and then publish it to Linux machines. It can also be virtualized. The emergence of container technology makes distributed systems that were originally deployed based on virtual machine nodes also gradually carry out containerized operations. For example, Google's kubernetes, swarm, etc. have all implemented containerized deployment, and the use of container technology is increasingly widespread.
When running a containerized application, the usual practice is to pull a mirror image from a repository (Registry), for example, to download the image from a public repository “docker hub”. For an internal system of an enterprise, because customized images are often used, from the perspective of security and efficiency, a private repository service will be used to distribute the image.
When deploying a larger container application cluster, hundreds of nodes are used (for example, a kubernetes cluster system claims to support 2000 nodes). The main time of the deployment process is spent in the distribution of the image. Therefore, when a container application cluster is deployed in a large number of nodes for the first time and when images are updated, a processing bottleneck of the registry of the private repository will be incurred. This includes the processing bottleneck of the server where the private repository locates, and the network bandwidth bottleneck of the server. In the case of limited network bandwidth and processing capacity, because the Registry faces a large number of concurrent operations, a queue will inevitably appear for waiting of image pulling, so that the efficiency is relatively slow.
At present, there is a technology that combines the Bit Torrent protocol to speed up the distribution of images. The main principle is to share the image of each node through the Bit Torrent protocol to achieve P2P acceleration. Due to the principle of P2P technology, this technology needs to divide and compress an image file, and transmit and accelerate in all the nodes in the cluster, which will consume the network bandwidth resources to a certain extent. In addition, because the image file needs to be divided, and the synchronization of each divided file needs to be maintained, the complexity of the system is also increased.
In view of this, the object of the present invention is to provide a method and system for image pulling to improve the efficiency of image pulling.
In order to deal with the above technical problem, the technical schemes adopted by the present invention are as follows:
In accordance with an aspect of the present invention, there is provided a method for image pulling, which is applied to a server, including:
receiving an image pull request from a requesting client;
judging whether a current load state of the server is a high-load running state;
determining, in response to a judgment result being yes, a first client, wherein the first client is configured with an image corresponding to the image pull request:
sending a redirection pull image message to the requesting client, wherein the redirection pull image message is used for the requesting client to pull the image from the first client.
In a possible design, after receiving at least one image pull request from a requesting client, the method includes:
judging whether the number of received image pull requests exceeds a preset threshold; and
selecting one or more of the image pull requests for priority processing according to a preset priority pulling principle in response to a judgment result being yes.
In a possible design, the selecting multiple image pull requests for priority processing according to a preset priority pulling principle includes:
determining a subnet to which the requesting client sending the image pull request belongs; and
selecting the image pull request in different belonged subnets.
In a possible design, the selecting the image pull request in different belonged subnets includes:
selecting at least one of the image pull requests in each different belonged subnet in response to the number of the belonged subnets being less than or equal to a preset threshold:
preferentially selecting an image pull request from the belonged subnets that have a larger number of online clients in response to the number of the belonged subnets being greater than the preset threshold.
In a possible design, the determining the first client includes:
judging whether an online client exists in the subnet where the requesting client locates, wherein the online client is configured with an image corresponding to the image pull request;
determining that the online client is the first client in response to a judgment result being yes;
determining the clients in other subnets that have the image corresponding to the image pull request in response to the judgment result being no, and selecting at least one of the clients closest to the server as the first client.
In a possible design, before sending the redirection pull image message to the requesting client, the method includes:
judging whether a distance between the requesting client and the first client is greater than a predetermined value; and
in response to being yes, waiting for the server to establish an image transmission.
In a possible design, the receiving an image pull request from a requesting client includes:
obtaining IP information of the requesting client to judge whether the network exit of the requesting client has been mapped by NAT.
According to the present invention, there is further provided a system for image pulling, which is applied to a server, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program, when executed by the processor, performs the steps of the method for image pulling provided by an embodiment of the present invention.
According to the present invention, there is further provided a computer-readable storage medium storing an image pulling program which, when executed by a processor, cause the processor to perform the steps of the method for image pulling provided by an embodiment of the present invention.
According to the present invention, there is further provided a system for image pulling, including: a server and at least one client; wherein the server is configured to send a redirection pull image message to the client or establish an image transmission with the client according to the image pull request sent by the client; wherein the redirection pull image message is used for the requesting client to pull an image from a first client, and the first client is configured with the image corresponding to the image pull request.
The method and system for image pulling provided by the embodiments of the present invention utilize the network bandwidth of each subnet to realize sharing and transmission of images in each subnet. The transmission protocol used is simple, requiring only a point-to-point transmission protocol to solve the problem of the bottleneck of the network and processing capacity of the service of the private repository (Registry). The invention can also optimize the priority order of pushed images according to the network conditions of a specific cluster, and maximize the use of the switch network to accelerate the pulling of the cluster images.
The drawings are provided for a further understanding of the technical schemes of the present application, and constitute a part of the description. The drawings and the embodiments of the present application are used to explain the technical schemes of the present application, and not intended to limit the technical schemes of the present application.
The implementation, functional characteristics and advantages of the present invention will be further described in conjunction with the embodiments and with reference to the drawings.
In order to make the technical problems to be solved, technical schemes and beneficial effects of the present invention clearer, the present invention will be described below in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not intended to limit the present invention.
In the present invention, both the server and the client provide repository services, so that container engines (such as Docker) can pull images from the repository of the server or the local client. The client itself is also an image, which needs to be pulled from the server first and run the client container.
As shown in
At 101, starting.
After a client runs, it will connect to the server (which can be understood as a login operation) and send basic information of a local node, such as local IP. If the connection is successful, a heartbeat connection is maintained between them. Similar to the heartbeat mechanism of the client and the server of communication software, it only takes up a small amount of resources. As long as the heartbeat continues, the server considers these clients to be online and to be online clients.
At 102, receiving at least one image pull request from a requesting client.
When an online client sends an image pull request, it is a requesting client.
At 103, judging whether a current load state of the server is a high-load running state, and if yes, going to step 104; if no, going to step 105.
At 104, determining a first client and sending a redirection pull image message to the requesting client.
The first client is configured with an image corresponding to the image pull request. The redirection pull image message is used for the requesting client to pull the image from the first client.
At 105, directly transmitting the image corresponding to the image pull request to the requesting client.
At this time, if the server is in a high-load running state, queuing is performed.
At 106, ending.
In this embodiment, after the server completes a push task, it will record which images the client currently stores, and maintain synchronization through a regular synchronization mechanism. When a new image pull request arrives, if the server is in a high load stage, it will search for the client which has the image among the current online clients, and send a message to the requesting client letting the requesting client to pull the image from the first client having the image.
As shown in
At 201, starting.
After a client runs, it will connect to the server (which can be understood as a login operation) and send basic information of the local node, such as local IP. If the connection is successful, a heartbeat connection is maintained between them. Similar to the heartbeat mechanism of the client and the server of communication software, it only takes up a small amount of resources. As long as the heartbeat continues, the server considers these clients to be online and to be online clients.
At 202, receiving at least one image pull request from a requesting client.
When an online client sends an image pull request, it is the requesting client.
At 203, judging whether a current load state of the server is a high-load running state, and if yes, going to step 204; if no, going to step 206.
At 204. Searching whether there is a first client which is configured with an image corresponding to the image pull request; if yes, going to step 205; if no, going to step 206.
At 205, sending a redirection pull image message to the requesting client, so that the requesting client can pull the image from the first client.
At 206, judging whether the number of received image pull requests exceeds a preset threshold; if yes, going to step 207; if no, going to step 208.
At 207, selecting one or more of the image pull requests for priority processing according to a preset priority pulling principle.
The preset priority pulling principle gives priority to covering all subnets. In this embodiment, the server maintains a table of client network information based on the client IP information that has been acquired, and classifies the node where the client locates according to the subnet and a routing path.
At 208, directly transmitting the image corresponding to the image pull request to the requesting client.
At 209, ending.
In this embodiment, when the server faces a large number of concurrent image pulling operations, the preset priority pulling principle is used to adjust a sending weight, and the request that needs to be processed preferentially is selected. For the remaining requests, if the server is in a heavy load period, it will ask whether the online client has the same image. If so, the server will send a resource redirection message to the requesting client to let the client to pull the image from another node where the image exists. The server will preferentially provide hosts that are on the same subnet as or close to the client that requests the image, usually a list of hosts, for the client to try to connect for multiple times.
In this way, due to the full use of the switch network bandwidth of each subnet, on the one hand, the pressure on the main repository can be greatly reduced, and on the one hand, the distribution speed of the image is accelerated. Theoretically, it can be seen that the speed of image distribution can be doubled over time, and the traffic of image file transmission will gradually be controlled in the switch network of the subnet.
Based on the embodiment corresponding to
Determining a subnet to which the requesting client sending the image pull request belongs; and
Selecting the image pull request in different belonged subnets.
For example, there are 300 nodes in a government-enterprise network, and a large container application cluster needs to be deployed, and there is a need for subsequent updates and patches. For container application clusters, deploying, updating, and patching means replacing the boot image.
When adopting the method for accelerating the large-scale image pulling provided by the present invention, a main repository server is required first, and the server provides a private repository service for all nodes to pull the corresponding client image for the first time and run it.
The client image itself also provides repository services. When all 300 nodes run the client image, the client will send network information of the node to the server, including the network card IP address, mask and gateway for external communication. And the clients maintain a heartbeat connection with the server, as long as the heartbeat continues, the server considers these clients to be online. After the server receives the information from the client, it will maintain the information of the client, and group the client according to the subnet, based on the client's IP address source, combined with the server's routing path to each client. It is assumed here that these 300 nodes belong to 3 subnets: Subnet A, Subnet B, and Subnet C.
After the 300 nodes have all run the client, in order to deploy or connect with a new application cluster, an operation and maintenance tool can be used to perform a batch of image pulling operations on the 300 nodes. Each node will first access the local client's repository service. If the local repository has the image, the image will be pulled directly. If not, a request to pull the image is made to the server. In the case of the first deployment, each node does not have the image. At this time, the client needs to pull the image from the server. For the server, in a time period, there may be a plurality of requests to pull the image, and then the server will select a plurality of image pull requests according to a preset priority pulling principle request to preferentially process the requests of each node.
In this embodiment, the preset priority pulling principle includes: determining the priority principle of different subnets according to the number of the subnets. Assuming that there are requests from nodes x1, x2, and x3 at the same time, and after considering the current bandwidth usage and efficiency, the server can only handle 2 requests first. According to the information of x1, x2 and x3, it is known that x1 and x2 belong to the same subnet A, then the server will first satisfy the pull request of x1 and x3 or x2 and x3, that is, the request from the nodes of different subnets, trying to cover all subnets first.
During specific implementation, the selecting image pull requests in different subnets includes:
If the number of the belonged subnets is less than or equal to the preset threshold, selecting at least one of the image pull requests in each different belonged subnet:
If the number of the belonged subnets is greater than the preset threshold, an image pull request is preferentially selected from the belonged subnets that have a larger number of online clients. In general, if the threshold is n, an image pull request is preferentially selected from each of the n subnets that have a larger number of online clients.
That is, in this embodiment, the preset priority pulling principle further includes a principle of large subnet priority determined according to the number of online clients owned by the subnet. At this time, if the nodes x1, x2, and x3 belong to three different subnets A, B, and C, but according to the current subnet information of the clients, it is known that 100 clients of the subnet where x1 locates remain online, and the subnet where x2 locates has 80 online clients, and the subnet where x3 is locates has 60 online clients. Then, according to the principle of large subnet priority, requests from x1 and x2 are preferentially satisfied.
More specifically, it is assumed that the government-enterprise network is composed of three subnets, subnet A, subnet B, and subnet C. There is no NAT mapping. The relationship between its nodes, that is the client IPs, and subnets is as follows:
Subnet A: 10.113.4.2˜10.113.4.241, a total of 240 nodes are online and connected to the server
Subnet B: 10.113.3.2˜10.113.3.201, a total of 200 nodes are online
Subnet C: 10.115.2.2˜10.115.2.101, a total of 100 nodes are online
Using operation and maintenance tools, these nodes may send image pull requests to the server at the same time or within a certain period of time. According to the principle of different subnet priority and the principle of large subnet priority, if, within a short period of time (assumed to be within 3 seconds), 3 subnets each have 2 requests (6 requests in total) arriving at the server in succession, the server (assuming that the server can only satisfy n concurrent requests at a time) will first satisfy a request of subnet A (assuming the request from 10.113.4.2 is satisfied), subnet B (assuming the request from 10.113.3.2 is satisfied) and subnet C (assuming the request from 10.115.2.2 is satisfied) respectively (the principle of diffident subnet priority, that is, the connection capacity should cover as many different subnets as possible).
For example, assuming that n is 4, at this time, since the number of subnets is 3, which is less than n, the server will first satisfy a request from the node of subnet A, a request from the node of subnet B. and a request from the node of subnet C, respectively. The remaining connection capacity will be given to subnet A. This is based on the principle of large subnet. Due to the large number of online clients on subnet A, the remaining connections are temporarily queued. In other words, two requests from subnet A will be processed first, and one request from each of subnets B and C will be processed first, that is, at least one request for each subnet will be selected.
If the conditions of multiple requests meet the above requirements, for example, there are the same number of online clients in subnets A, B and C, then at this time, the remaining one connection capacity may be allocated according to the arrival time of the request, that is, the preset priority principle of pulling can also include: the time priority principle determined according to time sequence.
For another example, assuming that n is 2, at this time, since the number of subnets is 3, which is greater than n, a request from the node of subnet A and a request from the node of subnet B are first satisfied respectively. This is because subnet A and subnet B has more online clients than subnet C.
In the conditions of multiple requests meet the above requirements, for example, there are the same number of online clients in subnets A, B and C, then at this time, the two connection capacities may be allocated according to the arrival time of the request, that is, the preset priority principle of pulling can also include: the time priority principle determined according to time sequence.
Based on any one of the foregoing embodiments, according to an embodiment of the present invention, there is provided a method for image pulling, as shown in
At 301, starting.
At 302, judging whether an online client exists in the subnet where the requesting client locates; if yes, going to step 303; if no, going to step 304.
The online client is configured with an image corresponding to the image pull request.
At 303, determining that the online client is the first client; and going to step 305.
At 304, determining the clients in other subnets that have the image corresponding to the image pull request, and selecting at least one client closest to the server as the first client:
At 305, ending.
It can be seen that in this embodiment, a node on the same subnet as the requesting node is preferentially searched to query whether the online client of the subnet also has the same image. If yes, a redirection pull image message is sent to the requesting client to let the requesting client to pull the image from the first client with the same image in its subnet. There may be multiple first clients.
For example, as mentioned earlier, if the nodes x1, x2, and x3 belong to three different subnets A, B, and C, after the first transmission, x1, x2, and x3 all have a certain image X, and now there is a new request for pulling the image X from x4, x5, and x6, where x4, x5 belong to subnet A, and x6 belongs to subnet C. Assuming that the server is only able to satisfy the request of x4 due to the heavy load at this time, and then for x5 and x6, by querying the client information, it is learned that x1 and x3 also have the image X, and a message is sent to x1 and x3 for confirmation, and because that x1 and x5 belong to subnet A, and x3 and x6 belong to subnet C, the server will send a redirection pull image message to x5 and x6 to let x5 pull the image X from x1 and x6 pull the image from x3, so that the traffic for x5 and x6 to pull the image is limited to subnets A and C. At the same time, using the switch network of the subnet, the speed has been greatly improved. When x5 and x6 have finished pulling the image, they will send feedback to the server to inform that they have finished pulling the image X.
The image pull requests from x4, x5, and x6 are the same as above, but the server queries out that the subnet A to which x5 belongs does not have the image X, but it is known by querying the online client that a node x7 of subnet B and a node x8 of the subnet C have the image X. Then at this time, the server preferentially selects the node of the subnet closer to the subnet A to which x5 belongs, and let x5 to redirect the pull request.
For example, if the routing path of the server to the node of subnet A is r1, r2, r3, the routing path to subnet B is r1, r2, r4, and the routing path to the node of subnet C is r1, r2, r5, r6, then it can be seen that subnet B and subnet A is more closer, especially in the case of government-enterprise network leased lines and network conditions, the routing path selection is basically determined. In this scenario, it can be considered that the number of routing hops for traffic transmission between the nodes of subnet B and subnet A is less than that between the nodes of the subnet C and subnet A. In this way, in the above scenario, the server will give priority to x5 to pull the image from the node x7 of subnet B.
A distance between the subnets can be determined by the following formula:
d(Sri,Srj)=card(Sri∩Srj)−card(Sri∩Srj) (1)
Where, Sri, Srj are an ordered set of routing paths of subnets i and j respectively. Taking the above case as an example, the set of routing paths from the server to subnet A is {r1, r2, r3}, and the set of routing paths from the server to subnet B is {r1, r2, r4}, and the set of routing paths from the server to subnet C is {r1, r2, r5, r6}, then according to formula (1), the routing distance between subnets A and B is:
The distance between subnets A and C is
Therefore, the distance between subnet B and subnet A is closer.
On the basis of any of the foregoing embodiments, before the sending the redirection pull image message to the requesting client, the method further includes:
Judging whether the distance between the requesting client and the first client is greater than a predetermined value:
If yes, waiting for the server to establish an image transmission.
Generally, if the requesting client and the first client belong to the same subnet, the distance between the two will definitely not be greater than the predetermined value. If they belong to different subnets, when the distance between the two is greater than the predetermined value, even if a node in the subnet has the image required by the requesting node, the server will consider that it is not suitable for file transmission between the two subnets because the efficiency is not high. At this time, the node needs to wait for the server to establish the image transmission.
The following is supplemented by examples to further illustrate this embodiment:
When all the first batch of requests are processed and there are more requests that exceed the concurrent processing capacity of the server, the server will perform a redirection pull request. Assuming that there is a node 10.113.4.5 from subnet A requesting an image pulling at this time, and the server does not have excess connection capabilities, then the server will try to analyze the existing online client through a client management analysis module. The server first inquires whether there is the image on the online client of the node in the same subnet (subnet A) as the node 10.113.4.5. If it is found that the node 10.113.4.2 in subnet A also has the image corresponding to the request, a resource redirection message is sent to let the client of 10.113.4.5 request to pull the image from the client of the 10.113.4.2.
If the result of querying clients of subnet A is none, then the server turns to a neighboring subnet to query.
If the routing path of subnet A is:
={10.113.23.169, 10.114.51.121, 10.114.4.128}
The routing path of subnet B is:
={10.113.23.169, 10.114.51.121, 10.114.3.128}
The routing path of subnet C is:
={10.113.23.169, 10.114.51.121, 10.114.55.1, 10.114.5.128}
According to formula (1), it can be known that subnet B is closer to subnet A. Therefore, if no node with the image is found in subnet A, the server will search whether the client in the neighboring subnet B has the image. And so on, if there is none in subnet B, the server will turn to search for the nodes in subnet C.
When the distance between the neighboring subnet and the subnet where the requesting node locates is large, even if a node in the neighboring subnet has the image, this cross-subnet transmission will not improve the efficiency because the transmission crosses many routing points. Therefore, in this case, the server may not redirect the pull request, but will provide the image transmission.
Based on any one of the foregoing embodiments, the receiving an image pull request from a requesting client includes:
Obtaining IP information of the requesting client to judge whether a network exit of the requesting client has been mapped by NAT.
Considering that some subnet environments will do NAT mapping, so from the perspective of the server, the source IP of these clients are the same, but the ports are different, so this is why the client needs to send the client's local IP information to the server. In this case, as long as there are not many NAT mapping layers, the image distribution acceleration function can still be achieved.
For example, a host in a subnet uses C-type web addresses (IPs beginning with 192) for communication, and accesses hosts outside the subnet by NAT mapping through routing. The mapped values are all 10.116.53.22. Then, for the server, the source of the request IP it sees is 10.116.53.22, which is inconsistent with the IP information 192.xxx passed by the client, so it can be considered that the network exit of the client is mapped by NAT. At this time, the processing method of the server is the same as before. If a client with a local IP of 192.168.2.2 and a source IP of 10.116.53.22 requests to pull the image X, the server searches and finds that an online client under the same routing path with a source IP of 10.116.53.22 and a local IP of 192.168.2.3 also has the image X, then the server will send a “redirection pull image message” to 192.168.2.2, and let it pull the image X from 192.168.2.3. By repeating the above steps, the bandwidth of the local network within each subnet is utilized to the greatest extent, which greatly accelerates the speed of image distribution of the entire cluster node.
More specifically, if there is now a subnet D, as follows
Subnet D: 60 nodes from 192.168.2.2 to 192.168.2.61 are online
Subnet D uses Class C addresses internally and performs NAT mapping to the outside. Assume that the subnet D routing path is:
={10.113.23.169, 10.114.51.121, 10.114.6.128}
From the perspective of the server, the IPs of the nodes in subnet D are all 10.114.6.128, but because the client will also provide local IP information, the information seen by the server is as follows:
Client 1: Source IP: 10.114.51.121, Reported IP: 192.168.2.2
Client 2: Source IP: 10.114.51.121, Reported IP: 192.168.2.3
Client 3: Source IP: 10.114.51.121, Reported IP: 192.168.2.4
. . .
Client 60: Source IP: 10.114.51.121, Reported IP: 192.168.2.61
In this scenario, if the node of 192.168.2.5 of subnet D requests to download an image, since its IP is displayed as 10.114.51.121 and the routing path is 110.113.23.169, 10.114.51.121, 10.114.6.1281, the server will query the client of subnet D for the existence of the image. If it is found that the node of 192.168.2.2 has the image, then the server will let the node of 192.168.2.5 pull the image from 192.168.2.2, so as to achieve the purpose of network acceleration by using the switch network.
After the image transmission is completed, the server will also check the integrity of the image. At the same time, if the client tries to pull the image from another client and fails, a feedback will be given to the server, and the server will redirect to other clients to download or the server itself provides image transmission.
According to the present invention, there is further provided a system for image pulling, which is applied to a server and includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program, when executed by the processor, performs the steps of the method for image pulling provided by any embodiment of the present invention.
It should be noted that the above system embodiment of image pulling and the method embodiment belong to the same concept, and the specific implementation process is described in detail in the method embodiments, and the technical features in the method embodiments are correspondingly applicable in the system embodiments of image pulling, which will not be repeated here.
In addition, according to an embodiment of the present invention, there is further provided a computer-readable storage medium storing an image pulling program, which, when executed by a processor, cause the processor to perform the steps of the method for the image pulling provided by the embodiment of the present invention.
It should be noted that the above-described image pull program embodiment on the computer-readable storage medium and the method embodiment belong to the same concept, and the specific implementation process is described in detail in the method embodiments, and the technical features in the method embodiments are correspondingly applicable in the embodiments of computer-readable storage medium, which will not be repeated here.
As shown in
According to the present invention, there is provided a method and system for image pulling, especially for a cluster composed of multiple subnet nodes in a government-enterprise network. The invention makes full use of the switch network inside the subnet, does not occupy the bandwidth of the main road, greatly reduces the pressure on the server of the main repository, and avoids the congestion of the main road bandwidth.
The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, which does not limit the scope of rights of the present invention. Those skilled in the art can make many variations to realize the present invention without deviating from the scope and essence of the present invention. For example, the features of one embodiment can be used in another embodiment to obtain another embodiment. Any modification, equivalent replacement and improvement made within the technical concept of using the present invention shall fall within the scope of the rights of the present invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 201810082188.7 | Jan 2018 | CN | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/CN2019/073581 | 1/29/2019 | WO | 00 |
