Patent Application
20250209070
This application claims priority to Chinese patent application No. 202311766870.5, filed on Dec. 20, 2023 and entitled “METHOD AND APPARATUS FOR PROCESSING LOG REQUEST, DEVICE, AND STORAGE MEDIUM”. The content of the above application is hereby incorporated by reference in its entirety.
Embodiments of the present disclosure relate to the field of cloud service technologies, and in particular, to a method and apparatus for processing a log request, a device, and a storage medium.
With the development of cloud service technologies, a plurality of microservices may be deployed on a cloud service system, and various types of network businesses are processed through the microservices. When a corresponding business is processed through a microservice, log information corresponding to the microservice may be generated, and a problem existing in the cloud service system may be queried through the log information.
In the related art, the log information corresponding to the microservice may be stored by a log storage system. When the cloud service system processes the corresponding business through the microservice, the cloud service system generates the log information corresponding to the microservice, and sends a write request for the log information to the log storage system. The log storage system stores the log information corresponding to the microservice, and establishes a corresponding index relationship, so that querying of the log information corresponding to the microservice is facilitated.
However, the inventors have found that the prior art has at least the following technical problems: when there are a large quantity of microservices, the quantity of log information generated by the microservices is also large, resulting in low efficiency of the log storage system in storing the log information.
Embodiments of the present disclosure provide a method and apparatus for processing a log request, a device, and a storage medium, which can improve efficiency of a log storage system in storing log information.
In a first aspect, an embodiment of the present disclosure provides a method for processing a log request, applied to a log storage system, where the log storage system includes a request processing unit and a log storage unit, where the request processing unit includes a plurality of log servers; and the method includes:
In a second aspect, an embodiment of the present disclosure provides an apparatus for processing a log request, applied to a log storage system, where the log storage system includes a request processing unit and a log storage unit, where the request processing unit includes a plurality of log servers; and the apparatus includes:
In a third aspect, an embodiment of the present disclosure provides an electronic device, including:
In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, where the computer-readable storage medium stores computer-execution instructions, and when a processor executes the computer-execution instructions, the method for processing a log request according to the first aspect is implemented.
In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method for processing a log request according to the first aspect is implemented.
The method and apparatus for processing a log request, the device, and the storage medium are provided in the embodiments of the present disclosure. The method includes: obtaining, for the write request for the log information of each service, the service identifier of the service and the traffic type of the service, where the traffic type is used to indicate the write traffic range of the log information of the service within the preset duration; determining, based on the service identifier and the traffic type of the service, at least one target log server that processes the write request, and determining, based on the service identifier and the traffic type of the service, the storage path information of the log information of the service on the log storage unit; and writing, through the at least one target log server, the log information of the service into the log storage unit based on the storage path information. In the embodiments of the present application, when a write request for log information of a certain service is received, a log server that processes the write request and storage path information of the log information are determined based on the traffic size of the service. In this way, log information of different services can be stored in different locations through different log servers, that is, an architecture with computing and storage separated is adopted, and larger-scale log writing and storage are supported, so that writing and querying of the log information are facilitated. Therefore, efficiency of storing the log information is improved, and management of high-traffic log information is realized.
In order to more clearly describe the technical solutions in the embodiments of the present disclosure or in the prior art, the following briefly introduces the accompanying drawings required for describing the embodiments or the prior art. Apparently, the accompanying drawings in the following description show some embodiments of the present disclosure, and a person of ordinary skill in the art may still derive other drawings from these accompanying drawings without creative efforts.
In order to make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the following clearly and completely describes the technical solutions in the embodiments of the present disclosure with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are some but not all of the embodiments of the present disclosure. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts shall fall within the protection scope of the present disclosure.
It should be noted that user information (including but not limited to user equipment information and user personal information) and data (including but not limited to data for analysis, stored data, and displayed data) involved in the present application are information and data authorized by users or fully authorized by all parties. The collection, use, and processing of related data need to comply with relevant laws, regulations, and standards, and corresponding operation entries are provided for users to choose to authorize or refuse.
With the development of cloud service technologies, a plurality of microservices may be deployed on a cloud service system, and various types of network businesses are processed through the microservices. When a corresponding business is processed through a microservice, log information corresponding to the microservice may be generated, and a problem existing in the cloud service system may be queried through the log information.
In the related art, the log information corresponding to the microservice may be stored by a log storage system. When the cloud service system processes the corresponding business through the microservice, the cloud service system generates the log information corresponding to the microservice, and sends a write request for the log information to the log storage system. The log storage system stores the log information corresponding to the microservice, and establishes a corresponding index relationship, so that querying of the log information corresponding to the microservice is facilitated. However, when there are a large quantity of microservices, the quantity of log information generated by the microservices is also large, resulting in low efficiency of the log storage system in storing the log information.
It can be seen that how to improve the efficiency of the log storage system in storing the log information is a technical problem to be solved urgently at present.
To solve the above problems, the following technical concept is provided in the embodiments: from a dimension of log information of each service, a log server that processes the service and storage path information of log information are determined based on the traffic size of the service. In this way, high-traffic services and low-traffic services are processed through different log servers, so that efficiency of writing and querying of the log information can be improved.
Accordingly, specific steps may include: first, obtaining, for a write request for log information of each service, a service identifier of the service and a traffic type of the service, where the traffic type is used to indicate a write traffic range of the log information of the service within a preset duration; then, determining, based on the service identifier and the traffic type of the service, at least one target log server that processes the write request, and determining, based on the service identifier and the traffic type of the service, storage path information of the log information of the service on the log storage unit; and finally, writing, through the at least one target log server, the log information of the service into the log storage unit based on the storage path information.
In this case, when a write request for log information of a certain service is received, a log server that processes the write request and storage path information of the log information are determined based on the traffic size of the service. In this way, log information of different services can be stored in different locations through different log servers, that is, an architecture with computing and storage separated is adopted, and larger-scale log writing and storage are supported, so that writing and querying of the log information are facilitated. Therefore, efficiency of storing the log information is improved, and management of high-traffic log information is realized.
An application scenario of the embodiments of the present disclosure is described below.
The method for processing a log request provided in the embodiments of the present disclosure may be applied to a scenario in which a write request for log information of any microservice is processed.
S201: obtaining, for a write request for log information of each service, a service identifier of the service and a traffic type of the service, where the traffic type is used to indicate a write traffic range of the log information of the service within a preset duration.
In the embodiments of the present disclosure, the service may be an application or a microservice installed on a terminal. During running, the service generates the corresponding log information, and sends the write request for the log information to the log storage system. The write request may be sent when the log information is generated, or may be sent periodically.
The write request carries the service identifier of the service. The service identifier is used to distinguish different services. In an implementation, the service identifier is a service name.
In an implementation, the log storage system stores a correspondence between service identifiers and traffic types of services. Correspondingly, the log storage system may determine, through the obtained service identifier, the traffic type of the service corresponding to this service identifier from the stored correspondence between the service identifiers and the traffic types of services.
The traffic type of the service may be determined through a traffic range in which write traffic of the log information of the service within the preset duration is located. For example, the traffic type of the service may be determined through a traffic range in which write traffic of the log information of the service yesterday (for 24 hours) is located.
In an implementation, the traffic type includes a first traffic type, a second traffic type, and a third traffic type, where write traffic of log information within the preset duration corresponding to a service of the first traffic type is located in a first traffic range, write traffic of log information within the preset duration corresponding to a service of the second traffic type is located in a second traffic range, and write traffic of log information within the preset duration corresponding to a service of the third traffic type is located in a third traffic range. The first traffic range is greater than the second traffic range, and the second traffic range is greater than the third traffic range. For example, the first traffic type, the second traffic type, and the third traffic type are a high-traffic type, a medium-traffic type, and a low-traffic type in sequence.
The embodiments of the present disclosure does not specifically limit values of the first traffic range, the second traffic range, and the third traffic range. In an implementation, the write traffic of the log information within the preset duration may be an average write traffic within the preset duration. For example, the first traffic range in which the average write traffic of the service of the high-traffic type is located is greater than or equal to 1 G/s. The second traffic range in which the average write traffic of the service of the medium-traffic type is located is greater than 30 M/s and less than 1 G/s. The third traffic range in which the average write traffic of the service of the low-traffic type is located is less than 30 M/s.
In the embodiments of the present disclosure, a value of the preset duration is not specifically limited. In an implementation, the preset duration is 1 hour, 12 hours, 24 hours, or the like. It should be noted that in different time periods, a traffic type of a same service may be different.
S202: determining, based on the service identifier and the traffic type of the service, at least one target log server that processes the write request, and determining, based on the service identifier and the traffic type of the service, storage path information of the log information of the service on the log storage unit.
In the embodiments of the present disclosure, the request processing unit includes a plurality of log servers, and the plurality of log servers correspond to a plurality of log server identifiers sorted by size. The log server identifiers are used to distinguish different log servers. In an implementation, a log server identifier may be a number corresponding to a server. For example, the plurality of log servers are 100 log servers. The plurality of log server identifiers sorted by size are log server 1, log server 2, log server 3, . . . , and log server 100 in sequence.
In an implementation, the determining, based on the service identifier and the traffic type of the service, at least one target log server that processes the write request may include the following steps (a1) to (a3).
(a1) Performing hash modulo processing on the service identifier based on a quantity of a plurality of log server identifiers, to determine an obtained remainder as a first hash parameter corresponding to the service identifier, and selecting a first target server identifier corresponding to the first hash parameter from the plurality of log server identifiers.
In this step, the hash modulo processing is performed on the service identifier based on the quantity of the plurality of log server identifiers sorted by size. That is, hash operation (for example, consistency hash operation) is first performed on the service identifier, to obtain a hash value corresponding to the service identifier; then modulo is performed on the hash value based on the quantity of the plurality of log server identifiers, and the obtained remainder is used as the first hash parameter corresponding to the service identifier. The first hash parameter (namely, the obtained remainder) is the same as a sequence number of the first target server identifier in the plurality of log server identifiers sorted by size. For example, the first hash parameter is 3, and thus the first target server identifier is the third log server identifier in the plurality of log server identifiers, namely, the log server 3.
(a2) Determining, based on the traffic type of the service and the first target server identifier, sort position information of at least one target log server that processes the write request.
In the embodiments of the present disclosure, the service of the high-traffic type has a relatively high average write traffic, and a corresponding quantity of log servers is relatively large; the service of the low-traffic type has a relatively low average write traffic, and a corresponding quantity of log servers is relatively small.
In an implementation, the sort position information is a sort range, in the plurality of log server identifiers sorted by size, of at least one target server identifier corresponding to the at least one target log server.
Accordingly, this step may include: determining a start sequence number of the sort range based on the first target server identifier, and determining an end sequence number of the sort range based on the traffic type of the service, where a quantity of sequence numbers within the sort range is the same as a quantity of target log servers that process the service.
In an implementation, a correspondence between traffic types of services and quantities of log servers that process the services may be stored in advance, so that based on the traffic type of the service, the quantity of the target log servers that process the service can be determined from the stored correspondence between the traffic types of services and the quantities of log servers.
For example, the traffic type of the service includes the high-traffic type, the medium-traffic type, and the low-traffic type. The correspondence between the traffic types of services and the quantities of log servers is as follows: a quantity of log servers corresponding to the high-traffic type is 20, a quantity of log servers corresponding to the medium-traffic type is 10, and a quantity of log servers corresponding to the low-traffic type is 3. Assuming that the traffic type of the service is the high-traffic type, the quantity of the target log servers corresponding to the traffic type of the service is determined from the stored correspondence between the traffic types of services and the quantities of log servers as 20. In this case, if the plurality of log server identifiers sorted by size are the log server 1, the log server 2, the log server 3, . . . , and the log server 100 in sequence, and the first target server identifier is log server 11 (the start sequence number is 11), the end sequence number of the sort range is determined based on the traffic type of the service as 30 (corresponding to log server 30). In this case, the quantity (20) of sequence numbers within the sort range (11-30) is the same as the quantity (20) of the target log servers that process the service.
(a3) Selecting the at least one target log server that processes the write request from the plurality of log servers based on the sort position information.
For example, the sort position information is the start sequence number of 11 and the end sequence number of 30. Correspondingly, the at least one target log server that is selected from the plurality of log servers and processes the write request is the log server 11, log server 12, . . . , and the log server 30 in sequence.
Here, since for each service, the at least one target log server corresponding to the traffic type of the service is selected from the plurality of log servers, efficiency of writing the log information is improved.
In the embodiments of the present disclosure, the log storage unit includes a plurality of storage directories, and each storage directory includes a plurality of storage partitions sorted by size. It should be noted that the storage directory and the storage partition are logical levels, and do not represent specific physical storage space. For example, the storage directory may be a table name.
In an implementation, the determining, based on the service identifier and the traffic type of the service, the storage path information of the log information of the service on the log storage unit may include the following steps (b1) to (b3).
(b1) Selecting a target storage directory corresponding to the traffic type of the service from a plurality of storage directories, and determining a quantity of a plurality of storage partitions that the target storage directory includes and are sorted by size.
For example, the target storage directory may be storage table 1. The storage table 1 may include 20 storage partitions sorted by size, and the 20 storage partitions are storage partition 1, storage partition 2, storage partition 3, . . . , and storage partition 20 in sequence.
(b2) Performing hash modulo processing on the service identifier based on the quantity of the plurality of storage partitions that the target storage directory includes and are sorted by size, to determine an obtained remainder as a second hash parameter corresponding to the service identifier, and selecting a target storage partition corresponding to the second hash parameter from the plurality of storage partitions that the target storage directory includes and are sorted by size.
In this step, the hash modulo processing is performed on the service identifier based on the quantity of the plurality of storage partitions sorted by size. That is, hash operation (for example, consistency hash operation) is first performed on the service identifier, to obtain a hash value corresponding to the service identifier; then modulo is performed on the hash value based on the quantity of the plurality of storage partitions, and the obtained remainder is used as the second hash parameter corresponding to the service identifier. The second hash parameter (namely, the obtained remainder) is the same as a sequence number of the target storage partition in the plurality of storage partitions sorted by size. For example, the second hash parameter is 5, and thus the target storage partition is the fifth storage partition in the plurality of storage partitions, namely, storage partition 5.
(b3) Determining the target storage partition within the target storage directory as the storage path information of the log information of the service on the log storage unit.
For example, the storage partition 5 in the storage table 1 is determined as the storage path information of the log information of the service on the log storage unit.
Here, since for each service, the storage path information (for example, a storage node) corresponding to the traffic type of the service is selected from the plurality of storage directories within the log storage unit, writing affinity binding between the service and the storage node is realized, and locality of service logs is effectively improved, so that it is convenient to quickly query log information corresponding to a certain service.
S203: writing, through the at least one target log server, the log information of the service into the log storage unit based on the storage path information.
In an implementation, when there is a large quantity of log information of the service, the log information may be distributed to each target log server by quantity, to ensure load balancing of a plurality of target log servers; or a plurality of pieces of log information may be distributed to each target log server based on a type of the log information.
In the method for processing a log request provided in the embodiments, for the write request for the log information of each service, the service identifier of the service and the traffic type of the service are obtained, where the traffic type is used to indicate the write traffic range of the log information of the service within the preset duration; at least one target log server that processes the write request is determined based on the service identifier and the traffic type of the service, and the storage path information of the log information of the service on the log storage unit is determined based on the service identifier and the traffic type of the service; and the log information of the service is written into the log storage unit based on the storage path information through the at least one target log server. In the embodiments of the present application, when a write request for log information of a certain service is received, a log server that processes the write request and storage path information of the log information are determined based on the traffic size of the service. In this way, log information of different services can be stored in different locations through different log servers, that is, an architecture with computing and storage separated is adopted, and larger-scale log writing and storage are supported, so that writing and querying of the log information are facilitated. Therefore, efficiency of storing the log information is improved, and management of high-traffic log information is realized.
It should be noted that within different time periods, a traffic type of a same service may be different, and accordingly, storage path information may also be different. As shown in
S301: for each service, obtaining a plurality of storage time periods corresponding to log information of the service and storage path information corresponding to each storage time period.
S302: establishing a correspondence between a service identifier, the storage path information, and the storage time periods, and store the correspondence between the service identifier, the storage path information, and the storage time periods in the metadata server.
In the embodiments of the present disclosure, the log information of the service may be queried in a plurality of manners. For example, retrieval modes such as inverted indexing, full- text word segmentation indexing, and pattern matching may be customized. In some embodiments, a query request may include a target service identifier and a query time period. Correspondingly, as shown in
S401: in response to receiving a query request for log information of a target service, obtaining a target service identifier of the target service and a query time period.
In some embodiments, the log information of the service may be queried through the service identifier and the query time period. Correspondingly, this step is: in response to receiving the query request for the log information of the target service, obtaining the target service identifier of the target service and the query time period from the query request.
In some other embodiments, a plurality of services are used to process one business request, and log information of the plurality of services related to a business identifier may be queried through the business identifier. Correspondingly, in response to receiving the query request for the log information of the target service, obtaining the target service identifier of the target service and the query time period includes: in response to receiving a query request for a target business identifier, obtaining at least one target service corresponding to the target business identifier; and for each target service, determining that a query request for log information of the target service is received, and obtaining a target service identifier of the target service and a query time period.
In an implementation, when a service receives a business request, a business identifier (for example, logid) may be generated. When a downstream service is called, the logid is transparently transmitted to the downstream service. Here, the logid may be used as a standard field of the log information, to connect upstream and downstream log information of different microservices in series. Logids corresponding to the plurality of services that process the same business request are the same.
S402: determining, based on the target service identifier and the query time period, target storage path information corresponding to the target service identifier and the query time period from a correspondence between service identifiers, storage path information, and storage time periods that is stored in the metadata server; and determining, based on the target service identifier and the query time period, a target log server that processes the query request.
In an implementation, the plurality of log servers correspond to a plurality of log server identifiers sorted by size; and correspondingly, the determining, based on the target service identifier and the query time period, the target log server that processes the query request may include: determining, from the correspondence between the service identifier, the storage path information, and the storage time periods of each service, a plurality of storage time periods that correspond to the target service identifier and are sorted by sequence; determining a sequence number of the query time period in the plurality of storage time periods sorted by sequence; performing hash modulo processing on the sequence number corresponding to the query time period based on a quantity of the plurality of log server identifiers, to determine an obtained remainder as a third hash parameter corresponding to the sequence number, and selecting a second target server identifier corresponding to the third hash parameter from the plurality of log server identifiers; and determining a log server corresponding to the second target server identifier as the target log server that processes the query request.
The hash modulo processing is performed on the sequence number corresponding to the query time period based on the quantity of the plurality of log server identifiers. That is, hash operation (for example, consistency hash operation) is first performed on the sequence number corresponding to the query time period, to obtain a hash value corresponding to the sequence number; then modulo is performed on the hash value based on the quantity of the plurality of log server identifiers, and the obtained remainder is used as the third hash parameter corresponding to the sequence number. The third hash parameter (namely, the obtained remainder) is the same as a sequence number of the second target server identifier in the plurality of log server identifiers sorted by size. For example, the third hash parameter is 5, and thus the second target server identifier is the fifth log server identifier in the plurality of log server identifiers, namely, log server 5.
S403: querying, through the target log server, log information of the target service within the query time period from the log storage unit based on the target storage path information.
In the embodiments of the present disclosure, since log information of different services can be stored in different locations through different log servers, when querying log information of a certain service, the querying is performed from a corresponding storage location instead of querying in the entire log storage unit. Therefore, a search range is reduced, and efficiency of querying the log information is improved.
In some other embodiments, the log information of the target service within the query time period includes at least one log packet and a packet identifier corresponding to each log packet, and when referring to log information of a certain service, a user usually hopes to view a complete log stream of the service on a platform. To this end, a capability of context retrieval is realized: assigning a unique id (also the packet identifier) to each log packet, and assigning id ±1 to a previous packet and a next packet of the log packet. The log stream may be viewed in sequence or in reverse sequence on the platform through an auto-increase/auto-decrease operation on the id.
In an implementation, the query request may be a packet identifier corresponding to a log packet. As shown in
S501: displaying, on an information display interface, a first log packet in the log information and a first packet identifier corresponding to the first log packet.
For example, log information corresponding to service A includes 20 log packets. A first log packet (for example, the tenth log packet) in the log information and a first packet identifier (for example, log packet 10) corresponding to the first log packet are displayed on the information display interface.
S502: in response to receiving a first switching operation for the first packet identifier, displaying, on the information display interface, a second log packet in the log information and a second packet identifier corresponding to the second log packet, where the second packet identifier is a next packet identifier of the first packet identifier.
In an implementation, the first switching operation may be a trigger operation on an “increase button”. For example, the “increase button” is “+”, and is located on a right side of the first packet identifier.
S503: in response to receiving a second switching operation for the first packet identifier, displaying, on the information display interface, a third log packet in the log information and a third packet identifier corresponding to the third log packet, where the third packet identifier is a previous packet identifier of the first packet identifier.
In an implementation, the second switching operation may be a trigger operation on a “decrease button”. For example, the “decrease button” is “−”, and is located on a left side of the first packet identifier.
In the embodiments of the present application, the log information of the target service is displayed by means of a log packet, so that the user can obtain required log information from the context of the log packet.
It should be noted that the log information further includes a log type, where the log type may include an error log type. In the method, the log information of the error log type may further be separately shunted and stored, to facilitate the user to directly query abnormal log information and facilitate aggregation on indicators of the abnormal log information. In an implementation, the user may query the corresponding abnormal log information through a regular expression.
The obtaining module 601 is configured to obtain, for a write request for log information of each service, a service identifier of the service and a traffic type of the service, where the traffic type is used to indicate a write traffic range of the log information of the service within a preset duration.
The determining module 602 is configured to determine, based on the service identifier and the traffic type of the service, at least one target log server that processes the write request, and determine, based on the service identifier and the traffic type of the service, storage path information of the log information of the service on the log storage unit.
The processing module 603 is configured to write, through the at least one target log server, the log information of the service into the log storage unit based on the storage path information.
According to one or more embodiments of the present disclosure, the plurality of log servers correspond to a plurality of log server identifiers sorted by size; and correspondingly, when determining, based on the service identifier and the traffic type of the service, at least one target log server that processes the write request, the determining module 602 is specifically configured to: perform hash modulo processing on the service identifier based on a quantity of the plurality of log server identifiers, to determine an obtained remainder as a first hash parameter corresponding to the service identifier, and select a first target server identifier corresponding to the first hash parameter from the plurality of log server identifiers; determine, based on the traffic type of the service and the first target server identifier, sort position information of the at least one target log server that processes the write request; and select the at least one target log server that processes the write request from the plurality of log servers based on the sort position information.
According to one or more embodiments of the present disclosure, the log storage unit includes a plurality of storage directories, and each storage directory includes a plurality of storage partitions sorted by size; and correspondingly, when determining, based on the service identifier and the traffic type of the service, the storage path information of the log information of the service on the log storage unit, the determining module 602 is specifically configured to: select a target storage directory corresponding to the traffic type of the service from the plurality of storage directories, and determine a quantity of the plurality of storage partitions that the target storage directory includes and are sorted by size; perform hash modulo processing on the service identifier based on the quantity of the plurality of storage partitions that the target storage directory includes and are sorted by size, to determine an obtained remainder as a second hash parameter corresponding to the service identifier, and select a target storage partition corresponding to the second hash parameter from the plurality of storage partitions that the target storage directory includes and are sorted by size; and determine the target storage partition within the target storage directory as the storage path information of the log information of the service on the log storage unit.
According to one or more embodiments of the present disclosure, the log storage system further includes a metadata server; and the apparatus further includes: a storage module; wherein the storage module is configured to: for each service, obtain a plurality of storage time periods corresponding to log information of the service and storage path information corresponding to each storage time period; and establish a correspondence between a service identifier, the storage path information, and the storage time periods, and store the correspondence between the service identifier, the storage path information, and the storage time periods in the
According to one or more embodiments of the present disclosure, the apparatus further includes: a query module; wherein the query module is configured to: in response to receiving a query request for log information of a target service, obtain a target service identifier of the target service and a query time period; determine, based on the target service identifier and the query time period, target storage path information corresponding to the target service identifier and the query time period from a correspondence between service identifiers, storage path information, and storage time periods that is stored in the metadata server, and determine, based on the target service identifier and the query time period, a target log server that processes the query request; and query, through the target log server, log information of the target service within the query time period from the log storage unit based on the target storage path information.
According to one or more embodiments of the present disclosure, the plurality of log servers correspond to a plurality of log server identifiers; and when determining, based on the target service identifier and the query time period, the target log server that processes the query request, the query module is specifically configured to: determine, from the correspondence between the service identifier, the storage path information, and the storage time periods of each service, a plurality of storage time periods that correspond to the target service identifier and are sorted by sequence; determine a sequence number of the query time period in the plurality of storage time periods sorted by sequence; perform hash modulo processing on the sequence number corresponding to the query time period based on a quantity of the plurality of log server identifiers, to determine an obtained remainder as a third hash parameter corresponding to the sequence number, and select a second target server identifier corresponding to the third hash parameter from the plurality of log server identifiers; and determine a log server corresponding to the second target server identifier as the target log server that processes the query request.
According to one or more embodiments of the present disclosure, a plurality of services are used to process one business request; and correspondingly, when in response to receiving the query request for the log information of the target service, obtaining the target service identifier of the target service and the query time period, the query module is specifically configured to: in response to receiving a query request for a target business identifier, obtain at least one target service corresponding to the target business identifier; and for each target service, determine that a query request for log information of the target service is received, and obtain a target service identifier of the target service and a query time period.
According to one or more embodiments of the present disclosure, the log information of the target service within the query time period includes at least one log packet and a packet identifier corresponding to each log packet; and correspondingly, the apparatus further includes: a display module, wherein the display module is configured to: display, on an information display interface, a first log packet in the log information and a first packet identifier corresponding to the first log packet; in response to receiving a first switching operation for the first packet identifier, display, on the information display interface, a second log packet in the log information and a second packet identifier corresponding to the second log packet, where the second packet identifier is a next packet identifier of the first packet identifier; and in response to receiving a second switching operation for the first packet identifier, display, on the information display interface, a third log packet in the log information and a third packet identifier corresponding to the third log packet, where the third packet identifier is a previous packet identifier of the first packet identifier.
The obtaining module 601, the determining module 602, and the processing module 603 are connected in sequence. The apparatus for processing a log request provided in the embodiments may execute the technical solutions in the above method embodiments. The implementation principles and technical effects thereof are similar, and are not described here again in the embodiments.
As shown in
Generally, the following apparatuses may be connected to the I/O interface 705: an input apparatus 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, and a gyroscope; an output apparatus 707 including, for example, a liquid crystal display (LCD for short), a speaker, and a vibrator; the storage apparatus 708 including, for example, a tape and a hard disk; and a communication apparatus 709. The communication apparatus 709 may allow the electronic device 700 to perform wireless or wired communication with other devices to exchange data. Although
In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, where the computer program includes program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded from a network through the communication apparatus 709 and installed, or installed from the storage apparatus 708, or installed from the ROM 702. When the computer program is executed by the processing apparatus 701, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.
It should be noted that the above computer-readable medium described in the present disclosure may be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be, for example but not limited to, electric, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of the computer- readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer magnetic disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as a part of a carrier, where the data signal carries computer-readable program code. The propagated data signal may be in various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium. The computer-readable signal medium can send, propagate, or transmit a program used by or in combination with an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium may be transmitted by any suitable medium, including but not limited to: electric wires, optical cables, radio frequency (RF), or the like, or any suitable combination thereof.
The above computer-readable medium may be contained in the above electronic device. Or, the computer-readable medium may exist independently, without being assembled into the electronic device.
The above computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.
The computer program code for performing the operations in the present disclosure may be written in one or more programming languages or a combination thereof, where the programming languages include an object-oriented programming language, such as Java, Smalltalk, C++, and further include conventional procedural programming languages, such as “C” language or similar programming languages. The program code may be completely executed on a user computer, partially executed on a user computer, executed as an independent software package, partially executed on a user computer and partially executed on a remote computer, or completely executed on a remote computer or server. In the circumstance involving the remote computer, the remote computer may be connected to the user computer over any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, connected over the Internet using an Internet service provider).
The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions, and operations of the possible implementations of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, program segment, or part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the blocks may also occur in an order different from that marked in the accompanying drawings. For example, two blocks shown in succession may actually be performed substantially in parallel, or they may sometimes be performed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and/or the flowcharts, and a combination of the blocks in the block diagrams and/or the flowcharts may be implemented by a dedicated hardware-based system that executes specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.
The units involved in descriptions of the embodiments of the present disclosure may be implemented by software, or may be implemented by hardware. The name of a unit does not constitute a limitation on the unit itself in some cases. For example, the first obtaining unit may also be described as “a unit for obtaining at least two Internet Protocol addresses”.
The functions described herein above may be performed at least partially by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system on chip (SOC), a complex programmable logic device (CPLD), and the like.
In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program used by or in combination with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include but is not limited to electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples of the machine-readable storage medium may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
In a first aspect, according to one or more embodiments of the present disclosure, there is provided a method for processing a log request, applied to a log storage system, where the log storage system includes a request processing unit and a log storage unit, where the request processing unit includes a plurality of log servers; and the method includes:
According to one or more embodiments of the present disclosure, the plurality of log servers correspond to a plurality of log server identifiers sorted by size; and correspondingly, the determining, based on the service identifier and the traffic type of the service, at least one target log server that processes the write request includes: performing hash modulo processing on the service identifier based on a quantity of the plurality of log server identifiers, to determine an obtained remainder as a first hash parameter corresponding to the service identifier, and selecting a first target server identifier corresponding to the first hash parameter from the plurality of log server identifiers; determining, based on the traffic type of the service and the first target server identifier, sort position information of the at least one target log server that processes the write request; and selecting the at least one target log server that processes the write request from the plurality of log servers based on the sort position information.
According to one or more embodiments of the present disclosure, the log storage unit includes a plurality of storage directories, and each storage directory includes a plurality of storage partitions sorted by size; and correspondingly, the determining, based on the service identifier and the traffic type of the service, the storage path information of the log information of the service on the log storage unit includes: selecting a target storage directory corresponding to the traffic type of the service from the plurality of storage directories, and determining a quantity of a plurality of storage partitions that the target storage directory includes and are sorted by size; performing hash modulo processing on the service identifier based on the quantity of the plurality of storage partitions that the target storage directory includes and are sorted by size, to determine an obtained remainder as a second hash parameter corresponding to the service identifier, and selecting a target storage partition corresponding to the second hash parameter from the plurality of storage partitions that the target storage directory includes and are sorted by size; and determining the target storage partition within the target storage directory as the storage path information of the log information of the service on the log storage unit.
According to one or more embodiments of the present disclosure, the log storage system further includes a metadata server; and the method further includes: for each service, obtaining a plurality of storage time periods corresponding to log information of the service and storage path information corresponding to each storage time period; and establishing a correspondence between a service identifier, the storage path information, and the storage time periods, and storing the correspondence between the service identifier, the storage path information, and the storage time periods in the metadata server.
According to one or more embodiments of the present disclosure, the method further includes: in response to receiving a query request for log information of a target service, obtaining a target service identifier of the target service and a query time period; determining, based on the target service identifier and the query time period, target storage path information corresponding to the target service identifier and the query time period from a correspondence between service identifiers, storage path information, and storage time periods that is stored in the metadata server; and determining, based on the target service identifier and the query time period, a target log server that processes the query request; and querying, through the target log server, log information of the target service within the query time period from the log storage unit based on the target storage path information.
According to one or more embodiments of the present disclosure, the plurality of log servers correspond to a plurality of log server identifiers; and the determining, based on the target service identifier and the query time period, the target log server that processes the query request includes: determining, from the correspondence between the service identifier, the storage path information, and the storage time periods of each service, a plurality of storage time periods that correspond to the target service identifier and are sorted by sequence; determining a sequence number of the query time period in the plurality of storage time periods sorted by sequence; performing hash modulo processing on the sequence number corresponding to the query time period based on a quantity of the plurality of log server identifiers, to determine an obtained remainder as a third hash parameter corresponding to the sequence number, and selecting a second target server identifier corresponding to the third hash parameter from the plurality of log server identifiers; and determining a log server corresponding to the second target server identifier as the target log server that processes the query request.
According to one or more embodiments of the present disclosure, a plurality of services are used to process one service request; and correspondingly, in response to receiving the query request for the log information of the target service, obtaining the target service identifier of the target service and the query time period includes: in response to receiving a query request for a target business identifier, obtaining at least one target service corresponding to the target business identifier; and for each target service, determining that a query request for log information of the target service is received, and obtaining a target service identifier of the target service and a query time period.
According to one or more embodiments of the present disclosure, the log information of the target service within the query time period includes at least one log packet and a packet identifier corresponding to each log packet; and correspondingly, after querying, through the target log server, the log information of the target service within the query time period from the log storage unit based on the target storage path information, the method further includes: displaying, on an information display interface, a first log packet in the log information and a first packet identifier corresponding to the first log packet; in response to receiving a first switching operation for the first packet identifier, displaying, on the information display interface, a second log packet in the log information and a second packet identifier corresponding to the second log packet, where the second packet identifier is a next packet identifier of the first packet identifier; and in response to receiving a second switching operation for the first packet identifier, displaying, on the information display interface, a third log packet in the log information and a third packet identifier corresponding to the third log packet, where the third packet identifier is a previous packet identifier of the first packet identifier.
In a second aspect, according to one or more embodiments of the present disclosure, there is provided an apparatus for processing a log request, applied to a log storage system, where the log storage system includes a request processing unit and a log storage unit, where the request processing unit includes a plurality of log servers; and the apparatus includes:
According to one or more embodiments of the present disclosure, the plurality of log servers correspond to a plurality of log server identifiers sorted by size; and correspondingly, when determining, based on the service identifier and the traffic type of the service, at least one target log server that processes the write request, the determining module is specifically configured to: perform hash modulo processing on the service identifier based on a quantity of the plurality of log server identifiers, to determine an obtained remainder as a first hash parameter corresponding to the service identifier, and select a first target server identifier corresponding to the first hash parameter from the plurality of log server identifiers; determine, based on the traffic type of the service and the first target server identifier, sort position information of the at least one target log server that processes the write request; and select the at least one target log server that processes the write request from the plurality of log servers based on the sort position information.
According to one or more embodiments of the present disclosure, the log storage unit includes a plurality of storage directories, and each storage directory includes a plurality of storage partitions sorted by size; and correspondingly, that when determining, based on the service identifier and the traffic type of the service, the storage path information of the log information of the service on the log storage unit, the determining module is specifically configured to: select a target storage directory corresponding to the traffic type of the service from the plurality of storage directories, and determine a quantity of the plurality of storage partitions that the target storage directory includes and are sorted by size; perform hash modulo processing on the service identifier based on the quantity of the plurality of storage partitions that the target storage directory includes and are sorted by size, to determine an obtained remainder as a second hash parameter corresponding to the service identifier, and select a target storage partition corresponding to the second hash parameter from the plurality of storage partitions that the target storage directory includes and are sorted by size; and determine the target storage partition within the target storage directory as the storage path information of the log information of the service on the log storage unit.
According to one or more embodiments of the present disclosure, the log storage system further includes a metadata server; and the apparatus further includes: a storage module; where the storage module is configured to: for each service, obtain a plurality of storage time periods corresponding to log information of the service and storage path information corresponding to each storage time period; and establish a correspondence between a service identifier, the storage path information, and the storage time periods, and store the correspondence between the service identifier, the storage path information, and the storage time periods in the
According to one or more embodiments of the present disclosure, the apparatus further includes: a query module; where the query module is configured to: in response to receiving a query request for log information of a target service, obtain a target service identifier of the target service and a query time period; determine, based on the target service identifier and the query time period, target storage path information corresponding to the target service identifier and the query time period from a correspondence between service identifiers, storage path information, and storage time periods that is stored in the metadata server; and determine, based on the target service identifier and the query time period, a target log server that processes the query request; and query, through the target log server, log information of the target service within the query time period from the log storage unit based on the target storage path information.
According to one or more embodiments of the present disclosure, the plurality of log servers correspond to a plurality of log server identifiers; and when determining, based on the target service identifier and the query time period, the target log server that processes the query request, the query module is specifically configured to: determine, from the correspondence between the service identifier, the storage path information, and the storage time periods of each service, a plurality of storage time periods that correspond to the target service identifier and are sorted by sequence; determine a sequence number of the query time period in the plurality of storage time periods sorted by sequence; perform hash modulo processing on the sequence number corresponding to the query time period based on a quantity of the plurality of log server identifiers, to determine an obtained remainder as a third hash parameter corresponding to the sequence number, and select a second target server identifier corresponding to the third hash parameter from the plurality of log server identifiers; and determine a log server corresponding to the second target server identifier as the target log server that processes the query request.
According to one or more embodiments of the present disclosure, a plurality of services are used to process one service request; and correspondingly, when in response to receiving the query request for the log information of the target service, obtaining the target service identifier of the target service and the query time period, the query module is specifically configured to: in response to receiving a query request for a target business identifier, obtain at least one target service corresponding to the target business identifier; and for each target service, determine that a query request for log information of the target service is received, and obtain a target service identifier of the target service and a query time period.
According to one or more embodiments of the present disclosure, the log information of the target service within the query time period includes at least one log packet and a packet identifier corresponding to each log packet; and correspondingly, the apparatus further includes: a display module, where the display module is configured to: display, on an information display interface, a first log packet in the log information and a first packet identifier corresponding to the first log packet; in response to receiving a first switching operation for the first packet identifier, display, on the information display interface, a second log packet in the log information and a second packet identifier corresponding to the second log packet, where the second packet identifier is a next packet identifier of the first packet identifier; and in response to receiving a second switching operation for the first packet identifier, display, on the information display interface, a third log packet in the log information and a third packet identifier corresponding to the third log packet, where the third packet identifier is a previous packet identifier of the first packet identifier.
In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, including: a processor, and a memory communicatively connected to the processor;
In a fourth aspect, according to one or more embodiments of the present disclosure, there is provided a computer-readable storage medium, where the computer-readable storage medium stores computer-execution instructions, and when a processor executes the computer-execution instructions, the method for processing a log request according to the first aspect and any possible design of the first aspect is implemented.
In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method for processing a log request according to the first aspect and any possible design of the first aspect is implemented.
The foregoing descriptions are merely preferred embodiments of the present disclosure and an illustration of the applied technical principles. A person skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by specific combinations of the above technical features, and shall also cover other technical solutions formed by any combination of the above technical features or equivalent features thereof without departing from the above concept of disclosure, for example, technical solutions formed by replacing the above features and technical features having similar functions as disclosed in the present disclosure (but not limited thereto) with each other.
In addition, although the various operations are depicted in a specific order, it should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the foregoing discussions, these details should not be construed as limiting the scope of the present disclosure. Some features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. In contrast, various features described in the context of a single embodiment may also be implemented in a plurality of embodiments individually or in any suitable sub-combination.
Although the subject matter has been described in a language specific to structural features and/or logical actions of methods, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely exemplary forms of implementing the claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202311766870.5 | Dec 2023 | CN | national |
