Patent Application
20240428097
This application belongs to the field of mobile communication technologies, and to a method and an apparatus for determining model accuracy, and a network-side device.
In a communication network, some network elements are introduced for intelligent data analysis, and some data analysis results (analytics) (or referred to as inference data results) of tasks are generated. The data analysis results can assist devices in and out of the network in policy decision-making. In this way, intelligent policy decision-making of the devices is improved by using an artificial intelligence (AI) method.
Embodiments of this application provide a method and an apparatus for determining model accuracy, and a network-side device.
According to a first aspect, a method for determining model accuracy is provided and applied to a first network element. The method includes:
According to a second aspect, an apparatus for determining model accuracy is provided, including:
According to a third aspect, a method for determining model accuracy is provided and applied to a second network element. The method includes:
According to a fourth aspect, an apparatus for determining model accuracy is provided, including:
According to a fifth aspect, a method for determining model accuracy is provided and applied to a third network element. The method includes:
According to a sixth aspect, an apparatus for determining model accuracy is provided, including:
According to a seventh aspect, a network-side device is provided. The network-side device includes a processor and a memory. The memory stores a program or an instruction that can be run on the processor, and when the program or the instruction is executed by the processor, the steps of the method according to the first aspect are implemented, the steps of the method according to the third aspect are implemented, or the steps of the method according to the fifth aspect are implemented.
According to an eighth aspect, a system for determining model accuracy is provided, including: a network-side device, where the network-side device includes a first network element, a second network element, and a third network element, the first network element may be configured to perform the operations of the method for determining model accuracy as described above, the second network element may be configured to perform the operations of the method for determining model accuracy as described above, and the third network element may be configured to perform the operations of the determining model accuracy method as described above.
According to a ninth aspect, a non-transitory readable storage medium is provided. The non-transitory readable storage medium stores a program or an instruction. When the program or the instruction is executed by a processor, the steps of the method according to the first aspect are implemented, the steps of the method according to the third aspect are implemented, or the steps of the method according to the fifth aspect are implemented.
According to a tenth aspect, a chip is provided. The chip includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is configured to run a program or an instruction to implement the method in the first aspect, or implement the method in the third aspect, or implement the method in the fifth aspect.
According to an eleventh aspect, a computer program/program product is provided. The computer program/program product is stored in a non-transitory storage medium. The computer program/program product is executed by at least one processor to implement the method in the first aspect, or implement the method in the third aspect, or implement the method in the fifth aspect.
Technical solutions in embodiments of this application are clearly described in the following with reference to accompanying drawings in embodiments of this application. It is clear that the described embodiments are merely some rather than all of embodiments of this application. All other embodiments obtained by a person skilled in the art based on embodiments of this application fall within the protection scope of this application.
In the specification and claims of this application, terms such as “first” and “second” are used to distinguish similar objects, but are not used to describe a specific sequence or order. It should be understood that terms used in this way are exchangeable in a proper case, so that embodiments of this application can be implemented in an order different from the order shown or described herein. In addition, objects distinguished by using “first” and “second” are usually of a same type, and the terms do not limit a quantity of the objects. For example, a first object can be one object or more objects. In addition, “and/or” used in this specification and the claims represents at least one of connected objects. The character “/” usually indicates an “or” relationship between associated objects.
It should be noted that technologies described in embodiments of this application are not limited to a long term evolution (LTE) system/an LTE-Advanced (LTE-A) system, and may also be applied to other wireless communication systems such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal frequency division multiple access (OFDMA), and single-carrier frequency division multiple access (SC-FDMA). Terms “system” and “network” in embodiments of the application are often used interchangeably. The described technologies can be applied to the foregoing systems and radio technologies, and can also be applied to other systems and radio technologies. The following describe a 5th generation mobile communication technology (5G) system for an example purpose, and 5G terms are used in most of the following descriptions, but the technologies may also be applied to applications other than the 5G system application, for example, a 6th generation (6G) communication system.
A network data analytics function (NWDAF) may perform AI or machine learning (ML) model training based on training data, to obtain a corresponding model suitable for a specific AI task. Inference is performed on inference input data of the specific AI task based on an AI/ML model, to obtain inference result data corresponding to the specific AI task. A policy control function (PCF) entity performs intelligent policy control and charging (PCC) based on inference result data, for example, formulates an intelligent user camping policy based on inference result data of user service behavior, to improve service experience of a user. Alternatively, an access and mobility management function (AMF) performs an intelligent mobility management operation based on inference result data of a specific AI task, for example, intelligently pages a user based on inference result data of a movement track of a user, to improve paging reachability.
The devices in and out of the network make correct and optimized policy decisions based on AI data analysis results as long as the data analysis results are correct. If accuracy of the data analysis result is low, and the data analysis result is provided as incorrect information to the devices in and out of the network for reference, a wrong policy decision is made or an inappropriate operation is performed. Therefore, it is necessary to ensure the accuracy of the data analysis result.
Although accuracy in training (AiT) of the model in a training phase meets an accuracy requirement for the model, whether inference accuracy (AiU) of the model can meet an accuracy requirement when the model is put into use cannot be determined. A gap may exist due to different data distributions, a low generalization capability of the model, and the like. As a result, the accuracy of the inference result data obtained by using the model is low. When the inference result data is provided to the devices in and out of the network for reference, it is likely that wrong policy decisions are made or inappropriate operations are performed.
The following describes, with reference to the accompanying drawings, a method and an apparatus for determining model accuracy and a network-side device provided in embodiments of this application by using some embodiments and application scenarios thereof.
As shown in
In an implementation, the first network element may be a network element having both a model inference function and a model training function. For example, the first network element is an NWDAF, and the NWDAF may include an analytics logical function (AnLF) network element, and a model training logical function (MTLF) network element.
In another implementation, the first network element includes a model training function network element, and a second network element includes a model inference function network element. For example, the first network element is the MTLF, and the second network element is the AnLF.
If the NWDAF is the first network element, the second network element and the first network element are a same network element in the following embodiments. In other words, the MTLF and the AnLF are combined into the NWDAF.
It should be understood that the first model may be constructed and trained based on an actual requirement, for example, an AI/ML model. Training data is collected by the MTLF, and model training is performed based on the training data. After training is completed, the MTLF sends information about a trained first model to the AnLF.
After determining a triggered task, the AnLF performs inference on the task based on the first model, to obtain inference result data.
It should be understood that the task is a data analysis task and is used to indicate a task type rather than a single task. After triggering the task, the AnLF may determine a first model corresponding to the task based on identification information (analytics ID) of the task, and then perform inference on the task based on the corresponding first model, to obtain inference result data. For example, if an analytics ID of the task is UE mobility, the task is used for predicting a movement track of a terminal (also referred to as user equipment (UE)). In this case, the AnLF may perform inference on the task based on a first model corresponding to UE mobility, and obtained inference result data is predicted terminal location (UE location) information.
The AnLF may perform one or more inferences on the task based on the first model, to obtain a plurality of pieces of inference result data or inference result data including a plurality of output result values.
It should be understood that inference performed on the task by the AnLF may be triggered by a third network element sending a task request message. The third network element is a network element that triggers the task. The third network element includes a consumer network function (consumer NF). The consumer NF may be a 5G network element, an application function (AF), a terminal, or the like. Alternatively, the task may be actively triggered by the AnLF. For example, a phase of validation and measurement is set. In the phase of validation and measurement, the triggered task is actively simulated by the AnLF for measuring the accuracy of the first model.
For brevity, in the following embodiments, examples in which the MTLF is the first network element, the AnLF is the second network element, and the consumer NF is the third network element are used for description.
In a case that it is determined that the first network element is configured to perform task inference, the MTLF calculates the first accuracy corresponding to the first model in the inference process, and the first accuracy is an AiU. There may be various manners of calculating the first accuracy, and only an implementation is provided in embodiments of this application. The step S210 includes:
Optionally, a quantity of correct results in the inference result data may be determined by comparing the inference result data with the corresponding label data, and the quantity of correct results divided by a total number of times is used as the first accuracy. A formula is expressed as follows.
The first accuracy is equal to the quantity of correct results/the total number of times.
Herein, the correct result may indicate that inference result data is consistent with label data, or that a difference between the inference result data and the label result data is within an allowable range.
There may be various expression forms of the first accuracy. The expression form is not limited to a specific percentage value, for example, 90%, or may be a classification expression form, for example, high, medium, low, or the like, or may be normalized data, for example, 0.9.
In an implementation, that the first network element obtains first data of the task includes:
A source device of the label data may be determined by the MTLF based on the first model and task-related information.
The MTLF determines, based on whether the first accuracy meets the preset condition, whether the accuracy of the first model meets the accuracy requirement or degrades. The preset condition may include that the first accuracy is less than a preset threshold or a degrade amplitude reaches a specific degree.
When determining that the accuracy of the first model does not meet the accuracy requirement or degrades, the MTLF sends the first information to the AnLF, to indicate the AnLF that the accuracy of the first model does not meet the accuracy requirement or degrades, so that the AnLF performs a corresponding operation based on the first information. For example, information indicating that the accuracy of the first model does not meet the accuracy requirement or degrades may be sent to the consumer NF.
It can be learned from the technical solution in the foregoing embodiment that, in this embodiment of this application, the first accuracy of the first model is determined by the first network element, where the first accuracy indicates an accuracy of a result of inference performed on the task by using the first model. The first information is sent to the second network element in the case where the first network element determines that the first accuracy meets the preset condition, where the first information indicates that the accuracy of the first model does not meet the accuracy requirement or degrades, so that the accuracy of the model in an actual application process can be monitored, and when the accuracy degrades, corresponding measures can be taken in time to prevent wrong policy decision-making or an inappropriate operation.
Based on the foregoing embodiments, as shown in
The first model is pretrained by the MTLF and provided to the AnLF. A process of training the first model by the MTLF may include steps A1 and A2.
In an implementation, after the MTLF completes training of the first model, the MTLF may perform step A4 of sending information about the trained first model to the AnLF. The information about the first model may be sent by using an Nnwdaf_MLModelProvision_Notify message or an Nnwdaf_MLModelInfo_Response message.
In an implementation, before step A4, the method further includes:
In an implementation, in the training phase of the first model in step A2 or post-training test phase, the MTLF needs to evaluate the accuracy of the first model and calculate second accuracy of the first model, that is, an AiT. The second accuracy may be obtained according to the same calculation formula for the first accuracy. Optionally, the MTLF may set a verification dataset for evaluating the second accuracy of the first model. The verification dataset includes input data of the first model and corresponding label data. The MTLF inputs the input data to the trained first model to obtain output data to determine whether the output data is consistent with the label data through comparing, and then calculates the second accuracy of the first model according to the foregoing formula.
Correspondingly, in an implementation, when sending the information of the first model to the AnLF in step A4, the MTLF may also send the second accuracy of the first model simultaneously, or send the second accuracy of the first model to the AnLF by using an independent message.
In an implementation, if the task is actively triggered by the AnLF, the AnLF may perform step A6 after triggering the task.
In another implementation, if the task is triggered by the consumer NF, before step A6, the method further includes step A5.
The description information of the task may include various types of information, and may include identification information of the task, filter information (analytics filter information) of the task, target information (analytics target) of the task, and the like. A target, a range, and the like related to the task may be determined based on the description information of the task.
The inference input data used for inference of the task in step A6 may be obtained from a source device of the inference input data. Optionally, the source device from which the inference input data corresponding to the task is obtained may be determined based on the task request message received in step A5. An inference input data request message is sent to the source device of the inference input data to obtain the inference input data, and inference is performed on the inference input data based on the first model to obtain the inference result data. The inference input data request message may be sent by using an Nnwdaf_AnalyticsSubscription_Subscribe message or an Nnwdaf_AnalyticsInfo_Request message.
For example, based on the first model corresponding to analytics ID=UE mobility, the AnLF performs inference on the inference input data corresponding to the task, for example, values such as a UE ID, a time, and a current UE service status, to obtain the inference result data, that is output data of the UE location.
In an implementation, step A5 may be before step A3. That is, the AnLF first receives the task request message from the consumer NF, determines the first model corresponding to the task based on the task request message, and then sends a model request to the MTLF providing the first model.
For the first model corresponding to the task, a first model used for the task may be determined based on a task type indicated by an analytics ID in the task request message, or may be determined based on a mapping relationship between an analytics ID and a first model. The first model may be represented by model identification information (model ID) of the model, for example, a model 1.
In another implementation, the steps A1 to A4 may all be after step A5. That is, after receiving the model request message sent by the AnLF, the MTLF trains the first model corresponding to the task, and sends the information of the trained first model to the AnLF.
In an implementation, in a case that the task is triggered by the third network element by using the task request message, after step A6, the method further includes the following steps.
Before step S210, the method further includes:
The second information includes at least one of the following:
In an implementation, step S210 includes steps A9 to A11.
In an implementation, the source device providing the first data is determined based on at least one piece of the following information:
In an implementation, the MTLF may determine at least one piece of the following information based on the foregoing information:
The first model corresponding to the task may determine the first model used for the task based on the task type indicated by the analytics ID in the task request message, or based on the mapping relationship between an analytics ID and a first model.
The type information of the input data of the first model may also be referred to as metadata information, and may include a terminal identifier (UE ID), a time, a current terminal service status, and the like corresponding to the input data.
The type information of the output data of the first model includes a data type, and for example, indicates a tracking area (TA) or a cell of a UE location.
For the first source device, optionally, the AnLF determines, based on the information such as analytics filter information and analytics target in the task request message, the target and the range related to the task, and then determines, based on the target, the range, and metadata information, the network element from which the inference input data corresponding to the task can be obtained as the source device of the inference input data corresponding to the task.
For the second source device, optionally, the AnLF determines, based on the type information of the output data of the first model, a network element device type (an NF type) that can provide the output data; determines, based on the filter information of the task, the target information, and the like, a network element instance corresponding to the network element device type; and uses the network element instance as the source device of the label data. For example, based on a data type, namely, a UE location, of the output data of the first model corresponding to a task, namely, UE mobility, the AnLF determines that data of the UE location may be provided by a network element device type, namely, an AMF type. Then, the AnLF queries a unified data management (UDM) entity or a network repository function (NRF) based on filter information like AOI of the task and a target UE 1 of the task, and obtains that a corresponding AMF instance is an AMF 1. In this way, the AnLF uses the AMF 1 as a source device of label data, and subsequently obtains the label data of the UE location from the AMF 1.
In an implementation, that the MTLF may obtain the first data from the source device of the first data may include:
The first network element obtains the inference input data corresponding to the task from the first source device, that is, the MTLF collects the inference input data by sending the inference input data request message to the first source device.
The first network element obtains the label data corresponding to the task from the second source device, that is, the MTLF collects the label data by sending a label data request message to the second source device. The label data request message may carry type information of the label data, target information corresponding to the label data, time information (for example, a timestamp and a time period), and the like for determining which piece of label data is fed back to the source device of the label data.
The type information of the label data, the target information corresponding to the label data, the time information, and the like in the label data request message may be determined by the MTLF based on the type information of the output data of the first model, the target information of the task, the filter information of the task, and the like respectively. Optionally, the MTLF determines, based on the type information of the output data of the first model, the type information of the label data that needs to be obtained; the MTLF determines, based on the target information of the task, the target information of the label data that needs to be obtained; and if the MTLF determines, based on the filter information of the task, that an inference process of the task is statistical calculation made for a specific time in the past or prediction made for a specific time in the future, the MTLF further needs to obtain label data corresponding to the specific time in the past or the specific time in the future.
For example, the MTLF sends the label data request message to an AMF or a location management function (LMF), where the label data request message carries the data type, namely, the UE location, the target information, namely, a UE 1, and the time information, namely, a time period that correspond to the label data, and is used for requesting the AMF/LMF to feed back data of the UE location of the UE 1 within the time period.
It should be understood that if the AnLF obtains a plurality of pieces of inference result data by performing the inference process one or more times in step A6, the MTLF correspondingly needs to obtain a plurality of pieces of label data corresponding to the plurality of pieces of inference result data.
In another implementation, all or part of the first data may alternatively be carried by the second message. For example, the second message may include the inference input data and/or the inference result data of the task.
In an implementation, as shown in
The preset condition may be set based on an actual requirement. In an implementation, the preset condition includes at least one of the following conditions:
In an implementation, the first information includes at least one piece of the following information:
In an implementation, the first recommended operation information includes at least one of the following:
In an implementation, the fourth information includes at least one piece of the following information:
In an implementation, the second recommended operation information includes at least one of the following operations:
It can be learned from the technical solution in the foregoing embodiment that, in this embodiment of this application, the first accuracy of the first model is determined by the first network element, where the first accuracy indicates the accuracy of the result of inference performed on the task by using the first model. The first information is sent to the second network element in the case where the first network element determines that the first accuracy meets the preset condition, where the first information indicates that the accuracy of the first model does not meet the accuracy requirement or degrades, so that the accuracy of the model in an actual application process can be monitored, and when the accuracy degrades, corresponding measures can be taken in time to prevent wrong policy decision-making or an inappropriate operation.
Based on the foregoing embodiments, as shown in
In an implementation, training data for retraining the first model includes first data of the task, where the first data includes at least one of the following:
In an implementation, the third information further includes at least one of the following:
The new inference result data may carry the identification information of the task, for example, the analytics ID, indicating the corresponding task to the consumer NF.
It can be learned from the technical solutions in the foregoing embodiments that, in embodiments of this application, in the case where the first accuracy meets the preset condition, the first network element retrains the first model, and sends the information about the retrained first model to the second network element, so that when the accuracy of the model degrades, corresponding adjustment measures can be taken in time to quickly restore the accuracy of task inference, so as to prevent wrong policy decision-making or an inappropriate operation.
Based on the foregoing embodiments, as shown in
The new inference result data may carry the identification information of the task, for example, the analytics ID, indicating the corresponding task to the consumer NF.
It can be learned from the technical solutions in the foregoing embodiments that, in embodiments of this application, in the case where the first accuracy meets the preset condition, the second network element performs inference on the task by using the second model obtained from the fourth network element, to obtain the new inference result data, so that when the accuracy of the model degrades, corresponding adjustment measures can be taken in time to quickly restore the accuracy of task inference, so as to prevent wrong policy decision-making or an inappropriate operation.
In the method for determining model accuracy provided in embodiments of this application, an execution entity may be an apparatus for determining model accuracy. In embodiments of this application, an example in which the apparatus for determining model accuracy performs the method for determining model accuracy is used to describe the apparatus for determining model accuracy provided in an embodiment of this application.
As shown in
The execution module 601 is configured to determine first accuracy of a first model, where the first accuracy indicates an accuracy of a result of inference performed on a task by using the first model. The transmission module 602 is configured to send first information to a second network element in a case that it is determined that the first accuracy meets a preset condition, where the first information indicates that accuracy of the first model does not meet an accuracy requirement or degrades.
The apparatus for determining model accuracy is a network element providing the first model, and the second network element is a network element that performs inference on the task.
Optionally, the transmission module 602 is configured to obtain first data of the task.
The execution module 601 is configured to calculate the first accuracy of the first model based on the first data.
The first data includes at least one of the following:
Optionally, the transmission module 602 is configured to:
Optionally, the apparatus for determining model accuracy includes a model training function network element.
Optionally, the second network element includes a model inference function network element.
It can be learned from the technical solution in the foregoing embodiment that, in this embodiment of this application, the first accuracy of the first model is determined. The first information is sent to the second network element in a case that it is determined that the first accuracy meets a preset condition, where the first information indicates that the accuracy of the first model does not meet the accuracy requirement or degrades, so that the accuracy of the model in an actual application process can be monitored, and when the accuracy degrades, corresponding measures can be taken in time to prevent wrong policy decision-making or an inappropriate operation.
Based on the foregoing embodiment, optionally, the transmission module is further configured to receive second information from the second network element, where the second information indicates that the first model is already used for inference of the task.
Optionally, the second information includes at least one of the following:
Optionally, the source device providing the first data is determined based on at least one piece of the following information:
Optionally, the transmission module is configured to perform at least one of the following:
Optionally, the preset condition includes at least one of the following conditions:
Optionally, the first information includes at least one piece of the following information:
Optionally, the first recommended operation information includes at least one of the following operations:
It can be learned from the technical solution in the foregoing embodiment that, in this embodiment of this application, the first accuracy of the first model is determined. The first information is sent to the second network element in the case where it is determined that the first accuracy meets a preset condition, where the first information indicates that the accuracy of the first model does not meet the accuracy requirement or degrades, so that the accuracy of the model in an actual application process can be monitored, and when the accuracy degrades, corresponding measures can be taken in time to prevent wrong policy decision-making or an inappropriate operation.
Based on the foregoing embodiments, in a case that it is determined that the first accuracy meets the preset condition, the execution module is further configured to retrain the first model.
Optionally, training data for retraining the first model includes first data of the task, where
Optionally, after the first model is retrained, the transmission module is further configured to send third information to the second network element, where the third information includes information about a retrained first model.
Optionally, the third information further includes at least one of the following:
It can be learned from the technical solution in the foregoing embodiment that, in this embodiment of this application, in the case where the first accuracy meets the preset condition, the first model is retrained, and the information about the retrained first model is sent to the second network element, so that when the accuracy of the model degrades, corresponding adjustment measures can be taken in time to quickly restore the accuracy of task inference, so as to prevent wrong policy decision-making or an inappropriate operation.
In this embodiment of this application, the apparatus for determining model accuracy may be an electronic device, for example, an electronic device having an operating system, or may be a component in the electronic device, for example, an integrated circuit or a chip. The electronic device may be a terminal or a device other than the terminal. For example, the terminal may include but is not limited to the foregoing listed types of the terminal 11, and the another device may be a server, network attached storage (NAS), and the like. This is not specifically limited in this embodiment of this application.
The apparatus for determining model accuracy according to this embodiment of this application can implement all processes implemented in the method embodiments shown in
As shown in
Optionally, the first network element includes a model training function network element.
Optionally, the second network element includes a model inference function network element.
Optionally, after receiving the first information from the first network element, the method further includes:
Optionally, the third network element includes a consumer network function.
Optionally, the second information includes at least one of the following:
Optionally, the first information includes at least one piece of the following information:
Optionally, the first recommended operation information includes at least one of the following:
Optionally, the fourth information includes at least one piece of the following information:
Optionally, the second recommended operation information includes at least one of the following operations:
Steps S710 and S720 can implement the method embodiments shown in
It can be learned from the technical solution in the foregoing embodiment that, in this embodiment of this application, the second network element performs inference on the task based on the first model, and sends the second information to the first network element, where the second information indicates that the first model is already used for inference of the task. The second network element receives the first information from the first network element, where the first information indicates that the accuracy of the first model does not meet the accuracy requirement or degrades, so that the accuracy of the model in an actual application process can be monitored, and when the accuracy degrades, corresponding measures can be taken in time to prevent wrong policy decision-making or an inappropriate operation.
Based on the foregoing embodiments, after receiving the first information from the first network element, the method further includes:
Optionally, the third information further includes at least one of the following:
In this embodiment of this application, the method embodiment shown in
It can be learned from the technical solution in the foregoing embodiment that, in this embodiment of this application, in the case where the first accuracy meets the preset condition, the first network element retrains the first model, and sends the information about the retrained first model to the second network element, so that when the accuracy of the model degrades, corresponding adjustment measures can be taken in time to quickly restore the accuracy of task inference, so as to prevent wrong policy decision-making or an inappropriate operation.
Based on the foregoing embodiments, after receiving the first information from the first network element, the method further includes:
Optionally, the fourth network element includes a model training function network element.
In this embodiment of this application, the method embodiment shown in
It can be learned from the technical solution in the foregoing embodiment that, in this embodiment of this application, in the case where the first accuracy meets the preset condition, the second network element performs inference on the task by using the second model obtained from the fourth network element, to obtain the new inference result data, so that when the accuracy of the model degrades, corresponding adjustment measures can be taken in time to quickly restore the accuracy of task inference, so as to prevent wrong policy decision-making or an inappropriate operation.
In the method for determining model accuracy provided in this embodiment of this application, an execution entity may be an apparatus for determining model accuracy. In embodiments of this application, an example in which the apparatus for determining model accuracy performs the method for determining model accuracy is used to describe the apparatus for determining model accuracy provided in an embodiment of this application.
As shown in
The inference module 801 is configured to perform inference on a task based on a first model, and send second information to a first network element, where the second information indicates that the first model is already used for inference of the task. The transceiver module 802 is configured to receive first information from the first network element, where the first information indicates that accuracy of the first model does not meet an accuracy requirement or degrades.
Optionally, the first network element includes a model training function network element.
Optionally, the apparatus for determining model accuracy includes a model inference function network element.
Optionally, after receiving the first information from the first network element, the transceiver module 802 is further configured to: send fourth information to a third network element, where the fourth information indicates that the accuracy of the first model does not meet the accuracy requirement or degrades.
Optionally, the third network element includes a consumer network function.
Optionally, the second information includes at least one of the following:
Optionally, the first information includes at least one piece of the following information:
Optionally, the first recommended operation information includes at least one of the following:
Optionally, the fourth information includes at least one piece of the following information:
Optionally, the second recommended operation information includes at least one of the following operations:
It can be learned from the technical solution in the foregoing embodiment that, in this embodiment of this application, inference is performed on the task based on the first model, and the second information is sent to the first network element, where the second information indicates that the first model is already used for inference of the task. The first information is received from the first network element, where the first information indicates that the accuracy of the first model does not meet the accuracy requirement or degrades, so that the accuracy of the model in an actual application process can be monitored, and when the accuracy degrades, corresponding measures can be taken in time to prevent wrong policy decision-making or an inappropriate operation.
Based on the foregoing embodiments, after receiving the first information from the first network element, the transceiver module is further configured to: receive third information from the first network element, where the third information includes information about a retrained first model.
Optionally, the third information further includes at least one of the following:
It can be learned from the technical solution in the foregoing embodiment that, in this embodiment of this application, in the case where the first accuracy meets the preset condition, the first network element retrains the first model, and feeds back the information about the retrained first model, so that when the accuracy of the model degrades, corresponding adjustment measures can be taken in time to quickly restore the accuracy of task inference, so as to prevent wrong policy decision-making or an inappropriate operation.
Based on the foregoing embodiments, after receiving the first information from the first network element, the transceiver module is further configured to:
Optionally, the fourth network element includes a model training function network element.
It can be learned from the technical solution in the foregoing embodiment that, in this embodiment of this application, in the case where the first accuracy meets the preset condition, inference on the task is performed by using the second model obtained from the fourth network element, to obtain the new inference result data, so that when the accuracy of the model degrades, corresponding adjustment measures can be taken in time to quickly restore the accuracy of task inference, so as to prevent wrong policy decision-making or an inappropriate operation.
In this embodiment of this application, the apparatus for determining model accuracy may be an electronic device, for example, an electronic device having an operating system, or may be a component in the electronic device, for example, an integrated circuit or a chip. The electronic device may be a terminal or a device other than the terminal. For example, the terminal may include but is not limited to the foregoing listed types of the terminal 11, and the another device may be a server, network attached storage (NAS), and the like. This is not specifically limited in this embodiment of this application.
The apparatus for determining model accuracy according to this embodiment of this application can implement all processes implemented in the method embodiment shown in
As shown in
Optionally, after the receiving fourth information from the second network element, the method further includes:
The third network element performs at least one of the following operations based on the fourth information:
Optionally, the fourth information includes at least one piece of the following information:
Optionally, the second recommended operation information includes at least one of the following operations:
Optionally, the first network element includes a model training function network element.
Optionally, the second network element includes a model inference function network element.
Optionally, the third network element includes a consumer network function.
Optionally, the fifth network element includes a model inference function network element.
Steps S910 and S920 can implement the method embodiments shown in
It can be learned from the technical solution described in the foregoing embodiment that, in this embodiment of this application, the third network element sends the task request information to the second network element, where the task request information is used for requesting to perform inference on the task; and the third network element receives the fourth information from the second network element, where the fourth information indicates that the accuracy of the first model does not meet the accuracy requirement or degrades, so that the accuracy of the model in an actual application process can be monitored, and when the accuracy degrades, corresponding measures can be taken in time to prevent wrong policy decision-making or an inappropriate operation.
In the method for determining model accuracy provided in this embodiment of this application, an execution entity may be an apparatus for determining model accuracy. In embodiments of this application, an example in which the apparatus for determining model accuracy performs the method for determining model accuracy is used to describe the apparatus for determining model accuracy provided in an embodiment of this application.
As shown in
The sending module 1001 is configured to send task request information to a second network element, where the task request information is used for requesting to perform inference on a task; and the receiving module 1002 is configured to receive fourth information from the second network element, where the fourth information indicates that accuracy of a first model does not meet an accuracy requirement or degrades.
The first model is a model that is provided by a first network element for inference of the task, and the first network element determines that the accuracy of the first model does not meet the accuracy requirement or degrades.
Optionally, after receiving the fourth information from the second network element, the receiving module 1002 is further configured to perform at least one of the following operations based on the fourth information:
Optionally, the fourth information includes at least one piece of the following information:
Optionally, the second recommended operation information includes at least one of the following operations:
Optionally, the first network element includes a model training function network element.
Optionally, the second network element includes a model inference function network element.
Optionally, the apparatus for determining model accuracy includes a consumer network function.
Optionally, the fifth network element includes a model inference function network element.
It can be learned from the technical solution described in the foregoing embodiment that, in this embodiment of this application, the task request information is sent to the second network element, where the task request information is used for requesting to perform inference on the task; and the fourth information is received from the second network element, where the fourth information indicates that the accuracy of the first model does not meet the accuracy requirement or degrades, so that the accuracy of the model in an actual application process can be monitored, and when the accuracy degrades, corresponding measures can be taken in time to prevent wrong policy decision-making or an inappropriate operation.
In this embodiment of this application, the apparatus for determining model accuracy may be an electronic device, for example, an electronic device having an operating system, or may be a component in the electronic device, for example, an integrated circuit or a chip. The electronic device may be a terminal or a device other than the terminal. For example, the terminal may include but is not limited to the foregoing listed types of the terminal 11, and the another device may be a server, network attached storage (NAS), and the like. This is not specifically limited in this embodiment of this application.
The apparatus for determining model accuracy according to this embodiment of this application can implement all processes implemented in the method embodiment shown in
Optionally, as shown in
Optionally, an embodiment of this application further provides a network-side device. As shown in
Optionally, the network-side device 1200 in this embodiment of the present application further provides: an instruction or a program that is stored in the memory 1203 and that is executable on the processor 1201. The processor 1201 invokes the instruction or program in the memory 1203 to execute the method executed by the modules shown in
An embodiment of this application further provides a non-transitory readable storage medium. The non-transitory readable storage medium stores a program or an instruction. When the program or the instruction is executed by a processor, the processes of the foregoing method embodiments for determining model accuracy are implemented, and the same technical effects are achieved. To avoid repetition, details are not described herein again.
The processor is the processor in the terminal in the foregoing embodiments. The non-transitory readable storage medium includes a non-transitory computer-readable storage medium, for example, a computer read only memory ROM, a random-access memory RAM, a magnetic disk, an optical disk, and the like.
An embodiment of this application further provides a chip. The chip includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is configured to run a program or an instruction, so that the processes of the foregoing model accuracy determining method embodiments are implemented and the same technical effects are achieved. To avoid repetition, details are not described herein again.
It should be understood that the chip mentioned in this embodiment of this application may also be referred to as system on chip, system-on-chip, System on Chip, SoC, or the like.
An embodiment of this application further provides a computer program/program product. The computer program/program product is stored in a non-transitory storage medium. The computer program/program product is executed by at least one processor, so that the processes of the foregoing model accuracy determining method embodiments are implemented and the same technical effects are achieved. To avoid repetition, details are not described herein again.
An embodiment of this application further provides a system for determining model accuracy, including: a network-side device, where the network-side device includes a first network element, a second network element, and a third network element, the first network element may be configured to perform the operations of the method for determining model accuracy as described above, the second network element may be configured to perform the operations of the method for determining model accuracy as described above, and the third network element may be configured to perform the operations of the model accuracy determining method as described above.
It should be noted that in this specification, the terms “include”, “comprise”, and any variation thereof are intended to cover a non-exclusive inclusion. Therefore, a process, a method, a target, or an apparatus that includes a series of elements not only includes such elements, but also includes other elements not specified expressly, or may include inherent elements of the process, the method, the target, or the apparatus. Without more limitations, an element defined by the phrase “including one” does not exclude a case in which there are still other same elements in a process, a method, a target, or an apparatus including the element. Further, it should be noted that the scope of the method and apparatus in embodiments of this application is not limited to the shown or discussed orders in which the functions are implemented. Alternatively, depending on the functions, the functions maybe performed in a substantially simultaneous manner or in a reverse order. For example, the described method may be performed in an order different from the described order, and various steps may be added, omitted, or combined. In addition, features described with reference to some examples may be combined in other examples.
According to the descriptions in the foregoing implementations, a person skilled in the art may clearly learn that the method according to the foregoing embodiments may be implemented by relying on software and a necessary common hardware platform or by using hardware. However, in most cases, the former is more preferred. Based on such an understanding, the technical solutions of this application essentially, or the part contributing to the conventional technology, may be presented in a form of a computer software product. The computer software product is stored in a non-transitory storage medium (for example, a read only memory (ROM)/random-access memory (RAM), a magnetic disk, or an optical disc), including several instructions to enable a terminal (which may be a mobile phone, a computer, a server, an air conditioner, a network device, or the like) to perform the methods described in embodiments of this application.
Embodiments of this application are described above with reference to the accompanying drawings. This application is not limited to the embodiments described above, and the embodiments described above are merely examples but not limitations. A person of ordinary skill in the art may make various variations under the teaching of this application without departing from the purpose of this application and the protection scope of the claims, and such variations shall all fall within the protection scope of this application.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202210225777.2 | Mar 2022 | CN | national |
This application is a Bypass Continuation Application of International Patent Application No. PCT/CN2023/080030, filed Mar. 7, 2023, and claims priority to Chinese Patent Application No. 202210225777.2, filed Mar. 7, 2022, the disclosures of which are hereby incorporated by reference in their entireties.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/CN2023/080030 | Mar 2023 | WO |
| Child | 18826622 | US |
