Patent Application
20250217357
This application claims the benefit of Taiwan application Serial No. 113100294, filed Jan. 3, 2024, the disclosure of which is incorporated by reference herein in its entirety.
The disclosure relates in general to an instruction query method, a computer program product and an associated query system, and more particularly to an instruction query method, a computer program product and an associated query system which respond to the user in the user's familiar language.
Recently, the use of artificial intelligence (AI) for information query becomes increasing popular. Through generative pre-trained transformers (GPT), answers can be quickly obtained even though it used to take much time to search for related data. For example, ChatGPT can be trained with a large amount of data collected from the Internet by utilizing deep learning techniques, and thus can have conversations with users. Another example is that Microsoft Copilot can help users in dealing with relatively general problems or problems of upper-layer application software. For instance, Copilot in Outlook can draft emails, and Copilot in Word can generate copywritings.
The existing AI tools, Including ChatGPT and Copilot, developed based on large language models (LLM) are set in the cloud. For users, this means that the query efficiency depends on the network bandwidth. Furthermore, the current AI tools are associated with broad but rough data sources. If a user wants to ask more specific questions, these AI tools are still unable to accurately provide or respond with more accurate information.
At present, many electronic products have more and more functions. The users are likely to encounter unfamiliar or unknown functions and have no knowledge of how to start or set the functions. Unfortunately, the current AI tools can only reply with relatively vague messages. When a user asks for operating instructions related to the electronic product close at hand, the current AI tools may even cause hallucination. Therefore, the current AI tools still cannot help users to quickly and accurately obtain information for operating or setting the electronic products.
The disclosure is directed to an instruction query method, a computer program product and an associated query system. The instruction query method applied to the query system involves functions of language translation and interaction, and improves the interactivity based on a user manual in English version of the electronic device, thus helping the user in querying the operating instructions related to the electronic device.
According to one embodiment, an instruction query method is provided. The instruction query method includes the following steps. At first, a language translator translates a non-English query string into an English query string according to a language identification code. Then, a prompt tuning module transforms the English query string into an English interactive prompt string according to at least one high-relevance token vector. The at least one high-relevance token vector relates to operating instructions of an electronic device. Afterward, the language translator translates an English reply string into a non-English reply string according to the language identification code. The English reply string is generated based on the inferences made from the English interactive prompt string.
According to another embodiment, a computer program product storing therein a software program is provided. The software program performs an instruction query method, including the follow steps. At first, a non-English query string is translated into an English query string according to a language identification code. Then, the English query string is transformed into an English interactive prompt string according to at least one high-relevance token vector. The at least one high-relevance token vector relates to operating instructions of an electronic device. Afterward, an English reply string is translated into a non-English reply string according to the language identification code. The English reply string is generated based on the inferences made from the English interactive prompt string.
According to an alternative embodiment, an instruction query system is provided. The instruction query system includes a query platform. The query platform includes a language translator and a semantic kernel module. The language translator includes an input translation model and an output translation model. The input translation model translates a non-English query string into an English query string according to a language identification code. The semantic kernel module transforms the English query string into an English interactive prompt string according at least one high-relevance token vector. The at least one high-relevance token vector relates to operating instructions of an electronic device. The output translation model translates an English reply string into a non-English reply string according to the language identification code. The English reply string is generated based on the inferences made from the English interactive prompt string.
In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. It will be apparent, however, that one or more embodiments may be practiced without these specific details. In other instances, well-known structures and devices are schematically shown in order to simplify the drawing.
The present disclosure provides an instruction query system, as shown in
Please refer to
The manufacturer of a specific electronic device usually provides an English user manual 18a to assist users in operating the electronic device. The English user manual 18a fully explains the instructions and the usage/operation of the electronic device. Before the electronic device leaves the factory, the manufacturer uses the semantic parser 18c to perform parsing (syntax analysis) on the content of the English user manual 18a (including various operating instructions related to the electronic device) in advance.
After the parsing performed by the semantic parser 18c, the parsing result is stored as JavaScript Object Notation (JSON) files (Jfile) 18e. Afterward, the word vector model 18g performs vector calculation and reformatting for the JSON files (Jfile) 18e to generate pre-stored token vectors TOKvec and stores them in a vector database 131. The query platform 10 may be completely built into the electronic device, completely set on a server in signal communication with the electronic device, or provided by the electronic device and a server together.
The query platform 10 further includes a language translator 11, a semantic kernel module 13, and a pre-trained transformer 15. The semantic kernel module 13 further includes a vector database 131 and a prompt tuning module 133.
The vector database 131 is configured to enhance the storage efficiency and the search for the pre-stored token vectors TOKvec. The pre-stored token vectors TOKvec stored in the vector database 131 are high-dimensional vectors. Each pre-stored token vector TOKvec represents a characteristic or attribute of a piece of data in the English user manual 18a. Before the pre-stored token vectors TOKvec are stored in the vector database 131, the data in the JSON files (Jfile) 18e are processed based on the vector format. That is, after the semantic parser 18c generates the JSON files (Jfile) 18e, the word vector model 18g is needed to transform the tokens in the JSON files (Jfile) 18e into the pre-stored token vectors TOKvec.
The semantic parser 18c includes a tokenizer, an English lexical database, a small language model created by artificial neural networks, etc. The tokenizer is used for splitting texts of the English user manual 18a into smaller and more understandable tokens.
The details of how the semantic parser 18c performs sentence decomposition and token splitting and how the word vector model 18g performs data vectorization will not be described herein. The processes of token splitting and data vectorization for non-English languages are usually much more complicated than those processes for English. Therefore, the query platform 10 of the present disclosure further includes the language translator 11 so that the vector database 131 can just consider the pre-stored token vectors TOKvec based on English. Accordingly, the semantic parser 18c and the word vector model 18g of the present disclosure perform the parsing and data vectorization only on the English user manual 18a, but not other language versions of the user manual. Thus, the semantic parser 18c and the word vector model 18g of the present disclosure can have a much higher processing speed.
The vector database 131 records the word embeddings (vector values) of the pre-stored token vectors TOKvec, the content of the JSON files (Jfile) 18e, and the token number of the pre-stored token vectors TOKvec. The present disclosure does not specify the operation of the word vector model 18g and the creation of the vector database 131, including how to create a vector index, how to determine fields in the vector index, vector data types, algorithms for calculating similarity between vectors, and so on. Briefly, after the data vectorization performed by the word vector model 18g, the pre-stored token vectors TOKvec stored in the vector database 131 become structured data, which are appropriate for the vector database 131 to perform similarity search.
Before the electronic device leaves the factory, the pre-stored token vectors TOKvec have been generated by the manufacturer and stored in the vector database 131. The vector database 131 could be set in the cloud or a storage module of the electronic device. Considering the storage space, the pre-stored token vectors TOKvec stored in the vector database 131 may be associated with only one version of the English user manual 18a. Alternatively, considering the data completeness, the pre-stored token vectors TOKvec stored in the vector database 131 may be associated with various versions of English user manuals 18a. This variation can be adopted optionally to meet various requirements.
When the user operates the electronic device and has trouble using an unfamiliar function of the electronic device, the user can make queries about the setting steps of operating the electronic device through the query platform 10. Since the electronic device may be sold to any country, the user probably uses a non-English language to make queries about the operating instructions. The disclosure refers to the query content being input by the user as a non-English query string qryTXT(usrLNG).
The language translator 11 of the query platform 10 first translates the non-English query string qryTXT(usrLNG) into an English query string qryTXT(eng), and then transmits the English query string qryTXT(eng) to the prompt tuning module 133, and the vector database 131. It is to be noted that the language translator 11, the prompt tuning module 133, and the pre-trained transformer 15 may adopt different types/numbers of inference accelerators. For example, the inference accelerators could be selected from neural network processing units (NPU), graphics processing units (GPU), field programmable gate arrays (FPGA), and so forth.
According to the English query string qryTXT(eng), the vector database 131 searches for one or more token vectors TOKvec_rel, which are highly related to the English query string qryTXT(eng), among the pre-stored token vectors TOKvec in the vector database 131.
According to concepts of the present disclosure, the vector database 131 can cooperate with a large language model (LLM). After steps of, for example, semantic search, similarity search, and recommendation engine are performed for the pre-stored token vectors TOKvec, at least one high-relevance token vector TOKvec_rel, which is determined more consistent with the queried operating instructions, is found. The one or more high-relevance token vectors TOKvec_rel obtained from the search by the vector database 131 are then transmitted to the prompt tuning module 133.
If the vector database 131 finds more high-relevance token vectors TOKvec_rel from the search, the vector database 131 may further sort the high-relevance token vectors TOKvec_rel according to the relevance levels. Then, the vector database 131 selects several (for example, three) high-relevance token vectors TOKvec_rel to be transmitted to the prompt tuning module 133 according to the sorting result.
In practical applications, the language translator 11 may transmit the non-English query string qryTXT(usrLNG) with metadata related to the model of the electronic device. In this way, after receiving the English query string qryTXT(eng) from the language translator 11, the vector database 131 can find more accurate high-relevance token vector(s) TOKvec_rel according to the metadata related to the model of the electronic device. For example, the queries from an esports gaming PC user (for example, information about adjusting and optimizing computer performance) are usually more complicated than those from a basic PC user (for example, information about word processing). Therefore, if the language translator 11 additionally transmits the metadata in the background, the vector database 131 can provide the high-relevance token vector(s) TOKvec_rel more accurately.
As described above, the prompt tuning module 133 receives the English query string qryTXT(eng) from the language translator 11 and receives the high-relevance token vector(s) TOKvec_rel from the vector database 131. The prompt tuning module 133 is a transformation model that performs natural language processing (NLP) based on machine learning technology. In short, the prompt tuning module 133 transforms the English query string qryTXT(eng) together with the high-relevance token vector(s) TOKvec_rel provided by the vector database 131 into an English interactive prompt string pptTXT(eng), which is more understandable by the pre-trained transformer 15.
The English interactive prompt string pptTXT(eng) could be regarded as the combination of the English query string qryTXT(eng) and the high-relevance token vector(s) TOKvec_rel. As described above, the high-relevance token vector(s) TOKvec_rel is derived from the English user manual 18a of the electronic device. Therefore, when describing a query condition, the English interactive prompt string pptTXT(eng) is obviously closer to the real situation of the electronic device than the English query string qryTXT(eng) obtained from the translation.
In practical applications, the prompt tuning module 133 could transmit the English interactive prompt string pptTXT(eng) with prefix-tuning or prompt-tuning, presented in the form of a question, to the pre-trained transformer 15. The pre-trained transformer 15 could be any known language model utilizing generative artificial intelligence, for example, GPT3.5, GPT-3.5 Turbo, GPT-4, or Llama 2.
The pre-trained transformer 15 makes inferences in response to the English interactive prompt string pptTXT(eng), and then generates an English reply string rTXT(eng) according to the inference result. After the pre-trained transformer 15 transmits the English reply string rTXT(eng) to the language translator 11, the language translator 11 translates the English reply string rTXT(eng) into a non-English reply string rTXT(usrLNG) to be provided to the user.
According to concepts of the present disclosure, the implementation and location of the query platform 10 could be determined according to the functions of the user's electronic device and are not limited herein. For example, when the user's electronic device is a high-end product, it means that the processor of the electronic device is very fast and the storage module has a large capacity. In this condition, the manufacturer can set the entire query platform 10 on the electronic device 20, as shown in
Please refer to
According to concepts of the present disclosure, the way that the non-English query string qryTXT(usrLNG) is input to the query platform 10 is not limited. For example, the user can input by typing, handwriting or voice. Accordingly, depending on the input means for the non-English query string qryTXT(usrLNG), the input device may be a keyboard 203, a touch panel, or a microphone 201. Suppose the non-English query string qryTXT(usrLNG) is input through a touch panel 203 or a microphone 201. In that case, the electronic device 20 may further include a component such as a handwriting recognition (HWR) module or a voice recognition module (not shown). Similarly, the output means for the non-English reply string rTXT(usrLNG) is not limited. For example, the query platform 10 can be displayed with a display 207 or played with a speaker 205. Whether it is displayed with the display 207 or played through the speaker 205, the query platform 10 replies in the user's familiar language usrLNG, by translating the non-English reply string rTXT(usrLNG) in advance to allow the user to realize the reply without the language barrier. It is to be noted that the types of input devices and output devices are not limited to these examples.
In
Suppose the electronic device provided for querying the operating instructions is a low-end product. In that case, it means that the electronic device has a slower processing unit, and the storage module has a limited capacity. In this case, the manufacturer can set a portion or the entirety of the query platform 10 on a remote server. When the user wants to query the operating instructions related to the electronic device, the electronic device accesses the remote server through the network.
Please refer to
In
In practical applications, the manufacturer can set a part of the query platform 10 in the electronic device 30 and the other part of the query platform 10 in the cloud to reduce dependence on the network bandwidth. For example, one of the language translator 11, the semantic kernel module 13 and the pre-trained transformer 15 can be set on the electronic device 30, and the other two can be set on the server 35. Alternatively, two of the language translator 11, the semantic kernel module 13, and the pre-trained transformer 15 can be set on the electronic device 30, and the remaining one can be set on the server 35.
The manufacturer of the electronic devices 20 and 30 can determine the configuration of the query platform 10 based on the user's general operating habits. For example, after considering the execution of necessary or frequently-used software in the electronic device in most instances, if the manufacturer thinks that the available processor loading and storage space of the electronic device cannot support the operation of the query platform 10, the architecture of
It is to be noted that the single server 35 in this embodiment is given for illustrative purposes only. In practical applications, there may be one or more servers 35 used to implement the functions of the query platform 10, and the servers 35 could be colocated in a room or not.
In an embodiment, after considering execution of necessary or frequently-used software in the electronic device in most instances, the manufacturer may think that the available processor loading and storage space of the electronic device 30 can support execution of the language translator 11, but cannot further support execution of the semantic kernel module 13 and/or the pre-trained transformer 15 for performing the querying. In this condition, the language translator 11 can be set on the electronic device 30, and the semantic kernel module 13 and the pre-trained transformer 15 can be set in the cloud. The arrangement of which parts of the query platform 10 should be set on the electronic device 30 and which parts should be set on the server(s) is quite flexible and adjustable, and will not be described in detail herein.
Please refer to
As described above, the pre-stored token vectors TOKvec are created by the semantic parser 18c and the word vector model 18g and then stored in the vector database 131 before the electronic device 20/30 leaves the factory. Therefore, when the user operates the electronic device 20/30, no matter whether the query platform 10 adopts the architecture of
In addition to the vector database 131 and the prompt tuning module 133, the semantic kernel module 13 of the present disclosure could further include a vector data temporary storage module 131a. In short, the vector data temporary storage module 131a is regarded as a storage space for buffering. It is a specific or dedicated storage space for storing the high-relevance token vector(s) TOKvec_rel which is selected from the vector database 131 in response to a previous query.
In this way, when the user queries an operating instruction wherein a similar or related operating instruction has been queried before, the high-relevance token vector(s) TOKvec_rel can be retrieved from the vector data temporary storage module 131a without repetitively searching the vector database 131. For example, if the user has previously queried the resolution setting of the display, the vector data temporary storage module 131a may have recorded a high-relevance token vector TOKvec_rel associated with the string “resolution setting”. Later, if the user wants to query the “dual monitors setting”, the vector data temporary storage module 131a, instead of the vector database 131, can provide this recorded high-relevance token vector TOKvec_rel associated with the “display setting”. The mechanism of optimizing or accelerating the search is not limited.
As shown in
After the user inputs the non-English query string qryTXT(usrLNG) through the input device, the non-English query string qryTXT(usrLNG) is transmitted to the language identifier 111 and the input translation model intrMDL 113a. The language identifier 111 is configured to identify and determine the user language usrLNG corresponding to the non-English query string qryTXT(usrLNG), and generate the language identification code usrLNG_ID accordingly. Then, the language identifier 111 transmits the language identification code usrLNG_ID to the input translation model intrMDL 113a and the output translation model otrMDL 113c.
The input translation model intrMDL 113a receives the non-English query string qryTXT(usrLNG) through the input device, and receives the language identification code usrLNG_ID from the language identifier 111. The input translation model intrMDL 113a translates the non-English query string qryTXT(usrLNG) into the English query string qryTXT(eng) according to the language identification code usrLNG_ID.
The input translation model intrMDL 113a transmits the English query string qryTXT(eng) to the vector data temporary storage module 131a and/or the vector database 131. If the vector data temporary storage module 131a has stored the high-relevance token vector(s) TOKvec_rel associated with the English query string qryTXT(eng), the high-relevance token vector(s) TOKvec_rel is provided by the vector data temporary storage module 131a. On the contrary, if the vector data temporary storage module 131a does not store any high-relevance token vector TOKvec_rel associated with the English query string qryTXT(eng), the input translation model intrMDL 113a and/or the vector data temporary storage module 131a transmits the English query string qryTXT(eng) to the vector database 131, which then determines the high-relevance token vector(s) TOKvec_rel according to the English query string qryTXT(eng).
Subsequently, the prompt tuning module 133 transforms the English query string qryTXT(eng) to the English interactive prompt string pptTXT(eng) based on the high-relevance token vector(s) TOKvec_rel. The English interactive prompt string pptTXT(eng) is more understandable by the pre-trained transformer 15. Afterward, the pre-trained transformer 15 makes inferences from the English interactive prompt string pptTXT(eng) to generate the English reply string rTXT(eng), and transmits the English reply string rTXT(eng) to the output translation model otrMDL 113c.
The output translation model otrMDL 113c receives the English reply string rTXT(eng) from the pre-trained transformer 15, and receives the language identification code usrLNG_ID from the language identifier 111. The output translation model otrMDL 113c translates the English reply string rTXT(eng) into the non-English reply string rTXT(usrLNG) according to the language identification code usrLNG_ID.
In practical applications, the language identifier 111 is optional. For example, the electronic device 20/30 provides a language menu to allow the user to select a preferred user language usrLNG. Alternatively, the language of the operating system running in the electronic device 20/30 is viewed as the default user language usrLNG. For these cases, the language identification code usrLNG_ID is a default value and need not be judged by the language identifier 111. Therefore, in
Incidentally, if the user language usrLNG is English, the language translator 11 of the query platform 10 is not required in the query procedure. For example, the string being input through the input device could be treated as the English query string qryTXT(eng) and bypass the language translator 11. Also, the English reply string rTXT(eng) is directly output through the output device and bypasses the language translator 11. Such modifications of the applications are covered in the present disclosure.
Before the electronic device 20/30 leaves the factory, the manufacturer of the electronic device 20/30 uses the semantic parser 18c and the word vector model 18g to transform the content of the English user manual 18a into the pre-stored token vectors TOKvec in advance. Afterward, the pre-stored token vectors TOKvec are stored in the vector database 131. Depending on the specifications of the electronic device 20/30, the vector database 131 can be built into the electronic device 20 (local end), or set on the server 35. After the electronic device 20/30 leaves the factory, the query about the operating instructions is performed through the query platform 10 (
Next, the query platform 10 determines whether the user wants to query usage/operating instructions (step S303). If not, the method waits and then repeats the step S303. Otherwise, if the user wants to query the operating instructions related to the electronic device 20/30, the query platform 10 receives a non-English query string qryTXT(usrLNG) from the user through an input device (step S305).
After the language identifier 111 and the input translation model intrMDL 113a receive the non-English query string qryTXT(usrLNG) (step S307), the language identifier 111 identifies the user language usrLNG corresponding to the non-English query string qryTXT(usrLNG), and transmits a language identification code usrLNG_ID representing the user language usrLNG to the input translation model intrMDL and the output translation model otrMDL (step S309).
As mentioned above, in some applications, the user language usrLNG can be preset through the electronic device 20/30 and need not be determined by the language identifier 111. Therefore, steps S307 and S309, describing the actions of the language identifier 111, are optional rather than compulsory.
The input translation model intrMDL 113a translates the non-English query string qryTXT(usrLNG) into an English query string qryTXT(eng) based on the language identification code usrLNG_ID (step S311). The English query string qryTXT(eng) is sent to the vector database 131 and the prompt tuning module 133, respectively (step S313a and step S315).
After receiving the English query string qryTXT(eng), the vector database 131 searches among pre-stored token vectors TOKvec previously stored in the vector database 131 according to the English query string qryTXT(eng). After searching among the pre-stored token vectors TOKvec, the vector database 131 retrieves at least one high-relevance token vector TOKvec_rel (step S313c).
The prompt tuning module 133 transforms the English query string qryTXT(eng) into an English interactive prompt string pptTXT(eng) according to the high-relevance token vector(s) TOKvec_rel retrieved by the vector database 131 (step S317). The pre-trained transformer 15 generates an English reply string rTXT(eng) in response to the content of the English interactive prompt string pptTXT(eng) (step S319). The output translation model otrMDL 113c translates the English reply string rTXT(eng) into a non-English reply string rTXT(usrLNG) based on user language usrLNG according to the language identification code usrLNG_ID (step S321). The electronic device 20/30 outputs the content of the non-English reply string rTXT(usrLNG) to the user through voice playback or screen display. Afterward, the query platform 10 determines again whether the user wants to query other usage/operating instructions (step S303).
In conclusion, the instruction query system 1 of the present disclosure can generate the non-English reply string rTXT(usrLNG) in the user's familiar language, which is identified from the non-English query string qryTXT(usrLNG) being input by the user. Hence, the user can always face an operating interface presented with the user's familiar language. Further, the content of the non-English reply string rTXT(usrLNG) generated by the instruction query system 1 is derived from the content of the English user manual 18a. Therefore, the non-English reply string rTXT(usrLNG), which the user reads or listens to, indeed gives direct and concrete operating instructions guiding the usage/operation of the electronic device 20/30.
The instruction query method, according to concepts of the present disclosure, can be implemented by a software program stored in a computer program product or a computer-readable medium. For a user who is not familiar with the usage/operation of the electronic device 20/30, he/she can use his/her familiar language to query the operating instructions. Further, the reply from the query platform 10 is derived from the content of the English user manual 18a related to the electronic device 20/30 so that the replied suggestion will not be vague or inaccurate. On the other hand, for the manufacturer of the electronic device 20/30, there is no need to prepare user manuals in different languages for worldwide users in different countries. It significantly reduces the production costs.
In practical applications, the instruction query method of the present disclosure can be applied to various kinds of electronic devices, such as mobile phones, tablets, desktop computers, and notebook computers. Which electronic device can work with the instruction query method of the present disclosure is just a simple choice of those skilled in the art.
Those of ordinary skill in the art should understand that in the above description, the various logical blocks, modules, circuits, and steps taken as examples can be implemented by electronic hardware, computer software, or a combination thereof. The connections, regardless of being expressed as signal connections, connections, communication, coupling, electrical connections, or the like, represent that signal/data/information exchange or transmission for implementing the logical blocks, modules, circuits, and steps can be achieved through wired electronic signals, wireless electromagnetic signals or optical signals directly or indirectly. The terms used in the description do not limit the connection type of the present disclosure, and altering the connection type will not deviate from the scope of the present disclosure.
It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments. It is intended that the specification and examples be considered as exemplary only, with a true scope of the disclosure being indicated by the following claims and their equivalents.
| Number | Date | Country | Kind |
|---|---|---|---|
| 113100294 | Jan 2024 | TW | national |
