Patent Application
20190240842
This application claims priority to Taiwanese Invention Patent Application No. 107104579 filed on Feb. 8, 2018, the contents of which are incorporated by reference herein.
The present invention relates to a robot assisted interactive system and method of using the same, and more particularly to a system and method for operating a mobile device with interactive speech and gesture assistance from a robot.
Mobile devices such as cell phones and tablets greatly increase the convenience of people's lives. A wide range of applications is currently available for mobile devices to replace various traditional tools. For instance, there are currently many types of educational applications that utilize the multimedia functionalities of the mobile devices, turning them into powerful and effective tools for children education. In the meantime, there are new breeds of commercial robots (such as Pepper) capable of assisting and enhancing children learning experience. However, children may suffer from attention loss or frustration in the learning process due to lack of interaction or flexibility (e.g., monologue lecturing or rigid setting of content difficulties). Existing mobile devices or robots often does not provide interactive content adjustments according to children's real-time in-lesson conditions (e.g., emotional condition), resulting in a less than satisfactory learning experience.
Accordingly, there is a need for a more effective method to enhance the level of interaction with the learning applications provided by the mobile devices to increase learning effectiveness for children.
In view of this, one aspect of the present disclosure provides a robot assisted interactive system and method of using the same. The instantly disclosed robot assisted interactive system and method thereof facilitate deeper interaction between a user and the content of a mobile device (e.g., health & educational tutorial/material) with the help of a robot. By way of example, the robot assisted interactive system and method thereof of the present disclosure utilizes a robot or a mobile device to obtain a user's reaction feedback, and determine the state of the user through processing the feedback information by a backend server. The backend server may then generate a updated content material accordingly, and feed the updated content back to the mobile device and the robot, so as to enhance the user's interaction experience and thus achieve better education and learning results.
Accordingly, embodiments of the instant disclosure provides a robot assisted interactive system that comprises a robot, a mobile device, and a backend server. The mobile device includes a display unit, a touch control unit, a camera unit, a communication unit, and a processing unit. The display unit is configured to display a visual content. The touch control unit is configured to receive user input. The camera unit is configured to obtain user reaction information. The communication unit is configured to establish data connection with the backend server and transmit the use reaction information thereto. The processing unit is coupled to and configured to control the display unit, the touch control unit, the camera unit, and the communication unit. The robot includes a gesture module, a speech module, a communication module, and a control module. The gesture module is configured to generate a gesture output according to the visual content. The speech module is configured to generate a speech output according to the visual content. The communication module is configured to establish data connection with the backend server. The control module is coupled to and configured to control the gesture module, the speech module, and the communication module. The backend server is configured to generate a feedback signal in accordance with the reaction information and send the feedback signal to the mobile device; so as to cause the mobile device to update the visual content based on the feedback signal, and cause the robot to generate updated gesture output and speech output based on the updated visual content.
Embodiments of the instant disclosure further provides a method of robot assisted interaction using a robot assisted interaction system that includes a robot, a mobile device, and a backend server. The method comprises the following processes: establishing data connection between the mobile device and the robot; displaying, on the mobile device, a first visual content, wherein the robot generates a first speech output and a first gesture output based on the first visual content; determining, by the mobile device, a receipt of a user input signal; upon receipt of the user input signal by the mobile device, displaying on the mobile device a second visual content based on the user input signal, wherein the robot generates a second speech output and a second gesture output based on the second visual content; obtaining a use reaction information; sending the user reaction information to the backend server; generating a feedback signal, by the backend server, based on the user reaction information, and sending the feedback signal to the mobile device; and displaying, on the mobile device, a third visual content, wherein the robot generates a third speech output and a third gesture output based on the third visual content.
Accordingly, the instant disclosure provides a robot assisted interactive system and method of facilitating deeper interaction between a user and the content of a mobile device (e.g., health & educational tutorial/material) through the utilization of a robot. The robot assisted interactive system and method thereof of the present disclosure uses a robot or a mobile device to obtain a user's reaction feedback, and determine the state of the user through processing the feedback information by a backend server and generate a updated content material accordingly, and feed the updated content back to the user interface (e.g., the mobile device or the robot), so as to enhance the user's interaction experience and achieve better education and learning results.
Many aspects of the disclosure can be better understood with reference to the following drawings. The components in the drawings are not necessarily drawn to scale, the emphasis instead being placed upon clearly illustrating the principles of the disclosure. Moreover, in the drawings, like reference numerals designate corresponding parts throughout the several views.
Embodiments of the instant disclosure will be specifically described below with reference to the accompanying drawings. Like elements may be denoted with like reference numerals.
Please refer to
Referring to
In one embodiment, the robot 100 may ascertain its global latitude and longitude positions through an assisted global positioning system (A-GPS) or a global navigation satellite system (GLONASS). In the positioning of the regional space, the user may obtain an orientation status of the robot 100 through a digital compass. Likewise, an angle of the deflection of the robot 100 can be obtained by using an electronic gyroscope or an accelerometer. In some embodiments, a microphone array and ambient light sensor may be integrated as robot sensors to obtain relative motion/position of environmental objects around the robot 100, so can a charge-coupled device be used to obtain a two dimensional array of digital signal. The control module 110 of the robot 100 can perform three-dimensional positioning by using the spatial features of the acquired two-dimensional array of digital signals.
Referring to
Referring to
In one embodiment, the blood pressure, heartbeat, and blood oxygen data of the user U measured by the wearable device 400 can be associated with the physiological state of the user during a game. The level of concentration of the user may be determined through analyzing the various physiological data using applicable algorithms. According to studies, when a user is distracted, the number of heartbeats will rise, blood pressure will also increase, and the amount of blood oxygen in the brain will decrease. Conversely, if a user's concentration is high, the number of heartbeats will decrease, the blood pressure will maintain at a calm state, and blood oxygen level in the brain will increase. Through suitable analytical algorithm(s), the three types of physiological signals can be used to determine the degree of concentration of the user's when answering questions (as well as the stress level of a user in association with the difficulty level of the questions). Accordingly, the system may use the analytical results to interactively adjust the difficulty level of subsequent question, so as to suit the user's current condition to retain the attention thereof (e.g., by reducing the level of frustration thereof; after all, some educational programs are more effective by learning through playing).
In some embodiments, as shown in
Please refer to
In process S502, the mobile device 200 displays a first visual content; the robot 100 provides a first speech output and a first gesture output according to the first visual content. For example, after the user connects the mobile device 200 to the robot 100, the application (APP) is initiated, and the first visual content on the mobile device 200 may show a start screen that includes a welcome message for the user. Concurrently, the robot 100 can perform a hand waving motion as the first gesture output, and simultaneously call out the user name (or nickname) as first speech output. It can further generate a reminder for the user to click on the mobile device 200 to start interaction. Accordingly, the motion gesture and speech output of the robot 100 can be synced and matched to the visual output displayed on the mobile device 20, thereby increasing the level of user immersion during the interaction process. This would help to increase a user's fun factor when interacting with the mobile device 200 and retain user's attention for continued interaction.
In process S503, the mobile device 200 determines whether a user input signal is received. For example, after the user launches the APP, the user may carry out interactive functions of the application through touch control. The processing unit 210 of the mobile device 200 may determine whether the touch unit 222 has received a touch operation from a user. When the determination in S503 is NO, that is, if the mobile device 200 does not receive a user input signal, the process returns to S502 to remind the user to interact with the mobile device 200. On the other hand, when the determination in S503 is YES, that is, the mobile device 200 receives a user input signal, the process proceeds to S504. At this time, the user may start to interact with the mobile device 200 by using the touch control interface 220. For example, the user may touch the option icons provided on the user interface 220.
In process S504, the mobile device 200 may display a second visual content based on the user input signal. Meanwhile, the robot 100 would provide a second speech output and a second gesture output in association with the second visual content. For example, in a scenario where the user selects a right answer when using the APP, the mobile device 200 may display a smiling face as a second visual content. Concurrently, the robot 100 may generate a speech output “right answer, congratulations!” and perform a clapping motion with its arms raised over its head as the second speech and gesture outputs. In another scenario where the user selects a wrong answer, the mobile device 200 may display a crying face as a second visual content. Accordingly, the robot 100 may generate a speech out “pity, wrong answer!” and perform a head shaking gesture with its hands down as the second speech and gesture outputs.
In process S505, the robot assisted system 10 obtains a user reaction information. The user reaction information may include user image information and user voice information. By way of example, the mobile device 200 may acquire user image information and user voice information through its onboard sensors. Alternatively, the robot 100 may be used to acquire user image information and the user voice information. For example, when a user learns that he or she selects a right answer, they would naturally put out a happy facial expression and sounds. Either one of the mobile device 200 or the robot 100 may record the user visual and sound expressions as user response information. Conversely, when a user learns that his/her answer is wrong, he/she would inevitably put out frustrated facial expression and sound. The mobile device 200 or the robot 100 would likewise record the depressed expression and sound of the user as user response information. In certain cases, the user may begin losing interest in the visual content shown on the mobile device 100, and starts to appear absent-minded. In turn, at least one of the mobile device 200 or the robot 100 may record the performance of the user's absent-mindedness as user response information. In process S506, the mobile device 200 or the robot 100 transmits the user response information to the backend server 300. In some embodiments, the robot assisted system 10 may further include a wearable device 400. Process S505 may further include: acquiring user physiological information by the wearable device 400 and transmitting the user physiological information to the backend server 300. In one embodiment, the wearable device 400 is an electroencephalogram detection and analysis device.
In process S507, the backend server 300 generates a feedback signal according to the user reaction information, and transmits the feedback signal to the mobile device 200. Specifically, after the backend server 300 obtains the user response information, the voice processing unit 311 and the image processing unit 312 in the backend server 300 analyze the user voice information and the user visual information to determine the status of the user. In addition, the backend server 300 generates the feedback signal according to the state of the user, and transmits the feedback signal to the mobile device 200. For example, when the backend server 300 determines that the user is in a happy state, the backend server 300 generates a feedback signal with enhanced difficulty and transmits the feedback signal to the mobile device 200. Alternatively, when the backend server 300 determines that the user is in a frustrated state, the backend server 300 may generate a feedback signal with reduced difficulty to transmit to the mobile device 200. Alternatively, when the backend server 300 determines that the user is in an absent-minded state, the backend server 300 may generate a feedback signal that attracts the user's attention and transmits the feedback signal to the mobile device 200. In another embodiment, the processing unit 210 of the mobile device 200 generates a feedback signal according to the user response information; that is, the mobile device 200 can directly analyze the user response information and generate a feedback signal. Process S507 further includes: the backend server 300 generates the feedback signal according to the user physiological information. In addition to determining the state of the user by using the user response information, the backend server 300 can further determine the state of the user by using physiological information of the user (for example, blood pressure, heartbeat, or brain wave). In one embodiment, the backend server 300 uses the brainwaves of the user to analyze the degree of concentration of the user.
In process S508, the mobile device 200 displays a third visual content according to the feedback signal. Also, the robot 100 provides a third speech output and a third gesture output according to the third image content. For example, when the backend server 300 generates a feedback signal indicating a need for increased difficulty to the mobile device 200, the mobile device 200 would correspondingly display content with higher difficulty as the third image content. Accordingly, the robot may generate a message “this question is tough, keep it up!” with a corresponding cheering motion as the third speech and gesture outputs. Alternatively, when the backend server 300 generates a feedback signal indicating a need for reduced difficulty and transmits the feedback signal to the mobile device 200, the mobile device 200 would in turn display a lower difficulty content as the third image content. According, the robot may generate a speech output “take it easy; let's try again” in conjunction with a cheering motion as the third speech and gesture outputs. Alternatively, when the backend server 300 generates a feedback signal that indicates a need for attracting the user's attention and transmits the feedback signal to the mobile device 200, the mobile device 200 may display a content designed to attract the user's attention as the third image content. For instance, the robot 100 may start to perform singing and dancing as the third speech and gesture outputs to regain the user's attention.
Accordingly, the robot assisted interaction method S500 of the instant disclosure utilizes a robot to guide the user interaction with the content provided by a mobile device (e.g., such as medical/health education). Moreover, the robot assisted interaction method S500 of the present disclosure utilizes a robot or a mobile device to obtain the user's reaction, and determines an emotional/mental state of the user through the processing power of the backend server. Such interactive scheme may generate better subsequent content or response for the system user through the provision of better, more suitable, and more interactive feedback contents, thereby achieving better learning results.
In addition to the above-described embodiments, additional embodiments will be described below to illustrate the robot assisted interaction method of the present disclosure. In one embodiment, when user beings using the robot assisted interactive system, the robot may automatically identify the user and obtain the user's usage record and usage progress. Accordingly, during subsequent voice interaction, the robot may directly call out the user's name (or nickname). For user identification, the aforementioned robot can recognize the user's identity through the facial features of the user. Alternatively, the aforementioned robot can access the data in the mobile device, the wearable device worn by the user, or the RFID to obtain the corresponding identification code, and then query the database of the backend server to obtain user information. If user information cannot be found in the database of the backend server, the robot can ask the user whether to create new user information. In addition, third parties, such as the hospital's Hospital Information System (HIS) or the Nursing Information System (NIS), may provide the user information. Alternatively, the user information may be obtained from the default user information of the mobile device.
In one embodiment, the robot can possess face recognition capability. The face database can be provided by a third party, such as a hospital's HIS system or NIS system. Alternatively, the face database may be preset in the memory unit of the mobile device or in the database of the backend server. The face recognition method of the robot may be determined based on the relative position information between a user's facial contour and part of a facial characteristic feature. The characteristic positions may include the eyes, the tail of the eyes, the nose, the nose, the mouth, the sides of the lips, the middle, the chin, the cheekbones and the like. In addition, after each successful facial recognition process, the face information of the user in the face database may be updated or corrected to address possible gradual changes in the user's body (e.g., gaining/losing weight).
In one embodiment, after user identification is completed, the mobile device establishes connection to the backend server to receive the user information and load the APP of the mobile device (e.g., a health education APP) to setup a learning progress (and to adjust a starting material). If the user is a first time user for the health education APP, the APP will set the user's attributes from an initial stage. The APP has several built-in educational modules and small games of different topics but within a common subject. Each time a user completes a health education module, the mobile device uploads the user's performance result to the aforementioned back end server. The server may accordingly record the user's learning result in a so-called fixed upload mode.
In one embodiment, when performing face recognition, an image of the user's eye portion is obtained for determining whether the user is tired, e.g., by using observable signs from the image as such whether the user's eyelid is down, the eye-closing time is increasing, or the eyes turn red, and the like. When the system determines that the user is feeling exhausted, the robot may ask the user whether to continue or advise the user to take a rest.
In another embodiment, a user can be equipped with a brain wave measuring device, and the brain wave measuring device can establish data connection with the mobile device and/or the robot. The current physiological status of the user may thus be obtained by determining the change in the users' brainwave. The brain wave measuring device measures the brain wave (also called Electroencephalography, EEG) of the user through contact of the electrodes on the scalp (such as the forehead). EEG can be used to determine a user's concentration level. Moreover, a user's attention can be captured/concentrated through playback of appropriate music. Generally, the frequency of brain waves can be classified (from low to high) into δ wave (0.5˜4 Hz), θ wave (4˜7 Hz), a wave (8˜13 Hz), β wave (14˜30 Hz). The a wave represents a person in a stable and most concentrated state; the β wave indicates that the user may be nervous, anxious or excited, and uneasy; the θ wave (4˜7 Hz) represents the user's in a sleepy state. In general, θ and δ waves are rarely detected in adults when they are awake or attentive. Therefore, the brain wave measuring device can be used to judge the user's state of concentration based on the detected alpha wave, theta wave and the delta wave, and such brainwave information may then be used to adjust the content of the APP in a timely manner to capture/ regain the attention of a user.
In an embodiment, taking the medical education APP as an example, this type of application is presented in a sequential simple graphical/textual question and answer (QA) manner. The QA bank/database is preloaded in the application and can be downloaded from the backend server via automatic update. When a user starts the educational APP, the type of question offered to the user may be determined by the user information obtained at the time of connection. The APP may adjust difficulty level of the questions based on the result of the user's previous answers. In this embodiment, during the interaction with the application, the mobile device and the aforementioned robot use the camera unit and the camera module to capture image information of the user's facial expression at a fixed frequency (for example, five times per second). The image information is then transmitted to the aforementioned backend server for analysis. In addition, brain wave detection and analysis device can be used to generate analytic data regarding the user's concentration level in combination with the user's image and voice information, as well as taking into account the recorded input rate of the user. The above measured information may then be integrated and transmitted to the aforementioned back-end server. The backend server uses the above information analysis to analyze the user's emotional condition and generate a continuous emotion-condition distribution curve, and transmits the result in a feedback signal back to the mobile device. Accordingly, the mobile device adjusts and selects the content of the next educational game according to the feedback signal. This feedback adjustment process may be repeated during the game progression to ensure that the provided content is better received by the user, i.e., enhancing the user's learning quality by retaining the user's attention.
In one embodiment, taking the medical education APPs for example, to accompany the teaching material displayed by the mobile device, the robot may simulate/emulate the gesture of a lecturing instructor, in order to assist a user in understanding the content of the educational material. Meanwhile, the camera module of the robot or the camera unit of the mobile device can be used to capture the user's mood and concentration state in real time, so that the robot can respond accordingly. Moreover, the mobile device transmits the emotion and concentration state of the user to the backend server as the basis for the subsequent content provision. If it is determined that the user is unable to concentrate, the robot may perform a dynamic motion, or provide a joke with singing/dancing gestures to capture the user's attention. In addition, the robot-assisted interactive system of the present disclosure also supports broadcast of the educational video and the video conferencing functionality that enables real-time communication between medical staffs and a user.
In one embodiment, the mobile device transmits the user's usage data to the backend server. After the back-end server accumulates sufficient usage data from different users, a user mode analysis can be performed and used as a basis for adaptive content adjustment. Taking medical education APP as an example, the mobile device may adjust the question selection from its preset bank based on the user information (e.g., age of the user) and the selected mode of the response. The adaptive content selection of the in-game question bank may be performed by the back-end server by analyzing the user's voice recognition, face recognition, emotion recognition, and brain wave detection data, and correspondingly generate a feedback signal. The mobile device and the aforementioned robot may correspondingly perform content selection based on the feedback signal, so as to prepare the next phase of interactive content. In addition, through online update, the backend server can also update the question displayed on the mobile device, as well as the voice and gesture of the robot. The back-end server may collect user response information as a basis for adjusting subsequent question/content, as well as for user mode/preference analysis. The above-mentioned medical education APP may also provide offline mode, which may be used to help users familiarize with the operation/functionality of the APP and provide real-time data query.
Please refer to
When the user U executes the application 660 through the handheld device 600, the robot 100 may guide the user U to complete the interactive task of the application 660 using voice or gesture output. Reference can be made to the foregoing embodiments (the details may be referred to previous embodiment). The robot 100 may determine the state of the user U using onboard sensing devices, e.g., built-in microphone (such as the voice module 150 shown in
After the user finishes a session, the processor 610 may record the usage status information and transmit the recorded data to the backend server 300 for future reference of the application 660. For instance, each application 660 executes a plurality of different types of units, and the aforementioned processor 610 or the aforementioned backend server 300 may changes the order of the different types of units according to the user's past usage records.
In one embodiment, when the robot 100 determines that the current state of the user is unfocused, inattentive, or distracted, the robot 100 may transmit user status information to the handheld device. The handheld device 600 may correspondingly stop the application or change the content of the subsequent application in a timely manner, or remind the user to take a break (e.g., followed the gesture of the robot 100 to perform stretching exercises). In some embodiments, the robot 100 may generate message to remind the user that the usage time is too long, recommend the user to take a break and rest his/her eyes. In the mean time, the application 660 on the handheld device 600 may temporarily pause its functionality for a predetermined period of time, or play a relaxing picture, music, or video.
In another embodiment, the user's usage status or operation record is gathered by the handheld device 600 and transmitted back to the backend server 300 for real-time or further adjustment of the displayed content of the interactive game or education software. Alternatively, the user's usage status or operation record can also be directly stored in the handheld device 600, and the settings can be linked to the corresponding application.
In one embodiment, the robot 100 may continuously capture a user's image through a built-in camera (for example, the camera module 140 shown in
In one embodiment, when the user executes the application 660 of the handheld device 600 (e.g., a smart phone or a tablet), the application 660 may first generates a QR code. The robot 100 may perform authentication through scanning the QR code before establishing data connection with the handheld device 600. After the connection is completed, the handheld device 600 continues to execute the application 660. In another embodiment, the application 660 may prompt the user to log in, thereby accessing/recording the user's usage record. The login method may be performed through key pad input of the user's account password, through the fingerprint scanner on the handheld device, through performing face recognition through the handheld device 600 (or the image acquired by the camera device of the robot 100), or through the wearable device (such as a wristband or a Bluetooth watch) on the user.
When the user is operating the application 660, the robot 100 may respond correspondingly according to the instruction or the state of the user sent by the handheld device 100. For example, when the user completes a level of the application 660 or meets a predetermined condition, the handheld device 600 transmits a control signal to the robot 100. The robot 100 may generate a corresponding response, such as praising the user, performing a dance/singing gesture, playing a movie, and the like. The manner in which the robot 100 interacts with the handheld device 600 is described as follows.
A first scenario is when the application 660 is executed, the robot 100 takes instruction from the handheld device 600. The handheld device 600 determines whether to activate the robot 100 according to the usage status of the user. When the handheld device 600 determines to activate the robot 100, the handheld device 600 transmits detailed instructions and contents of the actions that the robot 100. Subsequently, the robot 100 executes an instruction to perform a corresponding operation. An advantage of this approach is that the robot 100 does not require the installation of self-automation programs, and is fully controlled by the handheld device 600. For example, when the application 660 is executed, the handheld device 600 can issue a voice command and text content, so that the robot 100 can play the text content by voice to guide the user to operate the application 660. In another case, in order to allow the robot 100 to perform an action correctly and avoid the data loss or corruption during real-time transmission, the handheld device 600 may package the instruction and the content into an executable file. When the robot 100 it may execute the executable upon receipt of the complete file. In one embodiment, the executable file is compatible with the operating system of the robot.
In the previous scenario, the robot 100 may require no pre-installed application or associated software, and is passively controlled by the handheld device 600. However, in a second scenario, the robot 100 may download and install the same application 660 or an auxiliary application associated with the application 660. The auxiliary application may be different from the application 660 installed on the handheld device 600 and cannot be used alone. In another embodiment, the auxiliary application is not available in the public app store, such as Apple's App store.
In one embodiment, the auxiliary application will confirm whether robot 100 is a compatible robot before installation. If the robot 100 not a compatible model, the auxiliary application will not be downloaded for installation. In another embodiment, the auxiliary application is to be installed on the robot 100 through an application on the handheld device 600. Specifically, the handheld device 600 first establishes connection with the robot 100, and the application on the handheld device 600 determines whether the connected robot is supported thereby. When the robot is determined to be compatible, the handheld device 600 transmits the auxiliary application to the robot 100. Alternatively, the handheld device 600 may transmit the download link of the auxiliary application to the robot 100, and a user may operate the user interface on the robot 100 to perform download operation.
In another embodiment, the auxiliary application includes control commands that correspond to several different types of robots, such that a single auxiliary application can enable the handheld device 600 to control different robots. Moreover, when the handheld device 600 is connected to the robot 100, the handheld device 600 can recognize the type of robot connected.
In another embodiment, the auxiliary application includes instruction conversion function. For example, the instruction sent by the application 660 on the handheld device 600 may be “mov fwd 10)” (move forward 10 steps). The auxiliary application may convert the application's instruction to a command that is readable by the particular type of robot 100 (for example, 0xf1h 10). The converted instruction code will then be transmitted to the robot 100 for execution.
The embodiments shown and described above are only examples. Many details are often found in this field of art thus many such details are neither shown nor described. Even though numerous characteristics and advantages of the present technology have been set forth in the foregoing description, together with details of the structure and function of the present disclosure, the disclosure is illustrative only, and changes may be made in the detail, especially in matters of shape, size, and arrangement of the parts within the principles of the present disclosure, up to and including the full extent established by the broad general meaning of the terms used in the claims. It will therefore be appreciated that the embodiments described above may be modified within the scope of the claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 107104579 | Feb 2018 | TW | national |
