Patent Application
20230324879
This disclosure relates generally to human-machine interaction systems, and more specifically to new and useful systems and methods for using user feedback in human-machine interaction systems.
Typical human-machine interaction systems receive user feedback received via an input device.
The following description of embodiments is not intended to limit the disclosure to these embodiments, but rather to enable any person skilled in the art to make and use the embodiments disclosed herein.
Embodiments herein include systems and methods that use an adaptive behavioral control system (of a human-machine interaction system) that uses a computer-readable computer-behavioral policy to control at least one of an audio output system, a display device, and a robot. The adaptive behavioral control system is constructed to update the computer-behavioral policy in response to information indicating a detected human reaction.
In some embodiments, the computer-behavioral policy specifies a plurality of actions for a specific action type and a weight value for each action. For example, for a “greeting” action type, the policy can include three different greeting action options, each with a corresponding weight value. In some embodiments, each weight value corresponds to a use frequency. In some embodiments, each weight value is a percentage corresponding to how often a particular action should be performed, and the weights of all actions for the action type total 100%. In some embodiments, each weight value is used to generate a weighted random sampling, and actions of a particular type of action are selected based on the weighted random sampling.
An initial computer-behavioral policy is generated, and an action for an action type is performed, the adaptive behavioral control system stores information indicating a human reaction detected in response to performance of the action in association with information identifying the action. In this manner, the adaptive behavioral control system creates a log of all recorded reactions for an action.
In a case where stored information indicating detected human reactions for an action satisfies an update condition, the adaptive behavioral control system updates the weight values for the action and the other actions of the same action type in the computer-behavioral policy. In some embodiments, in a case where stored information indicating detected human reactions for a first action indicates positive user feedback below a threshold value, the adaptive behavioral control system decreases the weight value of the first action. In some embodiments, in a case where stored information indicating detected human reactions for the first action indicates positive user feedback above the threshold value, the adaptive behavioral control system increases the weight value of the first action. In this manner, the adaptive behavioral control system updates weight values of actions in the computer-behavioral policy to elicit positive user feedback by a human interaction participant in response to control of the human-machine interaction system in accordance with the computer-behavioral policy.
In some embodiments, the human-machine interaction system is a robot that includes a mechanical body assembly (e.g., 104d of
In some embodiments, the adaptive behavioral control system is constructed to detect presence of interaction participants and respective locations, and generate a three-dimensional world model of detected locations of detected interaction participants. In some embodiments, during initiation of an interaction with a previously detected interaction participant, the adaptive behavioral control system controls the face assembly to orient the face assembly in a direction of a detected interaction participant by using the previously detected location included in the world model. In some embodiments, the adaptive behavioral control system is constructed to generate a three-dimensional world model of all detected faces, and wherein during initiation of an interaction with a participant associated with a previously detected face, the adaptive behavioral control system controls the face assembly to orient the face assembly in a direction of a detected face by using the previously detected direction included in the world model.
In some embodiments, weight values of the computer-behavioral policy are updated based on information received from at least one external human-machine interaction system.
In some embodiments, a human-machine interaction system bow includes adaptive behavioral control system 110, an interaction sub-system 112, and a human reaction detection sub-system 111, as shown in
In some embodiments, the interaction sub-system 112 includes an audio output sub-system (e.g., 130 of
In some embodiments, the audio output sub-system includes at least one speaker (e.g., 130 of
In some embodiments, the interaction sub-system 112 includes a light emission sub-system that includes an array of lights, and the light emission sub-system is constructed to individually control each light of the light emission sub-system. In some embodiments, the interaction sub-system 112 includes a LED ring that includes an array of lights, and the LED ring is constructed to individually control each light of the LED ring.
In some embodiments, the human reaction detection sub-system in includes an audio capture sub-system (e.g., 120 of
In some embodiments, the audio capture sub-system includes at least one audio capture device. In some embodiments, the audio capture sub-system includes at least one microphone. In some embodiments, the audio capture sub-system includes at least one microphone array (e.g., 120 of
In some embodiments, the video capture sub-system includes at least one video capture device. In some embodiments, the video capture sub-system includes at least one camera (e.g., 101c of
In some embodiments, the human reaction detection sub-system in includes a heat detection sub-system that is constructed to detect body heat of a human. In some embodiments, the human reaction detection sub-system in includes a heat detection sub-system that is constructed to detect body temperature of a human.
In some embodiments, the human reaction detection sub-system in includes a touch sensor.
In some embodiments, the human reaction detection sub-system in includes a piezoelectric pressor sensor.
In some embodiments, the human reaction detection sub-system in includes a capacitive touch sensor.
In some embodiments, the human reaction detection sub-system in includes a resistive touch sensor.
In some embodiments, the human reaction detection sub-system in includes a blood pressure sensor.
In some embodiments, the human reaction detection sub-system in includes a heart rate sensor.
In some embodiments, the human reaction detection sub-111 includes a biometric sensor.
In some embodiments, the human reaction detection sub-system 111 includes an interface (e.g., Bluetooth, NFC, WiFi, and the like) that is constructed to receive data from a remote sensor (e.g., a wearable device, such as, for example, a FitBit and an Apple Watch).
In some embodiments, the audio input processor 121 is constructed to process audio data generated by each of the microphones 120 and provide processed audio data to the adaptive behavioral control system 110. In some embodiments, the adaptive behavioral control system 110 includes the audio input processor 121. In some embodiments, the audio input processor 121 is constructed to process audio data generated by each of the microphones 120 by combining the audio data of each microphone 120 into a combined audio data stream. In some embodiments, the audio input processor 121 is constructed to process audio data generated by each of the microphones 120 by selecting at least one audio data stream of the microphones 120, combining each selected data stream, the combined audio data stream of the selected data streams to the adaptive behavioral control system 110.
In some embodiments, directional information is associated with each microphone 120, and the audio input processor 121 is constructed determine a direction of captured audio based on directional information associated with a microphone 120 whose captured audio has a highest volume. In this manner, the adaptive behavioral control system 110 can use the audio input processor 121 in conjunction with the microphones 120 to determine a location of an interaction participant based on captured audio.
In some embodiments, audio output controller 131 is constructed to control the speakers 130 in accordance with data provided to the audio output controller 131 by the adaptive behavioral control system 110. In some embodiments, the audio output controller 131 is constructed to select one of the plurality of speakers 130 based on information provided by the adaptive behavioral control system 110, and control the selected speaker 130 to emit audio in accordance with data provided to the audio output controller 131 by the adaptive behavioral control system 110.
In some embodiments, directional information is associated with each speaker, and the adaptive behavioral control system 110 controls the audio output controller 131 to emit audio by using a speaker 130 that matches directional information provided by the adaptive behavioral control system. In this manner, the adaptive behavioral control system 110 can use the audio output controller 131 in conjunction with the speakers 130 to emit audio in a direction of a particular interaction participant.
In some embodiments, the adaptive behavioral control system 110 includes the audio input processor 121. In some embodiments, the adaptive behavioral control system 110 includes the audio output controller 131.
In some embodiments, the human-machine interaction system 100d includes the adaptive behavioral control system (e.g., 110), an interaction sub-system that comprises a head assembly 103d and at least one appendage 105d, and a human reaction detection sub-system that comprises a camera 101d of the head assembly 103d and a microphone 106d of the head assembly 103d. In some embodiments, the head assembly 103d is mechanically attached to the body assembly 104d. In some embodiments, the head assembly 103d is mechanically attached to the body assembly 104d. In some embodiments, the human-machine interaction system 100d includes a motor assembly 163 that is constructed to rotate the head assembly 103d about a vertical axis 160d. In some embodiments, the human-machine interaction system 100d includes a motor assembly 162 that is constructed to move the head assembly 103d about a horizontal axis 161d.
In some embodiments, the head assembly 103d includes a speaker 102d.
In some embodiments, the head assembly 103d includes the adaptive behavioral control system (e.g., 110). In some embodiments, the body assembly 104d includes the adaptive behavioral control system (e.g., 110).
In some embodiments, the robot interaction system 100d includes the audio input processor 121, the audio input processor 121 is communicatively coupled to the adaptive behavioral control system of the system 100d, and the adaptive behavioral control system of the system 100d uses the audio input processor 121 to identify directions of interaction participant, as disclosed herein.
In some embodiments, the robot interaction system 100d includes the audio input processor 121, the audio input processor 121 is communicatively coupled to the adaptive behavioral control system of the system 100d, the audio input processor 121 is communicatively coupled to the microphone 106d, and the adaptive behavioral control system of the system 100d uses the audio input processor 121 to identify directions of interaction participant, based on volume of audio captured by the microphone 106d and a current position of the motor 163.
In some embodiments, the robot interaction system 100d includes the audio input processor 121, the audio input processor 121 is communicatively coupled to the adaptive behavioral control system of the system 100d, the audio input processor 121 is communicatively coupled a microphone array that includes a plurality of microphones arranged in different directions, and the adaptive behavioral control system of the system 100d uses the audio input processor 121 to identify directions of interaction participants, based on volumes of audio captured by the microphones of the microphone array. In some embodiments, the body assembly 104d includes the microphone array.
In some embodiments, the head assembly 103d includes the microphone array, the microphone array is rotated along with the head assembly 103d by the motor 163, and the adaptive behavioral control system of the system 100d uses rotational position information of the motor 163 to identify directions of interaction participants. In some embodiments, the head assembly 103d includes the microphone array, the microphone array is rotated along with the head assembly 103d by the motor 162, and the adaptive behavioral control system of the system 100d uses rotational position information of the motor 162 to identify directions of interaction participants. In some embodiments, the robot system 100d includes a GPS receiver, and the adaptive behavioral control system of the system 100d uses information of the GPS receiver to identify directions of interaction participants. In some embodiments, the robot system 100d includes an Inertial Measurement Unit (IMU), and the adaptive behavioral control system of the system 100d uses information of the IMU to identify directions of interaction participants. In some embodiments, the robot system 100d includes an accelerometer, and the adaptive behavioral control system of the system 100d uses information of the accelerometer to identify directions of interaction participants.
In some embodiments, the adaptive behavioral control system (e.g., 110) stores computer-readable data of computer-behavioral policy.
In some embodiments, the adaptive behavioral control system (e.g., 110) stores computer-readable data of a computer-behavioral policy for each interaction participant detected by the adaptive behavioral control system.
In some embodiments, the adaptive behavioral control system (e.g., 110) stores computer-readable data of a computer-behavioral policy for each of a plurality of interaction participants. In some embodiments, the adaptive behavioral control system (e.g., 110) stores an identical computer-behavioral policy for each interaction participant. In some embodiments, the adaptive behavioral control system (e.g., 110) updates the computer-behavioral policy for each interaction participant independently.
In some embodiments, the adaptive behavioral control system (e.g., 110) updates a computer-behavioral policy based on human reactions detected for the participant associated with the computer-behavioral policy. In some embodiments, the adaptive behavioral control system (e.g., 110) updates a computer-behavioral policy based on human reactions detected from the participant associated with the computer-behavioral policy. In some embodiments, computer-behavioral policy specifies the same actions for a first action type. In some embodiments, computer-behavioral policy specifies the different actions for a first action type. In some embodiments, the computer-behavioral policies of each participant are initialized with the same weight values. In some embodiments, the computer-behavioral policies of each participant are initialized with the same weighted sequence. In some embodiments, independent updating of the computer-behavioral policies of each participant results in at least two computer-behavioral policies having different weighted sequences. In some embodiments, independent updating of the computer-behavioral policies of each participant results in at least two computer-behavioral policies having different weight values.
In some embodiments, the data for the computer-behavioral policy includes instructions for performing an action of an action type. In some embodiments, the data for the computer-behavioral policy includes an identifier of an action of a particular type of action (an action type) and a corresponding weight value used to determine when to perform the action. In some embodiments, the data for the computer-behavioral policy includes an identifier of an action of a particular type of action (an action type) and a corresponding weight value used to determine when to prioritize the action. In some embodiments, the data for the computer-behavioral policy includes an identifier of an action of an action type and a corresponding weighted sequence of actions of the action type that is used to determine when to perform the action. In some embodiments, the data for the computer-behavioral policy includes computer-executable instructions of an action of an action type and a corresponding weight value used to determine when to perform the action. In some embodiments, the data for the computer-behavioral policy includes computer-executable instructions of an action of an action type and a corresponding weighted sequence of actions of the action type that is used to determine when to perform the action. In some embodiments, the data for the computer-behavioral policy includes a link to computer-executable instructions of an action of an action type and a corresponding weight value used to determine when to perform the action. In some embodiments, the data for the computer-behavioral policy includes a link to computer-executable instructions of an action of an action type and a corresponding weighted sequence of actions of the action type that is used to determine when to perform the action.
In some embodiments, the data for the computer-behavioral policy includes data for detected human reactions of interaction participants corresponding to performance of actions of the first action type. In some embodiments, the data for a detected human reaction is data indicating a detected emotion of an interaction participant. In some embodiments, the data for a detected human reaction is data indicating a classification of a detected emotion of an interaction participant. In some embodiments, the classification of a detected emotion of an interaction participant is one of “positive” and “negative”. In some embodiments, the classification of a detected emotion of an interaction participant is one of “positive”, “negative”, and “neutral”. In some embodiments, the classification of a detected emotion of an interaction participant is one a plurality of a plurality of rection values. In some embodiments, a reaction value is an integer with a range (e.g., 0-100) or a float value with a range (e.g., 0.0-1.0), with reaction values ordered from negative to positive. In some embodiments, a reaction value is a binary value indicating either a “positive” or a “negative” reaction.
In some embodiments, the method 400 includes: an adaptive behavioral control system (of a human-machine interaction system) controlling an interaction sub-system (e.g., 112 of
In some embodiments, the method 400 incudes; responsive to user-input received from an operator device communicatively coupled to the adaptive behavioral control system via a network interface (e.g., 211), the adaptive behavioral control system updating the computer-behavioral policy for the first action type based on the received user-input.
In some embodiments, the method 400 includes the adaptive behavioral control system storing information indicating human reactions detected by an external human-machine interaction system for performance of an action of the first action type by the external human-machine interaction system, and the adaptive behavioral control system stores the information detected by the external behavioral control system in association with information identifying the associated action. In this manner, the adaptive behavioral control system updates the policy used by the adaptive behavioral control system based on human reactions detected by external human-machine interaction systems as well as the human-machine interaction system of the adaptive behavioral control system.
In some embodiments, the policy specifies a weighted sequence of the plurality of actions for the first action type, and the adaptive behavioral control system controls the interaction sub-system to perform the plurality of actions for the first action type in accordance with the weighted sequence for the first action type. As a first example, the following is an exemplary evenly weighted sequence of actions A, B and C: {A, B, C, A, B, C, . . . }. As a second example, the following is an exemplary weighted sequence of actions A, B and C in which weight values of A, B and C are 70, 20% and 10%, respectively: {A, A, A, A, A, A, A, B, B, C, A, A, A, A, A, A, A, B, B, C, . . . }. As a third example, In some embodiment, the following is an exemplary weighted sequence of actions A, B and C in which weight values of A, B and C are 70, 20% and 10%, respectively, in which each action is sampled from a distribution: {C, A, A, A, C, B, A, A, A, A, B, C}.
In some embodiments, the method 400 includes: the adaptive behavioral control system selecting the computer-behavioral policy based on at least one of an identified interaction participant, a determined current interaction context, an action type, a current therapeutic goal, and a determined current therapy type. In some embodiments, the method 400 includes: the adaptive behavioral control system selecting the computer-behavioral policy based on a current therapeutic goal. In some embodiments, the method 400 includes: the adaptive behavioral control system determining the current therapy type being used during interaction with at least one participant, and selecting the computer-behavioral policy based on the determined current therapy type. In some embodiments, the current therapy type includes at least one of ABA (Applied Behavioral Analysis), OT (Occupational Therapy), and MI (Motivational Interviewing) therapy. In some embodiments, the method 400 includes: the adaptive behavioral control system identifying at least a first interaction participant, and selecting the computer-behavioral policy based on the identified first interaction participant. In some embodiments, the method 400 includes: the adaptive behavioral control system determining the current interaction context, and selecting the computer-behavioral policy based on the determined current interaction context. In some embodiments, the current interaction context includes one of a date, a location, an occasion (e.g., birthday), an event (e.g., funeral), a detected overall human emotion of at least one interaction participant (e.g., the participant seems happy or sad today), a detected human mood of at least one interaction participant.
In some embodiments, adaptive behavioral control system includes a plurality of computer-behavioral policies. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction participants. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapy types. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction contexts. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction contexts for at least one interaction participant. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapy types for at least one interaction participant. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction contexts for each of a plurality of interaction participants. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapy types for each of a plurality of interaction participants.
In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapeutic goals for each of a plurality of interaction participants. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapeutic goals for at least one interaction participant.
In some embodiments, the adaptive behavioral control system stores information indicating each detected human reaction in association with the selected computer-behavioral policy, and the adaptive behavioral control system provides information indicating each detected human reaction for the selected policy to an external system in response to a request provide by the external system via an API of the adaptive behavioral control system.
In some embodiments, controlling the interaction sub-system to perform a plurality of actions for a first action type in accordance with the computer-behavioral policy includes: controlling the interaction sub-system (e.g., 112, 131, 130, 102c, 102d, 103d, 105d, 106d) to perform a first action for the first action type based on weight values of the actions for the first action type included in the policy. In some embodiments, for a first instance of performing an action for the first action type, the adaptive behavioral control system (e.g., 110) selects the action having a highest weight value (as specified in the policy) and performs the selected action. In some embodiments, for a first instance of performing an action for the first action type, the adaptive behavioral control system (e.g., 110) randomly selects an action and performs the selected action.
In some embodiments, controlling the interaction sub-system (e.g., 112) to perform a plurality of actions for a first action type in accordance with the computer-behavioral policy includes: controlling the interaction sub-system to perform a first action for the first action type based on weight values of the actions for the first action type included in the policy and log information indicating actions previously performed by the human-machine interaction system for the first action type.
In some embodiments, controlling the interaction sub-system to perform a plurality of actions for a first action type in accordance with the computer-behavioral policy includes: controlling the interaction sub-system to select an action for the first action type based on a weighted sequence of the plurality of actions for the first action type. For example, the following is an exemplary evenly weighted sequence of actions A, B and C: {A, B, C, A, B, C, . . . }. For example, the following is an exemplary weighted sequence of actions A, B and C in which weight values of A, B and C are 70, 20% and 10%, respectively: {A, A, A, A, A, A, A, B, B, C, A, A, A, A, A, A, A, B, B, C, . . . }.
In some embodiments, an action type is a type of human interaction, such as, for example, a greeting, an acknowledgment, a gesture, a question, a statement, a reaction, an expression, and the like. In some embodiments, for at least one action type (e.g., a Greeting action type), the computer-behavioral policy specifies a plurality of alternative actions. In some embodiments, the plurality of alternative actions includes different actions for the action type (e.g., greeting actions). In some embodiments, the plurality of alternative actions includes different audible actions for the action type (e.g., greeting actions). In some embodiments, audible actions include actions performed by controlling a speaker (e.g., 130, 102d). In some embodiments, the plurality of alternative actions includes different physical movement actions for the action type (e.g., greeting actions). In some embodiments, physical movement actions include actions performed by controlling a motor (e.g., 162, 163 of
In some embodiments, the plurality of alternative actions include at least two actions for emission of audible sound at different volumes by controlling an audio output sub-system (e.g., 130, 102d).
In some embodiments, the plurality of alternative actions include at least two actions for emission of audible sound at different playback rates by controlling an audio output sub-system.
In some embodiments, the plurality of alternative actions include at least two actions for emission of spoken word sounds at different speeds by controlling an audio output sub-system.
In some embodiments, the plurality of alternative actions include at least two actions for emission of different sounds by controlling an audio output sub-system.
In some embodiments, the plurality of alternative actions include at least two actions for emission of audible output using different voices by controlling an audio output sub-system.
In some embodiments, the plurality of alternative actions include at least two actions for emission of audible output corresponding to spoken words using different vocal tones by controlling an audio output sub-system.
In some embodiments, the plurality of alternative actions include at least two actions for emission of audible output corresponding to spoken words corresponding to different verbal instructions by controlling an audio output sub-system.
In some embodiments, the plurality of alternative actions include at least two actions for display of different images by controlling a video display sub-system (e.g., 102c).
In some embodiments, the plurality of alternative actions include at least two actions for performing different physical gestures by controlling a mechanical robotic sub-system (e.g., 103d, 104d, 105d, 162, 163).
In some embodiments, the plurality of alternative actions include at least two actions for performing different physical gestures by controlling a mechanical robotic sub-system that includes at least one robotic appendage (e.g., 105d).
In some embodiments, the plurality of alternative actions include at least two actions for performing different physical gestures by controlling a mechanical robotic sub-system that includes at least one robotic appendage (e.g., 105d), a robotic head assembly (e.g., 103d) that is mechanically attached to a robotic body assembly (e.g., 104d), wherein the head assembly includes a motorized assembly (e.g., motor 162, 163) that is constructed to rotate the head assembly in at least one degree of freedom in relation to the body assembly.
In some embodiments, the plurality of alternative actions include at least one action performed by controlling a light emission sub-system that includes an array of lights, wherein the light emission sub-system is constructed to individually control each light of the light emission sub-system.
In some embodiments, the plurality of alternative actions include at least one action performed by controlling a LED ring that includes an array of lights, wherein the LED ring is constructed to individually control each light of the LED ring.
In some embodiments, the plurality of alternative actions include at least one action performed by controlling an audio output sub-system. In some embodiments, the plurality of alternative actions include at least one action performed by controlling an audio output sub-system and a video output sub-system. In some embodiments, the plurality of alternative actions include at least one action performed by controlling an audio output sub-system. In some embodiments, the plurality of alternative actions include at least one action performed by controlling an audio output sub-system, a video output sub-system, and a mechanical robotic sub-system.
In some embodiments, the plurality of alternative actions include at least one action performed by controlling at least one of audio output sub-system, a video output sub-system, and a mechanical robotic sub-system.
In some embodiments, the plurality of alternative actions include at least one action performed by controlling at least one of audio output sub-system, a video output sub-system, a mechanical robotic sub-system, a light emission sub-system, and a LED ring.
In some embodiments, the plurality of alternative actions include at least one action for performing a facial expression by controlling a robotic head assembly (e.g., 103d). In some embodiments, the head assembly includes at least one mechanical facial feature, and the robotic head assembly is constructed to control movement of the at least one mechanical facial feature to perform at least one facial expression. In some embodiments, the head assembly includes at least one display device, and the robotic head assembly is constructed to control display of the display device to perform at least one facial expression. In some embodiments, the head assembly includes at least one light emission subsystem, and the robotic head assembly is constructed to control light emission of the light emission display system to perform at least one facial expression. In some embodiments, the head assembly includes at least one LED ring, and the robotic head assembly is constructed to control at least one LED of the LED ring to perform at least one facial expression.
In some embodiments, the plurality of alternative actions include at least one action for a robotic head assembly (e.g., 103d) to output audio in a particular voice, by controlling an audio output sub-system of the robotic head assembly. In some embodiments, the plurality of alternative actions include at least one action for a robotic head assembly (e.g., 103d) to output audible speech, by controlling an audio output sub-system of the robotic head assembly.
In some embodiments, the adaptive behavioral control system detecting a human reaction (process S402) includes detecting a human reaction to the performance of the first action by an interaction participant (e.g., 810, 820 of
In some embodiments, the adaptive behavioral control system is constructed to detect a human reaction by using a gesture detection system that identifies human gestures based on video data of the interaction participant provided by the human reaction detection sub-system. In some embodiments, the adaptive behavioral control system includes a data structure that associates human gestures with one of a positive human reaction and a negative human reaction.
In some embodiments, the adaptive behavioral control system is constructed to detect a human reaction by using a smile detection system that identifies human gestures based on video data of the interaction participant provided by the human reaction detection sub-system.
In some embodiments, the adaptive behavioral control system detecting a human reaction (process S402) includes using an emotion detection system (e.g., 279 of
In some embodiments, the adaptive behavioral control system storing information indicating each detected human reaction (process S403) includes storing data indicating each detected emotion. In some embodiments, the adaptive behavioral control system storing information indicating each detected human reaction (process S403) includes determining a classification of each detected emotion, and storing data indicating the classification of each detected emotion. In some embodiments, the classification of a detected emotion of an interaction participant is one of “positive” and “negative”. In some embodiments, the classification of a detected emotion of an interaction participant is one of “positive”, “negative”, and “neutral”.
In some embodiments, the adaptive behavioral control system storing information indicating each detected human reaction (process S403) includes: for each detected human reaction, determining whether the reaction relates to at least one therapeutic goal, and storing data indicating each therapeutic goal related to the detected human reaction.
In some embodiments, at least one therapeutic goal is a short term interaction goal (e.g., smiles back at robot). In some embodiments, at least one therapeutic goal is a long term goal (e.g., manifesting long-term behavior skills).
In some embodiments, the adaptive behavioral control system storing information indicating each detected human reaction (process S403) includes: for each detected human reaction, determining whether the reaction relates to at least one interaction goal, and storing data indicating each interaction goal related to the detected human reaction.
In some embodiments, at least one interaction goal is a short term interaction goal (e.g., smiles back at robot). In some embodiments, at least one interaction goal is a long term goal (e.g., manifesting long-term behavior skills).
In some embodiments, the policy (e.g., 301-305) specifies a weighted sequence of the plurality of actions for the first action type, and the adaptive behavioral control system controls the interaction sub-system to perform the plurality of actions for the first action type in accordance with the weighted sequence for the first action type. In some embodiments, the adaptive behavioral control system storing information indicating each detected human reaction (process S403) includes determining a classification of each detected emotion, and storing data indicating the classification of each detected emotion (e.g., “positive”, “negative”, and “neutral”).
In some embodiments, updating the computer-behavioral policy for the first action type (process S404) includes: determining whether a distribution of detected positive human emotions stored for the actions of the first action type differs from the distribution of the actions in the weighted sequence of actions for the first action above a difference threshold based on results of a distribution comparison; and updating the computer-behavioral policy for the first action type responsive to a determination that the distribution of detected positive human emotions stored for the actions of the first action type differs from the distribution of the actions in the weighted sequence of actions for the first action above the difference threshold.
In some embodiments, updating the computer-behavioral policy for the first action type (process S404) includes: determining whether a distribution of detected negative human emotions stored for the actions of the first action type differs from the distribution of the actions in the weighted sequence of actions for the first action above a difference threshold based on results of a distribution comparison; and updating the computer-behavioral policy for the first action type responsive to a determination that the distribution of detected negative human emotions stored for the actions of the first action type differs from the distribution of the actions in the weighted sequence of actions for the first action above the difference threshold.
In some embodiments, updating the computer-behavioral policy for the first action type (process S404) includes: determining whether a distribution of detected human reactions related to a therapeutic goal from among the stored human reaction information for the actions of the first action type differs from the distribution of the actions in the weighted sequence of actions for the first action above a difference threshold based on results of a distribution comparison; and updating the computer-behavioral policy for the first action type responsive to a determination that the distribution of reactions related to the therapeutic goal for the actions of the first action type differs from the distribution of the actions in the weighted sequence of actions for the first action above the difference threshold.
In some embodiments, updating the computer-behavioral policy for the first action type (process S404) includes: determining whether a distribution of detected human reactions related to any one of a plurality of therapeutic goals from among the stored human reaction information for the actions of the first action type differs from the distribution of the actions in the weighted sequence of actions for the first action above a difference threshold based on results of a distribution comparison; and updating the computer-behavioral policy for the first action type responsive to a determination that the distribution of reactions related to any one of a plurality of therapeutic goals for the actions of the first action type differs from the distribution of the actions in the weighted sequence of actions for the first action above the difference threshold.
In some embodiments, the distribution comparison is performed after a predetermined number of iterations of the sequence are performed. In some embodiments, the distribution comparison is performed periodically at regular time intervals. In some embodiments, the distribution comparison is performed after a predetermined number of positive human emotions have been recorded. In some embodiments, the distribution comparison is performed after a predetermined number of positive human emotions have been recorded for a particular action of the first action type.
In some embodiments, the distribution comparison is performed after a predetermined number of negative human emotions have been recorded. In some embodiments, the distribution comparison is performed after a predetermined number of negative human emotions have been recorded for a particular action of the first action type.
In some embodiments, the distribution comparison is performed after a predetermined number of human reactions have been recorded for at least one therapeutic goal. In some embodiments, the distribution comparison is performed after a predetermined number of human reactions have been recorded for at least one therapeutic goal of the first action type.
In some embodiments, the distribution comparison is performed after a predetermined number of human reactions have been recorded for at least one interaction goal. In some embodiments, the distribution comparison is performed after a predetermined number of human reactions have been recorded for at least one interaction goal of the first action type.
For example, in a weighted sequence having a distribution of Action A (70%), Action B (20%), Action C (10%), in a case where positive reactions are recorded as Action A (10%), Action B (20%), Action C (70%), the computer-behavioral policy is updated for the first action type. In some embodiments, the weighted sequence is updated such that the distribution of actions within the sequence matches the distribution of positive reactions recorded for the actions of the first action type. In some embodiments, the weighted sequence is updated such that the distribution of actions within the sequence is incrementally adjusted based on the distribution of positive reactions recorded for the actions of the first action type.
In some embodiments, updating the computer-behavioral policy for the first action type (process S404) includes: determining whether a predetermined number of consecutive positive human emotions have been detected for at least one action of the first action type; and updating the computer-behavioral policy for the first action type responsive to a determination that the predetermined number of consecutive positive human emotions have been detected for at least one action of the first action type.
In some embodiments, updating the computer-behavioral policy for the first action type (process S404) includes: determining whether a predetermined number of consecutive positive human reactions have been detected for at least one action of the first action type; and updating the computer-behavioral policy for the first action type responsive to a determination that the predetermined number of consecutive positive human reactions have been detected for at least one action of the first action type.
In some embodiments, the method 500 includes: an adaptive behavioral control system (e.g., 110 of
In some embodiments, process S502 is similar to process S401 of
In some embodiments, the method 500 includes: the adaptive behavioral control system selecting the computer-behavioral policy based on at least one of an identified interaction participant, a determined current interaction context, an action type, a current therapeutic goal, and a determined current therapy type.
In some embodiments, the method 500 includes: the adaptive behavioral control system selecting the computer-behavioral policy based on a current therapeutic goal.
In some embodiments, the method 500 includes: the adaptive behavioral control system determining the current therapy type being used during interaction with at least one participant, and selecting the computer-behavioral policy based on the determined current therapy type. In some embodiments, the current therapy type includes at least one of ABA (Applied Behavioral Analysis), OT (Occupational Therapy), and MI (Motivational Interviewing) therapy. In some embodiments, the method 500 includes: the adaptive behavioral control system identifying at least a first interaction participant, and selecting the computer-behavioral policy based on the identified first interaction participant. In some embodiments, the method 500 includes: the adaptive behavioral control system determining the current interaction context, and selecting the computer-behavioral policy based on the determined current interaction context. In some embodiments, the current interaction context includes one of a date, an occasion (e.g., birthday), an event (e.g., funeral), a detected overall human emotion of at least one interaction participant (e.g., the participant seems happy or sad today).
In some embodiments, adaptive behavioral control system includes a plurality of computer-behavioral policies. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction participants. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapy types. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction contexts. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction contexts for at least one interaction participant. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapy types for at least one interaction participant. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction contexts for each of a plurality of interaction participants. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapy types for each of a plurality of interaction participants.
In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapeutic goals for each of a plurality of interaction participants. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapeutic goals for at least one interaction participant.
In some embodiments, the adaptive behavioral control system stores information indicating each detected human reaction in association with the selected computer-behavioral policy, and the adaptive behavioral control system provides information indicating each detected human reaction for the selected policy to an external system in response to a request provide by the external system via an API of the adaptive behavioral control system.
In some embodiments, the method 600 includes: an adaptive behavioral control system of a human-machine interaction system detecting a plurality of interaction participants (e.g., 810, 820 of
In some embodiments, the method 600 includes; responsive to user-input received from an operator device communicatively coupled to the adaptive behavioral control system via a network interface (e.g., 211), the adaptive behavioral control system updating at least one computer-behavioral policy based on the received user-input.
The computer-behavioral policy of each detected interaction participant is independently updated based on detected human reactions for the associated interaction participant.
In some embodiments, the adaptive behavioral control system stores at least a first computer-behavioral policy for a first interaction participant and a second computer-behavioral policy for a second interaction participant, the first policy and the second policy each specifying a same plurality of actions for the first action type, each action being a different alternative action for the action type.
In some embodiments, process S603 is similar to process S401 of
In some embodiments, the method 600 includes: the adaptive behavioral control system selecting a computer-behavioral policy for each detected interaction participant based on at least one of an identity of the interaction participant, a determined current interaction context for the interaction participant, an action type, a current therapeutic goal, and a determined current therapy type for the interaction participant.
In some embodiments, the method 600 includes: the adaptive behavioral control system selecting the computer-behavioral policy for at least one detected interaction participant based on a current therapeutic goal.
In some embodiments, the method 600 includes: for at least one detected interaction participant, the adaptive behavioral control system determining the current therapy type being used during interaction with the participant, and selecting the computer-behavioral policy based on the determined current therapy type. In some embodiments, the current therapy type includes at least one of ABA (Applied Behavioral Analysis), OT (Occupational Therapy), and MI (Motivational Interviewing) therapy. In some embodiments, the method 600 includes: for at least one detected participant, the adaptive behavioral control system selecting the computer-behavioral policy based on an identify of the interaction participant. In some embodiments, the method 600 includes: for at least one detected participant the adaptive behavioral control system determining the current interaction context of the participant, and selecting the computer-behavioral policy based on the determined current interaction context. In some embodiments, the current interaction context includes one of a date, an occasion (e.g., birthday), an event (e.g., funeral), a detected overall human emotion of at least one interaction participant (e.g., the participant seems happy or sad today).
In some embodiments, the adaptive behavioral control system includes a plurality of computer-behavioral policies. In some embodiments, the adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction participants. In some embodiments, the adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapy types. In some embodiments, the adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction contexts. In some embodiments, the adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction contexts for at least one interaction participant. In some embodiments, the adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapy types for at least one interaction participant. In some embodiments, the adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction contexts for each of a plurality of interaction participants. In some embodiments, the adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapy types for each of a plurality of interaction participants.
In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapeutic goals for each of a plurality of interaction participants. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapeutic goals for at least one interaction participant.
In some embodiments, the adaptive behavioral control system stores information indicating each detected human reaction in association with a corresponding computer-behavioral policy, and responsive to a request for human reaction information for a specified policy provided by an external system via an API of the adaptive behavioral control system, the adaptive behavioral control system provides information indicating each detected human reaction for the specified policy to the external system.
In some embodiments, the human-machine interaction system performing the method 700 includes a body assembly (e.g., 104d); a head assembly (e.g., 103d) mechanically attached to the body assembly via a motor assembly (e.g., 162, 163); the motor assembly, the motor assembly being constructed to rotate the head assembly about a vertical axis (e.g., 160d) and a horizontal axis (e.g., 161d); at least a first motorized appendage (e.g., 105d) constructed to perform movement in accordance with at least one human-interaction gesture (e.g., an arm gesture, a hand gesture, waving, sign language, etc.); an adaptive behavioral control system (e.g., 110) communicatively coupled to at least one of the motor assembly, the head assembly, and the appendage; a human reaction detection sub-system (e.g., 111, 101d, 106d) communicatively coupled to the adaptive behavioral control system; and an interaction sub-system (e.g., 102d) communicatively coupled to the adaptive behavioral control system.
In some embodiments, the method 700 includes: an adaptive behavioral control system (e.g., 110) of a human-machine interaction system (e.g., 100d of
In some embodiments, the method 700 includes; responsive to user-input received from an operator device communicatively coupled to the adaptive behavioral control system via a network interface (e.g., 211), the adaptive behavioral control system updating at least one computer-behavioral policy based on the received user-input.
The computer-behavioral policy of each detected interaction participant is independently updated based on detected human reactions for the associated interaction participant.
In some embodiments, detecting a plurality of interaction participants includes detecting locations of each detected interaction participant, and generating a three-dimensional world model (e.g., 278 of
In some embodiments, during initiation of an interaction with a previously detected interaction participant (e.g., 810, 820), the adaptive behavioral control system controls the motor assembly (e.g., 162, 163) to orient the head assembly (e.g., 103d) in a direction of the previously detected interaction participant by using the previously detected location included in the world model (e.g., 278).
In some embodiments, during processing of a response to an action performed by a previously detected interaction participant, the adaptive behavioral control system controls the motor assembly (e.g., 162, 163) to orient the head assembly (e.g., 103d) in a direction of the previously detected interaction participant by using the previously detected location included in the world model (e.g., 278).
In some embodiments, detecting a plurality of interaction participants includes detecting a face of each interaction participant. In some embodiments, an orientation of each detected face is recorded. In some embodiments, the three-dimensional world model (e.g., 278) indicates the orientation of each detected face, and during initiation of an interaction with a previously detected interaction participant, the adaptive behavioral control system controls the motor assembly to orient the head assembly in a direction of the face of a previously detected interaction participant by using the previously detected location and face orientation included in the world model.
In some embodiments, detecting a plurality of interaction participants includes detecting a face of each interaction participant, and the three-dimensional world model (e.g., 278) includes detected orientation of detected faces, and during processing of a response to an action performed by a previously detected interaction participant, the adaptive behavioral control system controls the motor assembly to orient the head assembly in a direction of the face of a previously detected interaction participant by using the previously detected location and face orientation included in the world model.
In some embodiments, the adaptive behavioral control system stores at least a first computer-behavioral policy for a first interaction participant and a second computer-behavioral policy for a second interaction participant, the first policy and the second policy each specifying a same plurality of actions for the first action type, each action being a different alternative action for the action type.
In some embodiments, process S703 is similar to process S401 of
In some embodiments, the method 700 includes: the adaptive behavioral control system selecting a computer-behavioral policy for each detected interaction participant based on at least one of an identity of the interaction participant, a determined current interaction context for the interaction participant, an action type, a current therapeutic goal, and a determined current therapy type for the interaction participant.
In some embodiments, the method boo includes: the adaptive behavioral control system selecting the computer-behavioral policy for at least one detected interaction participant based on a current therapeutic goal.
In some embodiments, the method 700 includes: for at least one detected interaction participant, the adaptive behavioral control system determining the current therapy type being used during interaction with the participant, and selecting the computer-behavioral policy based on the determined current therapy type. In some embodiments, the current therapy type includes at least one of ABA (Applied Behavioral Analysis), OT (Occupational Therapy), and MI (Motivational Interviewing) therapy. In some embodiments, the method 700 includes: for at least one detected participant, the adaptive behavioral control system selecting the computer-behavioral policy based on an identify of the interaction participant. In some embodiments, the method 700 includes: for at least one detected participant the adaptive behavioral control system determining the current interaction context of the participant, and selecting the computer-behavioral policy based on the determined current interaction context. In some embodiments, the current interaction context includes one of a date, an occasion (e.g., birthday), an event (e.g., funeral), a detected overall human emotion of at least one interaction participant (e.g., the participant seems happy or sad today).
In some embodiments, adaptive behavioral control system includes a plurality of computer-behavioral policies. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction participants. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapy types. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction contexts. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction contexts for at least one interaction participant. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapy types for at least one interaction participant. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of interaction contexts for each of a plurality of interaction participants. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapy types for each of a plurality of interaction participants.
In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapeutic goals for each of a plurality of interaction participants. In some embodiments, adaptive behavioral control system includes a computer-behavioral policy for each of a plurality of therapeutic goals for at least one interaction participant.
In some embodiments, the adaptive behavioral control system stores information indicating each detected human reaction in association with a corresponding computer-behavioral policy, and responsive to a request for human reaction information for a specified policy provided by an external system via an API of the adaptive behavioral control system, the adaptive behavioral control system provides information indicating each detected human reaction for the specified policy to the external system.
In some embodiments, an action of the first action type is an action for controlling a robotic body head assembly (e.g., 103d of
In some embodiments, during initiation of an interaction with a previously detected interaction participant, the human-machine interaction system (e.g., 100d of
In some embodiments, during processing of a response to an action performed by a previously detected interaction participant, the human-machine interaction system (e.g., 100d of
In some embodiments, the head assembly (e.g., 103d) includes a motorized assembly (e.g., 162, 163) that is constructed to rotate the head assembly (e.g., 103d) in at least one degree of freedom in relation to the body assembly (e.g., 104d), and during control of the robotic head assembly to orient the head assembly in a direction of the interaction participant, the adaptive behavioral control system controls the head assembly to rotate about a first axis (e.g., 160d) in the direction of the detected interaction participant.
In some embodiments, the head assembly includes a motorized assembly (e.g., 162, 163) that is constructed to rotate the head assembly in at least two degrees of freedom in relation to the body assembly, and during control of the robotic head assembly to orient the head assembly in a direction of a detected face of an interaction participant, the adaptive behavioral control system controls the head assembly to rotate about a first axis (e.g., 160d) in the direction of the interaction participant, and controls the head assembly to rotate about a second axis (e.g., 161d) towards the detected face. In some embodiments, the first axis is a vertical axis (e.g., 160d) and the second axis is a horizontal axis (e.g., 161d).
In some embodiments, the adaptive behavioral control system detects participants by using at least one of temperature data captured by a heat detection sub-system (e.g., infrared camera, heat sensor, and the like), image data captured by a video capture sub-system (e.g., 101d) and audio data captured by an audio capture sub-system (e.g., 106d).
In some embodiments, the adaptive behavioral control system determines a direction of a detected participant by using at least one of temperature data captured by a heat detection sub-system (e.g., infrared camera, heat sensor, and the like), image data captured by a video capture sub-system (e.g., 101d), audio data captured by an audio capture sub-system (e.g., 106d), and rotational position information of a motor assembly (e.g., 162, 163). In some embodiments, the adaptive behavioral control system determines a direction of a detected participant by using at least one of temperature data captured by a heat detection sub-system (e.g., infrared camera, heat sensor, and the like), image data captured by a video capture sub-system (e.g., 101d), audio data captured by an audio capture sub-system (e.g., 106d), IMU data generated by an IMU of the human-machine interaction system, accelerometer data generated by an accelerometer of the human-machine interaction system, and rotational position information of a motor assembly (e.g., 162, 163).
In some embodiments, for each detected interaction participant, the adaptive behavioral control system stores information identifying the detected participant in association with information indicating the determined direction of the detected participant in a world model (e.g., 278).
In some embodiments, the adaptive behavioral control system determines an orientation of a face of a detected participant by using at least one of temperature data captured by a heat detection sub-system (e.g., infrared camera, heat sensor, and the like), image data captured by a video capture sub-system (e.g., 101d), audio data captured by an audio capture sub-system (e.g., 106d), and rotational position information of a motor assembly (e.g., 162, 163). In some embodiments, the adaptive behavioral control system determines an orientation of a face of a detected participant by using at least one of temperature data captured by a heat detection sub-system (e.g., infrared camera, heat sensor, and the like), image data captured by a video capture sub-system (e.g., 101d), audio data captured by an audio capture sub-system (e.g., 106d), IMU data generated by an IMU of the human-machine interaction system, accelerometer data generated by an accelerometer of the human-machine interaction system, and rotational position information of a motor assembly (e.g., 162, 163).
In some embodiments, for each detected interaction participant, the adaptive behavioral control system stores information identifying the detected participant in association with information indicating the determined direction of the detected participant and information indicating the determined facial orientation of the participant in a world model (e.g., 278).
In some embodiments, the audio capture sub-system captures audio emitted by the participant, and generates audio data corresponding to the captured audio, and the adaptive behavioral control system uses the audio data to identify presence and general location of the participant. In some embodiments, the audio capture sub-system includes a microphone array that is constructed to generate directional information corresponding to captured audio based on a direction associated with a microphone of the array that captures the audio at a highest volume level among volume levels of audio captured by the microphones of the microphone array. As shown in
In some embodiments, the audio capture sub-system captures audio emitted by the participant, and generates audio data corresponding to the captured audio, and the adaptive behavioral control system uses the audio data to identify presence of the participant by recognizing the audio emitted by the participant as matching audio emitted by a human. In some embodiments, the audio capture sub-system captures audio emitted by the participant, and generates audio data corresponding to the captured audio, and the adaptive behavioral control system uses the audio data to identify presence of the participant by recognizing the audio emitted by the participant as matching audio emitted by a human that is not a currently recognized participant. In some embodiments, the audio capture sub-system captures audio emitted by the participant, generates audio data corresponding to the captured audio, determines whether the audio data corresponds to a human voice, determines whether the human voice corresponds to a current participant, and responsive to a determination that the human voice does not correspond to a current human interaction participant, the adaptive behavioral control system identifies the presence of a new participant.
In some embodiments, the audio capture sub-system captures audio and generates audio data corresponding to the captured audio, the adaptive behavioral control system determines whether the audio data corresponds to an environment change that corresponds to presence of a new interaction participant, and in a case where the adaptive behavioral control system determines that the audio data corresponds to an environment change that corresponds to presence of a new interaction participant, the adaptive behavioral control system identifies presence of a new interaction participant. In some embodiments, audio data that corresponds to an environment change that corresponds to presence of a new interaction participant includes audio data that corresponds to opening of a door. In some embodiments, audio data that corresponds to an environment change that corresponds to presence of a new interaction participant includes audio data that corresponds to footsteps. In some embodiments, audio data that corresponds to an environment change that corresponds to presence of a new interaction participant includes audio data that corresponds to breathing.
In some embodiments, the orientation of the face of the detected participant is determined by using image data captured by a video capture sub-system (e.g., wid of
In some embodiments, the adaptive behavioral control system is constructed to detect presence of interaction participants and respective locations, and generate a three-dimensional world model of detected locations of detected interaction participants. In some embodiments, the adaptive behavioral control system is constructed to detect presence of interaction participants by using a video capture sub-system of the human-machine interaction system, as described herein. In some embodiments, the adaptive behavioral control system is constructed to detect presence of interaction participants by using an audio capture sub-system of the human-machine interaction system, as described herein. In some embodiments, the adaptive behavioral control system is constructed to detect locations of interaction participants by using a video capture sub-system, as described herein. In some embodiments, the adaptive behavioral control system is constructed to detect locations of interaction participants by using an audio capture sub-system, as described herein.
In some embodiments, during initiation of an interaction with a previously detected interaction participant, the adaptive behavioral control system controls the face assembly to orient the face assembly in a direction of a detected interaction participant by using the previously detected location included in the world model.
In some embodiments, during processing of a response to an action performed by a previously detected interaction participant, the adaptive behavioral control system controls the face assembly to orient the face assembly in a direction of a detected interaction participant by using the previously detected location included in the world model.
In some embodiments, the adaptive behavioral control system is constructed to generate a three-dimensional world model of all detected faces, and wherein during initiation of an interaction with a participant associated with a previously detected face, the adaptive behavioral control system controls the face assembly to orient the face assembly in a direction of a detected face by using the previously detected direction included in the world model.
In some embodiments, bus 201 interfaces with an audio system device 271. In some embodiments, the audio system device 271 includes the audio input processor 121. In some embodiments, the audio system device 271 includes the audio output controller 131. In some embodiments, the audio system interface device 271 includes an audio system of the human reaction detection sub-system 111. In some embodiments, the audio system interface device 271 includes an audio system of the interaction sub-system system 111. In some embodiments, the audio system device 271 includes a speaker (e.g., 120 of
In some embodiments, bus 201 interfaces with a video system device 272. In some embodiments, the video system device 272 includes a video system of the human reaction detection sub-system 111. In some embodiments, the video system device 272 includes display system of the interaction sub-system 111. In some embodiments, the video system device 272 includes a camera (e.g., low of
In some embodiments, bus 201 interfaces with a motor controller 273. In some embodiments, the motor controller 273 is constructed to control at least one of the motors 162 and 163 of the robot system 100d of
In some embodiments, the processors include one or more of an ARM processor, an X86 processor, a GPU (Graphics Processing Unit), and the like. In some embodiments, at least one of the processors includes at least one arithmetic logic unit (ALU) that supports a SIMD (Single Instruction Multiple Data) system that provides native support for multiply and accumulate operations.
In some embodiments, at least one of a central processing unit (processor), a GPU, and a multi-processor unit (MPU) is included.
In some embodiments, the processors and the main memory form a processing unit 299. In some embodiments, the processing unit includes one or more processors communicatively coupled to one or more of a RAM, ROM, and machine-readable storage medium; the one or more processors of the processing unit receive instructions stored by the one or more of a RAM, ROM, and machine-readable storage medium via a bus; and the one or more processors execute the received instructions. In some embodiments, the processing unit is an ASIC (Application-Specific Integrated Circuit). In some embodiments, the processing unit is a SoC (System-on-Chip).
In some embodiments, the processing unit includes at least one arithmetic logic unit (ALU) that supports a SIMD (Single Instruction Multiple Data) system that provides native support for multiply and accumulate operations. In some embodiments the processing unit is a Central Processing Unit such as an Intel Xeon processor. In other embodiments, the processing unit includes a Graphical Processing Unit such as NVIDIA Tesla.
The network adapter device 211 provides one or more wired or wireless interfaces for exchanging data and commands. Such wired and wireless interfaces include, for example, a universal serial bus (USB) interface, Bluetooth interface, Wi-Fi interface, Ethernet interface, near field communication (NFC) interface, and the like.
In some embodiments, network device 211 is communicatively coupled to another human-machine interaction system. In some embodiments, network device 211 is communicatively coupled to another human-machine interaction system via a network (e.g., the Internet). In some embodiments, the network device 211 is communicatively coupled to a human-machine interaction server via a network (e.g., the Internet), and the human-machine interaction server is constructed to provide computer-behavioral policies to the adaptive behavioral control system via the network device 211. In some embodiments, the human-machine interaction server is constructed to provide information for detected human reactions recognized by other human-machine interaction systems for at least one action of a computer-behavioral policy of the adaptive behavioral control system via the network device 211. In this manner, the adaptive behavioral control system of
Machine-executable instructions in software programs (such as an operating system, application programs, and device drivers) are loaded into the memory (of the processing unit) from the processor-readable storage medium, the ROM or any other storage location. During execution of these software programs, the respective machine-executable instructions are accessed by at least one of processors (of the processing unit) via the bus, and then executed by at least one of processors. Data used by the software programs are also stored in the memory, and such data is accessed by at least one of processors during execution of the machine-executable instructions of the software programs.
The processor-readable storage medium 605 is one of (or a combination of two or more of) a hard drive, a flash drive, a DVD, a CD, an optical disk, a floppy disk, a flash storage, a solid state drive, a ROM, an EEPROM, an electronic circuit, a semiconductor memory device, and the like. The processor-readable storage medium 205 includes machine-executable instructions (and related data) for an operating system 212, software programs 213, device drivers 214, and machine-executable instructions for one or more of the processes of
In some embodiments, the processor-readable storage medium 205 includes machine-executable instructions for an emotion detection module 279. In some embodiments, emotion detection module 279 is constructed to detect an emotion based on captured image data. In some embodiments, emotion detection module 279 is constructed to detect an emotion based on captured audio data. In some embodiments, emotion detection module 279 is constructed to detect an emotion based on captured image data and captured audio data. In some embodiments, emotions detectable by the emotion detection module 279 include anger, contempt, disgust, fear, happiness, neutral, sadness, and surprise. In some embodiments, emotions detectable by the emotion detection module 279 include happy, sad, angry, confused, disgusted, surprised, calm, unknown. In some embodiments, the emotion detection module 279 is constructed to classify detected emotions as either positive, negative, or neutral. In some embodiments, the adaptive behavioral control system uses the emotion detection module 279 to obtain a determined emotion classification (e.g., positive, neutral, negative) after performance of an action of the first action type, and store the determined emotion classification in association with the performed action (e.g., in the storage medium 205).
The systems and methods of some embodiments and variations thereof can be embodied and/or implemented at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions. The instructions are preferably executed by computer-executable components. The computer-readable medium can be stored on any suitable computer-readable media such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives, floppy drives, or any suitable device. The computer-executable component is preferably a general or application specific processor, but any suitable dedicated hardware or hardware/firmware combination device can alternatively or additionally execute the instructions.
As a person skilled in the art will recognize from the previous detailed description and from the figures and claims, modifications and changes can be made to the embodiments disclosed herein without departing from the scope defined in the claims.
This application claims the benefit of U.S. Provisional Application Ser. No. 62/715,665, filed on 7 Aug. 2018, which is incorporated in its entirety by this reference.
| Number | Date | Country | |
|---|---|---|---|
| 62715665 | Aug 2018 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 18072685 | Nov 2022 | US |
| Child | 18208160 | US | |
| Parent | 17199083 | Mar 2021 | US |
| Child | 18072685 | US | |
| Parent | 16531118 | Aug 2019 | US |
| Child | 17199083 | US |
