The disclosure relates to human-machine interactions, and more particularly to enabling seamless indirect interactions among one or more members in a specific environment.
Smart devices in a particular environment may be capable of interacting with the outside world and with each other. However, there is a need for specific methods and devices to enable indirect interactions between devices within the environment.
As an example, consider an environment such as a home, in which a mother intends to go shopping. In order to prepare to do so, she may interact directly (e.g., over calls, chat, face-to-face, and so on) with every family member to get their specific requirements/needs.
As another example scenario, consider an environment such as a garage, in which a person may be stuck in a tight spot and may need a new tool box, but there is no one else in earshot. The person who is stuck may get out of the tight spot and then fetch the tool box by himself, or may contact another person to fetch the tool box.
As yet another example, consider an environment such as a home, in which a child may be studying and may be disturbed by the loud volume of a television. The child may get up, and either request the television to be turned down, or turn down the television.
These examples show that smart homes may be not smart enough to enable seamless indirect interactions, causing inconvenience, repetitive actions, and tiresome processes that require more effort and time. Even though smart homes provide some opportunities to interact indirectly, users may still choose less efficient direct interactions.
Provided are methods, systems, and apparatuses for enabling seamless indirect interactions in an environment, wherein the system enables seamless indirect interactions among devices and users present in an Internet of Things (IoT) environment.
Also provided are methods, systems, and apparatuses for predicting at least one context from an utterance received from the user.
Also provided are methods, systems, and apparatuses for identifying the one or more second users related to the predicted context.
Also provided are methods, systems, and apparatuses for predicting current location of the one or more second users based on an IoT data.
Also provided are methods, systems, and apparatuses for correlating the obtained user context data and the environment context data from a database with the at least one context from the utterance.
Also provided are methods, systems, and apparatuses for providing at least one of an interaction and a suggestion to the one or more second users through an interactable interface using a deep learning method.
Also provided are methods, systems, and apparatuses for appending one or more inputs from the second users related to one task that is to be performed by the first user.
Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the presented embodiments.
In accordance with an aspect of the disclosure, a method for enabling indirect interactions between users in an Internet of Things (IoT) environment includes: receiving, by a first device, an utterance from a first user, wherein the utterance relates to at least one task that is to be performed by the first user; based on receiving the utterance, identifying, by the first device, one or more second users related to the at least one task; providing, by the first device, an interactable interface to one or more second devices which are located closer to the one or more second users than the first device; receiving, by the first device, one or more inputs corresponding to the at least one task from the one or more second users through the interactable interface; and appending, by the first device, the received one or more inputs to the at least one task.
The identifying the one or more second users related to the task may include predicting, by the first device, at least one context based on the utterance received from the first user; and identifying, by the first device, the one or more second users related to the predicted at least one context.
The predicting the at least one context and the identifying the one or more second users may be performed using a trained learning method.
The method may further include: obtaining, by the first device, IoT data; based on the IoT data, determining, by the first device, location history of at least one device which was last accessed by the one or more second users, based on the IoT data; and determining, by the first device, a current location of the one or more second users based on the location history.
The one or more second devices may be selected by the first device based on an availability of the one or more second devices, and a capability of the one or more second devices for performing at least one of delivering and receiving messages.
The providing the interactable interface to the one or more second devices may include generating at least one of an interaction and a suggestion to provide to the one or more second users, and the generating the at least one of the interaction and the suggestion may include: obtaining, by the first device, user context data and environment context data; and correlating, by the first device, the user context data and the environment context data with the predicted at least one context.
The method may further include: based on at least a portion of the user context data and the environment context data being matched with the predicted at least one context, generating, by the first device, at least one suggestion, wherein the at least one suggestion is provided based on data stored in the one or more second devices or is provided as a recommendation that is relevant to the predicted at least one context; and based on the at least the portion of the user context data and the environment context data being not matched with the predicted at least one context, generating, by the first device, at least one interaction for directly conveying a message based on the utterance.
The one or more inputs may include at least one of a requirement corresponding to the at least one task and an action content to be performed according to the at least one task.
In accordance with an aspect of the disclosure, a device for enabling indirect interactions among users in an Internet of Things (IoT) environment includes: at least one processor configured to: receive an utterance from a first user, wherein the utterance relates to at least one task that is to be performed by the first user; based on receiving the utterance, identify one or more second users related to the at least one task; provide an interactable interface to one or more target devices which are located closer to the one or more second users than the device; receive one or more inputs corresponding to the at least one task from the one or more second users through the interactable interface; and append the received one or more inputs to the at least one task.
The at least one processor may be further configured to: predict at least one context based on the utterance received from the first user; and identify the one or more second users related to the predicted at least one context.
The at least one processor may be further configured to perform the predicting of the at least one context and the identifying of the one or more second users using a trained learning method.
The at least one processor may be further configured to: obtain IoT data; determine location history of at least one device which was last accessed by the one or more second users, based on the IoT data; and determine a current location of the one or more second users based on the location history.
The at least one processor may be further configured to select the one or more target devices based on an availability of the one or more target devices, and a capability of the one or more target devices for performing at least one of delivering and receiving messages.
The at least one processor may be further configured to provide the interactable interface to the one or more target devices by generating at least one of an interaction and a suggestion to provide to the one or more second users, and to generate the at least one of the interaction and the suggestion, the at least one processor may be further configured to: obtain user context data and environment context data; and correlate the user context data and the environment context data with the predicted at least one context.
The at least one processor may be further configured to: based on at least a portion of the user context data and the environment context data being matched with the predicted at least one context, generate at least one suggestion, wherein the at least one suggestion is provided based on data stored in the one or more target devices or is provided as a recommendation that is relevant to the predicted at least one context; and based on the at least the portion of the user context data and the environment context data being not matched with the predicted at least one context, generate at least one interaction for directly conveying a message based on the utterance.
In accordance with an aspect of the disclosure, a system for enabling indirect interactions among users in an Internet of Things (IoT) environment includes: one or more target devices; and a source device comprising at least one processor configured to: receive an utterance from a first user, wherein the utterance relates to at least one task that is to be performed by the first user; based on receiving the utterance, identify one or more second users related to the at least one task; provide an interactable interface to the one or more target devices, wherein the one or more target devices are located closer to the one or more second users than the source device; receive one or more inputs corresponding to the at least one task from the one or more second users through the interactable interface; and append the received one or more inputs to the at least one task.
The above and other aspects, features, and advantages of certain embodiments of the present disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which:
The present disclosure and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques may be omitted so as to not unnecessarily obscure the embodiments herein. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein may be practiced and to further enable those of skill in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.
The embodiments herein may relate to methods, apparatuses, and systems for enabling seamless indirect interactions in an Internet of Things (IoT) environment, in which an engine enables seamless indirect interactions among devices and users present in the IoT environment. Referring now to the drawings, and more particularly to
The source device 104 may include a processor 110 and a communication module 112. The source device 104 may be a real world device present in the real world environment of the first user 102. Examples of the source device 104 may include a desktop computer, a laptop computer, a mobile device such as a smart phone, a personal digital assistant, a wearable device, a kitchen appliance, and a smart appliance, but embodiments are not limited thereto.
In an embodiment, the processor 110 may be configured to enable a device such as the source device 104 to gather information or requirements from one or target users based on one or more tasks. The task may be a task which is to be performed by the first user 102 and/or on behalf of the first user 102, and which may depend upon the input of the one or more target users. The processor 110 may determine the locations of the one or more target devices 106 and entities related to the task, and may provide suggestions to one or more target users, such that the target users may be used by at least one of the source device 104 and the first user 102 to make informed decisions. A plurality of modules may be utilized for interfacing of a device to one or more target devices.
In an embodiment, the processor 110 may include one or more of microprocessors, circuits, and other hardware configured for processing. The processor 110 may be configured to execute instructions stored in a database.
The processor 110 may be at least one of a single processer, a plurality of processors, multiple homogeneous or heterogeneous cores, multiple Central Processing Units (CPUs) of different kinds, microcontrollers, special media, and other accelerators. The processor 110 may be an application processor (AP), a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and/or an Artificial Intelligence (AI)-dedicated processor such as a neural processing unit (NPU).
In an embodiment, the communication module 112 may be configured to enable communication between the source device 104 and one or more target devices 106. The server may be configured or programmed to execute instructions of the first device 104. The communication module 112 through which the source device 104 and the server communicate may be in the form of either a wired network, a wireless network, or a combination thereof. Examples wired and wireless communication networks may include Global Positioning System (GPS), Global System for Mobile Communication (GSM), Local Area Network (LAN), Wireless Fidelity (Wi-Fi) compatibility, and Near-Field Communication (NFC), but embodiments are not limited thereto. Examples of the wireless communication may further include one or more of Bluetooth, ZigBee, a short-range wireless communication such as Ultra-Wideband (UWB), a medium-range wireless communication such as Wi-Fi, and a long-range wireless communication such as 3G/4G/5G or Worldwide Interoperability for Microwave Access (WiMAX), according to the usage environment, but embodiments are not limited thereto.
Although
The database may comprise one or more volatile and non-volatile memory components which are capable of storing data and instructions to be executed. Examples of the memory module may include NAND, embedded Multimedia Card (eMMC), Secure Digital (SD) cards, Universal Serial Bus (USB), Serial Advanced Technology Attachment (SATA), and a solid-state drive (SSD), but embodiments are not limited thereto. The memory module may also include one or more computer-readable storage media. Examples of non-volatile storage elements may include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. In addition, the memory module may, in some examples, be considered a non-transitory storage medium. The term “non-transitory” may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term “non-transitory” should not be interpreted to mean that the memory module is non-movable. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in Random Access Memory (RAM) or cache).
The intelligent generator module 210 may generate at least one interaction and suggestion to provide to the one or more second users through the interactable interface using a deep learning method. The intelligent generator module 210 may obtain user context data and environment context data from the database. Further, the intelligent generator module 210 may correlate the obtained user context data and the environment context data with at least one context from the utterance. Further, the intelligent generator module 210 may generate at least one suggestion if at least a portion of the obtained user context data and the environment context data match at least one context from the utterance. The intelligent generator module 210 may provide the suggestion based on data stored in the one or more target devices 106 in accordance with information related to the past search or prior history or as a recommendation, relevant to the predicted at least one context. The intelligent generator module 210 may generate at least one interaction if at least a portion of the obtained user context data and the environment context data does not match with at least one context from the utterance for directly conveying a message from the utterance.
The system 100 may enable the first user 102 to gather information or requirements from the one or more second users 108 based on one or more tasks. Further, the system 100 may append one or more inputs from the one or more second users 108 related to the task that is to be performed by the first user 102, wherein the one or more inputs may include at least one of the requirements and action content to be performed.
At operation 304, the method includes predicting, by the source device 104, at least one context from the utterance received from the first user 102.
At operation 306, the method includes identifying, by the source device 104, the one or more second users 108 related to the predicted at least one context, wherein the method of predicting the at least one context and identifying the one or more second users 108 are performed using a trained learning method, and the learning method is trained using data maintained in the database.
At operation 308, the method includes obtaining, by the source device 104, IoT data, wherein the IoT data may include at least one of the camera data, the user profile data, and the historical data from the database, but embodiments are not limited thereto. Using the IoT data, the source device 104 may determine location history of at least one last accessed device of the one or more second users 108. The source device may predict a current location of the one or more second users 108 based on the determined location history.
At operation 310, the method includes identifying one or more target devices 108 which are located closer to the one or more second users 108 than the first device 104, wherein the source device 104 selects the one or more target devices 106 based on availability and capability for performing at least one of delivering and receiving messages, using the learning method.
At operation 312, the method includes providing, by the source device 104, an interactable interface via one or more target devices 106 present in a location closer to the one or more second users 108, wherein the source device 104 generates at least one of an interaction and suggestion to provide to the one or more second users 108 through the interactable interface using a deep learning method. The method further includes obtaining, by the source device 104, a user context data and an environment context data from the database and correlating, by the source device 104, the obtained user context data and the environment context data with the at least one context from the utterance.
At operation 314, the method includes generating, by the source device 104, at least one suggestion based on at least a portion of the obtained user context data and the environment context data being matched with the at least one context from the utterance.
At operation 316, the method includes generating, by the source device 104, at least one interaction based on the at least a portion of the obtained user context data and the environment context data being not matched with the at least one context from the utterance, for directly conveying a message from the utterance.
At operation 318, the method includes receiving, by the source device 104, one or more inputs corresponding to the task from the one or more second users 108 through the interface, and appending, by the source device 104, the received one or more inputs to the task that is to be performed by the first user 102.
One or more of the operations of method 300 described above may be performed in the order presented, in a different order, or simultaneously. Further, in some embodiments, some actions listed in
The processor 110 may control the processing of the input data in accordance with a predefined operating rule or Artificial Intelligence (AI) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning.
Here, being provided through learning may mean that a predefined operating rule or AI model of a desired characteristic is made by applying a learning algorithm to a plurality of learning data. The learning may be performed in a device itself in which AI according to an embodiment is performed, and/or may be implemented through a separate server/system.
The learning algorithm may be a method for training a predetermined device (for example, a robot) using a plurality of learning data to cause, allow, or control the device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.
In the example shown herein, the mom 502 provides an utterance asking her kids 508a-508d about their requirements, if any, as “Hey, kids, I am going shopping.” The system 100 provides the suggestion or the interaction to her kids 508a-508d through the interactable interface as “Your Mom is going shopping”. In bedroom 1, the system 100 provides the suggestion, “Should I tell her to buy an on-sale Samsung Galaxy Monitor with more screen size and refresh rate than your current monitor?” Kid 508a replied, “Yes”. In bedroom 2, kid 508b replied, “Ask Mom to get my laundry.” In bedroom 3, kid 508c and 508d replied, “We want Chocolates! Also, If Mom asks about chocolates in the fridge, tell her we finished those.” The system 100 appends the received input from the target devices 106 (associated with kids 508a-508d) to the source device 104 (associated with Mom 502) through the interactable interface as “Add Samsung Galaxy Monitor to Shopping List,”, “Create a reminder to bring kid 508b's Laundry”, “Add Chocolates to Shopping List,”, and “Create Action: If asked about “Chocolate in the fridge”, Then respond “Kids finished it,” respectively. Therefore, in the example shown herein, the system 100 provides suggestions only to one user (e.g., kid 508a) as a recommendation.
The context may be shopping, ordering a meal, preparing food, paying rent, and other things. In the example shown herein, the context identified as shopping.
In the example shown herein, after categorization, Mom 502 may be considered as a source entity or a source user 102, and kids 508a-508d may be considered as a target entity or one or more target users 108.
In the example shown herein, the location detector module 206 identifies the first user 102 (e.g., Mom 502) is in living room and the second users 108 are in various bedrooms (e.g., kid 508a is in bedroom 1, kid 508b is in bedroom 2, and kids 508c and 508d are in bedroom 3).
In bedroom 1, the target device selector module 208 may select a smart monitor as an input and output device. In bedroom 2, the target device selector module 208 may select a smart speaker as an input and output device. In bedroom 3, the target device selector module 208 may select a smart speaker as an input device and a smart TV as an output device.
The intelligent generator module 210 may provide at least one interaction and suggestion to the target users 108 through the interactable interface. The intelligent generator module 210 may not involve unintended entities. In the example shown herein, the husband may be in the kitchen, and may not be a target entity.
Probability Function (X:(User Context, Environment Context)|Y:(Utterance Context))={Suggestion, if X & Y Correlate} {Interaction, if X & Y Don't Correlate}. Equation 1:
In the example discussed above, the context (e.g., shopping) may be identified from the received input/utterance from the first user 102 (e.g., Mom 502) as “Kids I am going shopping”. Further, the environment context (e.g., Sale: (Samsung shop/Summer sale)) may be determined based on the identified context (e.g., shopping) as the one or more second users have been determined to be in different bedrooms.
In the case of kid 502a context, the obtained user context data and the environment context data may be matched with the one context from the utterance, and the system may generate the suggestion as a recommendation through the identified devices (monitors: old, small screen, and low resolution) as “Your Mom is going shopping. Should I tell her to buy the on-sale Samsung Galaxy Monitor with more screen size and refresh rate than your current monitor?” In the case of user ‘contexts for kids 502b-502d, the obtained user context data and the environment context data may be not matched with the one context from the utterance, so the system 100 provides interactions only in place of suggestions.
Embodiments herein enable conduct of concurrent interactions, wherein simultaneous multifarious interactions are possible in parallel. Embodiments herein enable ease of use, wherein context or specification of entities are not required explicitly. Embodiments herein enable a reduction of human efforts, wherein the system 100 suggests and executes actions based on prior context which reduces human effort. Embodiments herein provide a quicker execution, wherein the time taken to complete a task will be less than doing manually. Embodiments herein reduce the cognitive load, wherein the system 100 will provide most relevant suggestions based on the context and priority.
The various actions, acts, blocks, steps, or the like in the method 300 may be performed in the order presented, in a different order or simultaneously. Further, in some embodiments, some of the actions, acts, blocks, steps, or the like may be omitted, added, modified, skipped, or the like without departing from the scope of the disclosure.
The embodiments disclosed herein may be implemented through at least one software program running on at least one hardware device and performing network management functions to control the elements. The elements may be at least one of a hardware device, or a combination of hardware device and software module.
The foregoing description of the specific embodiments is not intended to be limiting, and the embodiments described above may be modified and/or adapted for various applications without departing from the generic concept. Therefore, such adaptations and modifications should and are intended to be comprehended within the meaning, range, and scope of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the particular embodiments are described above in terms of at least one embodiment, those skilled in the art will recognize that the embodiments herein may be practiced with modification within the spirit and scope of the embodiments as described herein.
Number | Date | Country | Kind |
---|---|---|---|
202241051693 | Sep 2022 | IN | national |
202241051693 | Aug 2023 | IN | national |
This application is a bypass continuation of International Application No. PCT/KR2023/013180, filed on Sep. 4, 2023, which is based on and claims priority to Indian Provisional Application No. 202241051693, filed on Sep. 9, 2022, in the Indian Intellectual Property Office, and Indian Complete Patent Application No. 202241051693, filed on Aug. 14, 2023, in the Indian Intellectual Property Office, the disclosures of which are incorporated by reference herein in their entireties.
Number | Date | Country | |
---|---|---|---|
Parent | PCT/KR2023/013180 | Sep 2023 | US |
Child | 18517995 | US |