Patent Application
20230034682
This application claims the benefit of U.S. patent application Ser. No. 17/165,031, filed Feb. 2, 2021, which is incorporated by reference herein in its entirety.
The present application relates generally to augmented reality, and more particularly to the use of augmented reality to train or teach a person how to complete a task.
Instruction manuals are commonly used to teach a user how to complete a task, such as assembling a product. One challenge with instruction manuals is that they are hard to understand for various reasons. For example, instructions may be poorly written so that they are unclear, overly complicated, or filled with unfamiliar jargon. Instruction manuals may not be in a language that the user fully understands. Another issue is that instruction manuals may not provide images of every step that a user needs to complete. In the past a solution might be to produce a video featuring a person completing the task with verbal instructions detailing each step to the user. One common problem with this (and with traditional instruction manuals) is that the instructions are presented from an unnatural viewpoint for the user, and the user is unable to see how their body is supposed to move to complete the task. Instruction manuals and videos are typically presented with a front view as opposed to a back view. In a front view, the user sees another person complete a task. In a back view, the user has the same view as when the user performs the task. Another issue for both instruction manuals and instruction videos is that the user receives no feedback on if they have correctly completed the step. Therefore, improvements are desirable.
In one aspect of the present disclosure, a method of improving a visual instruction during running of a visual instruction sequence includes playing a visual instruction sequence to a user from a point-of-view of the user; monitoring the user for user data related to the visual instruction sequence, using the user data to improve the visual instruction sequence; generating an improved visual instruction sequence; and playing the improved visual instruction sequence to the user from a point-of-view of the user.
In another aspect of the present disclosure, a system for improving a visual instruction during running of a visual instruction sequence includes a computer device for playing a visual instruction sequence to a user from a point-of-view of the user, monitoring the user for user data related to the visual instruction sequence, using the user data to improve the visual instruction sequence, generating an improved visual instruction sequence; and playing the improved visual instruction sequence to the user from a point-of-view of the user.
For a more complete understanding of the disclosed system and methods, reference is now made to the following descriptions taken in conjunction with the accompanying drawings.
Instruction manuals and videos allow users to perform tasks that they have little to no prior knowledge about or experience with. Instruction manuals have several issues. Instruction manuals can be long and make a task appear daunting. Instruction manuals can be hard to understand. They may be poorly written or be in a language that the user is not comfortable with. Instruction manuals can include images, but these images are often presented from a front view rather than a back view. A front view can cause confusion as the user must orient themselves to the image and determine if the right side of the image corresponds to the user's right side or the user's left. The user is unable to see how their body is supposed to move to complete the task. Also, the instruction manual may not provide images of every step, requiring the user to guess. Instruction manuals also lack the ability to provide feedback to the user about whether the user has successfully completed steps to the task or if the user has made an error that needs correction. Instructional videos can overcome some of these issues by demonstrating tasks to the user. However, instructional videos do not overcome all the challenges. Instructional videos are typically presented from a front view and have no ability to provide feedback. Augmented Reality can be used to overcome these issues.
Augmented Reality (“AR”) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory. AR allows users to have an interactive experience of a real-world environment where objects in the real world are enhanced by computer generated perceptual information. AR has three basic features: (1) a combination of real and virtual worlds, (2) real-time interaction, and (3) accurate 3D registration of real and virtual worlds. AR technology works by taking in the real-world environment and digitally manipulating it to include or exclude objects, sounds, and other things perceivable to the user. AR systems use various hardware components including a processor, a display or output devices, and input devices. Input devices may include sensors, cameras, microphones, accelerometers, GPS systems, and solid-state compasses. Modern mobile devices such as smartphones and tablet computers contain these elements.
The present disclosure teaches a system that uses AR to train a person how to complete a task. Task is broadly defined. Examples of a task include assembling, dissembling or repairing a product, playing a video game and completing an exercise routine. Tasks can be manually selected by the user or identified by the system via a smart search. For example, the user takes a picture of the product with an app. Based on the picture, the system can identify the product. Once the system has identified the object or task, it queries a knowledgebase for any and all resources related to the object or task, for example, user manuals, service manuals, how-to-videos, exploded diagrams, blueprints, other user comments, etc. Because the system is reading the instructions and diagrams and interpreting the information for the user, the system can help users who have trouble reading the instructions (because the font is too small, bad vision, lighting conditions, language difficulties, etc.) The system also helps to locate things that are not readily visible on the object being addressed, e.g., on the bottom.
The system uses the information stored in the knowledgebase to create AR patterns that instruct the user how to perform a task using an avatar of the user's body. In the above example of the product picture, the system would create AR patterns that instruct the user how to assemble, repair or dissemble the product. The AR pattern is displayed to the user by the system. The user follows the instructions provided by the avatar to complete the task. In some embodiments, the system could be configured to evaluate the user's performance and notify the user of any errors made. For example, if the AR pattern contains sound, the system will match the actual sound to the correct sound in the pattern and notify the user. If the AR pattern contained eye goggles for safely, the system would look for safety goggles on the user.
Once an AR pattern has been created, the system stores the AR pattern so that it can produce an AR pattern more efficiently when the same or similar task is identified in the future. The system uses artificial intelligence (“AI”) to improve and update AR patterns based on, among other things, user input and common errors experienced by users over time. AR patterns may also be retained by users for future use.
Referring to
Once the system has identified the task, at 208, the AR system queries the knowledgebase. The knowledgebase contains existing AR patterns as well as many documents including written instructions, diagrams, and other sources. At 210, the AR system develops an AR pattern. If an AR pattern does not exist, the system develops an AR pattern for completing the task using the documents in the knowledgebase. The AR pattern can include video, pictures, spoken instructions, background noise (such as hammering), etc.
If an AR pattern already exists, the AR system looks to develop an improved AR pattern using feedback from last use the AR pattern, user comments and other resources. Preferably, the AR pattern also uses actual pictures or video submitted by the user at 204. Each AR pattern is tailored to the current, specific task identified. For example, perhaps the nail is seated crooked in the picture submitted in 108 of
The AR system can determine the AR pattern from exploded diagrams or blueprints.
The AR system can use an existing video to develop the AR pattern. For example, from a video of the user assembling a product, an AR pattern can be created. The AR system can then create the reverse as well for dissembling the product. The AR pattern can show appropriate tools for the task or disable a machine before a task. The AR system can use laws of science and math to improve manufacturer's instructions. The AR pattern can include sounds and listen for the correct sounds, for example hammering of a nail by the user. The AR system can then verify that it is hearing the correct sound. Sound verification can be used as an accessibility feature for the hard of hearing.
At 212, using the AR pattern, the system instructs the user how to perform the task using an avatar of the user's body. In the example of
Using
The app then creates an avatar of the user's body and displays it along with the user's real image. Using the avatar, the app shows the user how the user should look after picking up the blade. The user moves her body to match this position; the app monitors the user's movements and tell her when she is in a position, which is close enough. The app can show the user from various viewpoints, such as looking down, looking in a mirror or a forward view of the user. The app slowly beings to nudge the avatar to perform the operation. As the user moves her arm, the movement is detected and transposed onto the avatar's movement. The app can follow the users lead to determine how fast the avatar should move. If the user makes a mistake, the app can instruct the user on the mistake to try to correct it. The app indicates when the task the completed and asks the user whether she wants to save the interaction. In an example of shooting a basketball, the user may use the AR pattern over and over again until the user develops a perfect shooting form.
Referring to
The user device 300 also contains memory 304. The memory 304 may include random access memory (“RAM”), which may be synchronous RAM (“SRAM”), dynamic RAM (“DRAM”), or the like. The user device 300 may utilize memory 304 to store the various data structures used by a software application. The memory may also contain include read only memory (“ROM”) which may be PROM, EPROM, EEPROM, optical storage, or the like. The ROM may store configuration information for booting the user device 300. The memory 304 holds user and system data and may be randomly accessed.
The user device 300 includes a communications adapter 306. The communications adaptor 306 may be adapted to couple the user device 300 to a network, which may be one or more of a LAN, WAN, and/or the Internet. The communications adapter 306 may also be adapted to couple the user device 300 to other networks such as a GPS or Bluetooth network. The communications adopter 306 may allow the user device 300 to communicate with an edge hosted knowledgebase.
The user device 300 also includes a display 308. The display device 308 allows the user device to display images, video, and text to the user. The display device may be a smartphone or tablet computer screen, an optical projection system, a monitor, a handled device, eyeglasses, a head-up display (“HUD”), a bionic contact lens, a virtual retinal display, and another display system known in the art.
The user device 300 also includes at least one input/output (“110”) device 310. The I/O devices allow the user to interact with the user device. I/O devices include cameras, video cameras, microphones, touch screens, keyboards, computer mice, accelerometers, global positioning systems (“GPS”), compasses, gyroscopes and other similar devices known to those of skill in the art.
Referring to
In one embodiment, the user interface device 510 is referred to broadly and is intended to encompass a suitable processor-based device such as user device 300, a desktop computer, a laptop computer, a personal digital assistant (PDA) or tablet computer, a smartphone, a gaming system such as a Sony PlayStation or Microsoft Xbox, or another mobile communication device having access to the network 508. The user interface device 510 may be used to access a web service executing on the server 502. When the device 510 is a mobile device, sensors (not shown), such as a camera or accelerometer, may be embedded in the device 510. When the device 510 is a desktop computer the sensors may be embedded in an attachment (not shown) to the device 510. In a further embodiment, the user interface device 510 may access the Internet or other wide area or local area network to access a web application or web service hosted by the server 502 and provide a user interface for enabling a user to enter or receive information.
The network 508 may facilitate communications of data, such as dynamic license request messages, between the server 502 and the user interface device 510. The network 508 may include any type of communications network including, but not limited to, a direct PC-to-PC connection, a local area network (LAN), a wide area network (WAN), a modem-to-modem connection, the Internet, a combination of the above, or any other communications network now known or later developed within the networking arts which permits two or more computers to communicate.
In one embodiment, the user interface device 510 accesses the server 502 through an intermediate sever (not shown). For example, in a cloud application the user interface device 510 may access an application server. The application server may fulfill requests from the user interface device 510 by accessing a database management system (DBMS). In this embodiment, the user interface device 510 may be a computer or phone executing a Java application making requests to a JBOSS server executing on a Linux server, which fulfills the requests by accessing a relational database management system (RDMS) on a mainframe server.
The computer system 600 also may include random access memory (RAM) 608, which may be synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), or the like. The computer system 600 may utilize RAM 608 to store the various data structures used by a software application. The computer system 600 may also include read only memory (ROM) 606 which may be PROM, EPROM, EEPROM, optical storage, or the like. The ROM may store configuration information for booting the computer system 600. The RAM 608 and the ROM 606 hold user and system data, and both the RAM 608 and the ROM 606 may be randomly accessed.
The computer system 600 may also include an input/output (I/O) adapter 610, a communications adapter 614, a user interface adapter 616, and a display adapter 622. The I/O adapter 610 and/or the user interface adapter 616 may, in certain embodiments, enable a user to interact with the computer system 600. In a further embodiment, the display adapter 622 may display a graphical user interface (GUI) associated with a software or web-based application on a display device 624, such as a monitor or touch screen.
The/O adapter 610 may couple one or more storage devices 612, such as one or more of a hard drive, a solid state storage device, a flash drive, a compact disc (CD) drive, a floppy disk drive, and a tape drive, to the computer system 600. According to one embodiment, the data storage 612 may be a separate server coupled to the computer system 600 through a network connection to the I/O adapter 610. The communications adapter 614 may be adapted to couple the computer system 600 to the network 608, which may be one or more of a LAN, WAN, and/or the Internet. The communications adapter 614 may also be adapted to couple the computer system 600 to other networks such as a global positioning system (GPS) or a Bluetooth network. The user interface adapter 616 couples user input devices, such as a keyboard 620, a pointing device 618, and/or a touch screen (not shown) to the computer system 600. The keyboard 620 may be an on-screen keyboard displayed on a touch panel. Additional devices (not shown) such as a camera, microphone, video camera, accelerometer, compass, and or gyroscope may be coupled to the user interface adapter 616. The display adapter 622 may be driven by the CPU 602 to control the display on the display device 624. Any of the devices 602-622 may be physical and/or logical.
