Patent Application
20160062473
The present invention relates to computer systems. More specifically, the present invention relates to a gesture-controlled computer system, and a gesture-based method for controlling a computer system.
Many businesses increasingly rely on mobile computing devices (e.g., portable computing devices) to provide computer resources to their employees, particularly employees whose job responsibilities require them to be on the move. For instance, package delivery companies frequently equip employees with mobile computing devices that allow the transport and delivery of packages to be tracked in real-time by user-input data. A warehouse worker may also utilize a forklift's integrated mobile computing device to track inventory stored in the warehouse, and to quickly determine the location of stored items to be transported using the forklift. Many of these mobile computing devices incorporate indicia-reading technology to enable efficient and accurate data entry.
Earlier mobile computing devices frequently received input through integrated keyboards. The keyboards were often miniaturized, at least to some extent, to maintain the portability of the mobile computing device. Integrated mouse technology, sometimes in the form of an integrated track ball or similar peripheral device, was also used to assist with receiving commands from the user. More recently, touchscreens have been incorporated to integrate device input and device display. While touchscreens are beneficial in maintaining the portability of the device and facilitating user input, there are instances where touchscreen input can be problematic.
While medical providers, such as hospitals, have come to rely on mobile computing devices, including indicia readers, to assist doctors and nurses as they make their rounds from patient to patient, hospitals remain concerned about the transmission of germs. Because touchscreens require tactile interaction between the user and the device, touchscreens tend to transmit germs interpersonally when operated by more than one user.
The spread of germs and other contaminants is also of particular concern to the food industry. Restaurant workers may find it difficult or cumbersome to maintain clean hands while performing tasks that involve both handling food and operating computer-based systems (e.g., kitchen display systems, point-of-sale systems, etc.).
Some jobs require an employee to wear gloves. Employees that work in extremely cold environments (e.g., cold climates or refrigerated environments) use gloves to maintain hand warmth. Harsh or hazardous work environments, such as where workers handle chemicals or sharp materials, may also require workers to use protective gloves. Touchscreen devices tend to perform poorly, if at all, when the user is wearing gloves.
Therefore, a need exists for a computer system capable of touchless control. In particular, a need exists for a computer system that can be controlled by gestures performed by a user and recognized by the system.
Accordingly, in one aspect, the present disclosure embraces a gesture-controlled computer system. The gesture-controlled computer system includes a mobile computing device, a range camera, and a gesture-control subsystem. The gesture-control subsystem is in communication with the mobile computing device and the range camera. The gesture-control subsystem includes a gesture-capturing module that is configured for acquiring a first-taken range image of an articulating-gesture-controller during the performance of a gesture. The gesture-control subsystem also includes a range-image-analysis module configured for generating a gesture-data-point profile derived at least in part from the first-taken range image. The gesture-data-point profile includes information regarding a plurality of gesture data points, including information about the relative position of the gesture data points. The gesture-control subsystem also includes a gesture-recognition module configured for (i) receiving the gesture-data-point profile from the range-image-analysis module, (ii) mapping the gesture-data-point profile to a corresponding device command, and (iii) transmitting the corresponding device command to the mobile device for execution of the device command by the mobile device.
In one embodiment, the gesture-capturing module is configured for acquiring a second-taken range image of the articulating-gesture-controller during the performance of a gesture. The range-image-analysis module is also configured for including in the gesture-data-point profile information regarding the relative motion of the plurality of gesture data points, the relative motion being determined at least in part by comparing the first-taken range image to the second-taken range image.
In another embodiment, the information regarding the relative motion of the plurality of gesture data points includes the direction of movement and velocity of the plurality of gesture data points.
In yet another embodiment, the gesture-data-point profile includes information regarding the orientation of the articulating-gesture-controller.
In yet another embodiment, the gesture-data-point profile includes information regarding the positional configuration of a plurality of positionable segments of the articulating-gesture-controller.
In yet another embodiment, the gesture-data-point profile includes information regarding the positional configuration of at least three positionable segments of the articulating-gesture-controller.
In yet another embodiment, the gesture-data-point profile includes information regarding the positional configuration of at least four positionable segments of the articulating-gesture-controller.
In yet another embodiment, the gesture-data-point profile includes information regarding the positional configuration of at least five positionable segments of the articulating-gesture-controller.
In yet another embodiment, the articulating-gesture-controller is a human hand, and each of the plurality of positionable segments is a different finger of the human hand.
In yet another embodiment, the articulating-gesture-controller is a portion of a human upper torso, and each of the plurality of positionable segments corresponds to a different one of an upper arm, a lower arm, a hand, or one of the fingers of the hand.
In another aspect, the present disclosure embraces a method for controlling a mobile computer device. A gesture is presented within a range camera's field-of-view using an articulating-gesture-controller. A first-taken range image of the articulating-gesture-controller is acquired. A gesture-data-point profile derived at least in part from the first-taken range image is acquired. The gesture-data-point profile includes information regarding a plurality of gesture data points, including information about the relative position of the gesture data points. The gesture-data-point profile is mapped to a corresponding device command. The corresponding device command is transmitted to the mobile computing device for execution of the device command by the mobile computing device.
The foregoing illustrative summary, as well as other exemplary objectives and/or advantages of the disclosure, and the manner in which the same are accomplished, are further explained within the following detailed description and its accompanying drawings.
The present disclosure embraces a gesture-controlled computer system. The gesture-controlled computer system utilizes a gesture-control subsystem to permit a user to command a mobile computing device using gestures that are recognizable by the gesture-control subsystem. The gestures are made with an articulating-gesture-controller.
As shown in
The gesture-controlled computer system 100 also includes a range camera 107. For the purposes of this disclosure, a range camera includes any three-dimensional imaging device that is capable of depicting the three-dimensional space within the range camera's field-of-view in the form of a two-dimensional image. The two-dimensional image that provides a representation of the three-dimensional space, including representations of objects within that three dimensional-space, is typically a digital image (e.g., a computer-readable image file). An image acquired by the range camera 105 is referred to as a range image. Typically, the range image associates each image pixel with a distance value. Typically, the association of an image pixel with a distance value provides an indication of the distance of the portion of the object represented by the image pixel from the range camera. The image pixel may display differently (e.g., on a display device) depending upon the associated distance value.
The range camera 107 may employ any of a variety of techniques for generating a range image. Without intending to limit the disclosure, examples of such techniques include stereo triangulation using a stereo camera system, sheet of light triangulation, structured light illumination, time-of-flight, interferometry, ultrasound, passive infrared, and coded aperture techniques. The range camera 107 may include more than one image-capturing device (e.g., camera).
Typically, the range camera 107 may be configured to allow a user to adjust the settings of the range camera 107. For example, the system may include a configuration utility that allows a user to change the range camera's settings, including frame rate, resolution, filter settings, range settings, etc. The configurable nature of the range camera 107 allows a user to obtain the best quality images for promoting the system's ability to recognize gestures.
The gesture-controlled computer system 100 also includes a gesture-control subsystem 110. The gesture-control subsystem 110 is in communication (e.g., electronic communication, wireless communication, etc.) with the mobile computing device 105 and range camera 107. For example, the gesture-control subsystem 110 is typically configured to receive range images from the range camera 107 and transmit device commands to the mobile device 105. The gesture-control subsystem 110 typically includes software (e.g., application software) stored in a memory store, the software being executable by a processor for carrying out the functions of the gesture-control-subsystem 110.
The-gesture-control subsystem 110 may be integral with the mobile computing device 105 and/or the range camera 107. In one embodiment, the mobile computing device 105 features an integrated range camera 107 and gesture-control subsystem 110. Alternatively, the gesture-control subsystem 110 may be, in whole or in part, remote from the mobile computing device 105 and/or the range camera 107. For example, the mobile computing device 105 may have an integrated range camera 107 to take a range image of the user's gestures. The range image(s) may be transmitted via a communications network (e.g., local area network, wide area network, personal area network, Internet, etc.) to a remote server computer that hosts the gesture-control subsystem 110 (or aspects thereof) for processing. The various components of the system 100 may be connected to the communications network via wired or wireless technology.
The gesture-control-subsystem 110 includes a gesture-capturing module 120. The gesture-capturing module 120 is configured for acquiring a first-taken range image 122 of an articulating-gesture-controller 126. (References in this disclosure to a first-taken range image should be understood to mean a range image that depicts the range camera's field-of-view at a point in time that is prior to the point in time depicted in a second-taken range image. A second-taken range image depicts a point in time prior to a third-taken range image, and so on.) The first-taken range image 122 is typically acquired by the range camera 107 by positioning the articulating-gesture-controller 126 within the field-of-view of the range camera 107. Typically, the gesture-capturing module 120 receives the first-taken range image 122 (e.g., a range image in the form of an electronic signal or a computer-readable image file) from the range camera 107. The gesture-capturing module 120 usually stores the first-taken range image 122 in the memory store, where it can be accessed for processing purposes.
The articulating-gesture-controller (AGC) 126 is utilized by a user of the gesture-controlled computer system 100 to make a gesture to control the gesture-controlled computer system 100. Typically, the AGC includes a positionable segment 127. More typically, the AGC includes a plurality of positionable segments 127. A positionable segment 127 is a portion of the AGC that can be configured by the user in more than one physical configuration (e.g., an extended position, a retracted position, a semi-retracted position, a rotated position, a curled position, etc.). The positionable segments 127 allow the AGC to be configured in multiple physical configurations, thereby enabling the user to create multiple different gestures. In addition, one or more positionable segments 127 may be moved, alone or in conjunction with the entire AGC, as a gesture. In other words, the gesture may be defined at least in part by the configuration of one or more positionable segments 127 and/or by the relative motion of one or more positionable segments 127.
As shown in
As discussed below, the gesture-controlled computer system 100 can, in one embodiment, recognize the motion of objects within the field-of-view of the range camera. The ability to recognize motion greatly increases the richness of the system's gesture vocabulary. In other words, by allowing the user to incorporate motion into gestures, the system provides an ability to recognize a much larger number of gestures than could be utilized without this ability.
As an example of a gesture that incorporates motion, a user may extend and then retract a finger 127A, which the system may recognize as a gesture that is distinct from a hand with an extended, motionless finger. By way of further example, the user may move the entire human hand 126A within the field-of-view of the range camera 107. The system may associate different gestures depending upon the direction of movement. For example, moving the human hand 126A upward may be associated with one gesture, and moving the human hand 126A downward may be associated with a different gesture. Similarly, moving the human hand 126A toward the range camera 107 may be associated with a different gesture from the gesture associated with moving the human hand 126A away from the range camera. The system may associate unique movements with a particular gesture. For example, moving the human hand 126A in a circular motion may represent a distinct gesture.
As stated above, the multi-configurability of the AGC 126, combined with the system's ability to recognize motion as a distinct element of a given gesture, allows for a rich vocabulary of gestures. The previous example of a circular hand movement can be used to illustrate this richness. Depending upon whether zero, one, two, three, four, or five fingers of the human hand were extended at the time of the circular hand motion, the user has the ability to create at least six distinct gestures based on that one motion alone. Additional gestures could be created using the same circular hand motion by incorporating movement of one or more fingers at the same time. For example, moving the entire human hand in a circular motion while continuously extending and retracting all five fingers could represent a distinct gesture.
It will be appreciated by a person of ordinary skill in the art that there are a large number of gestures that could be created (e.g., performed) utilizing an AGC 126 according to the present disclosure. The exemplary gestures described herein are for illustrative purposes only, and are not intended to limit the type or number of gestures that may be performed using the AGC 126, nor are they otherwise intended to limit the present disclosure.
Similarly, the description of a human hand 126A as an exemplary AGC 126 is not intended to limit the disclosure to a particular type of AGC 126. For instance, the AGC could also be a robotic hand or other inanimate object having the characteristics of an AGC 126 as disclosed herein. The AGC 126 could also be another portion of the human body. For example, the AGC 126 could be a portion of a human upper torso. Typically, this would allow the user to perform a gesture using, for example, an arm. Exemplary positionable segments in this exemplary scenario could be an upper arm, a lower arm, a hand, or one of the fingers of the hand. An exemplary gesture that a user could perform would be extending the upper arm horizontally and moving the lower arm back-and-forth in a substantially vertical position with only the thumb and little finger of the hand extended and the remaining fingers in a retracted position.
An exemplary AGC may also incorporate aspects of the human body (e.g., human hand) and an inanimate object (e.g., stylus). For example, a user may perform various gestures depending on the positioning of both the user's hand(s) and/or finger(s) as well as the position or location of a stylus within the user's hand (e.g., positioning the stylus between the thumb and first finger is one gesture, and positioning the stylus between the fourth and fifth fingers is another gesture). Similarly, an exemplary AGC may incorporate more than one body part (e.g., two hands) that the user can use to perform coordinated gestures (e.g., displaying two closed fists is a single gesture). In other words, an exemplary AGC may include two or more distinct objects that can be displayed together to perform gestures.
Returning now to
Typically, the system creates gesture data points 132 that are associated with distinct features of the AGC 126 that are utilized in the presentation of different gestures. For example, where the AGC 126 is a human hand, the system may associate one gesture data point with each of the five fingers. The system may optionally associate a gesture data point 132 with the palm of the hand. By establishing one gesture data point for a palm and one gesture data point for each of the five fingers, the system is able to determine the relative position of each of these component parts. For example, the system in this exemplary embodiment would be able to determine whether the little finger was above or below the middle finger in the three-dimensional space that is represented by the range image. The system would similarly be able to determine, by comparing the position of the palm gesture data point with the finger gesture data points, whether the hand was extended toward or away from the range camera.
It will be appreciated by a person of ordinary skill in the art that the greater number of gesture data points 132 that the system associates with the AGC 126, the greater the system's ability to distinguish between gestures having refined differences. For example, to distinguish between a finger that is extended versus one that is retracted, the system may need to assign one gesture data point to each of the finger's three joints (e.g., first knuckle, second knuckle, and third knuckle). Greater numbers of gesture data points 132 may be created to resolve even subtle differences between gestures (e.g., a cupped hand versus a flat hand). However, the employment of more gesture data points 132 may come at the expense of system efficiency (e.g., in terms of memory usage and processing time).
The gesture-data-point profile 135 includes information regarding all of the gesture data points 132 recognized by the range-image-analysis module 130. The information may typically include the position of the gesture-data-point 132 in the three-dimensional space captured by the range camera 107, as well as any additional information required to determine position of each gesture data point 132 vis-à-vis other gesture data points 132. Furthermore, the gesture-data-point profile 135 may include information from which the system could determine the position, orientation, and or movement of the AGC 126 and/or its positionable segments 127. Typically, the gesture-data-point profile 135 is included in an electronic data file that is processed by the system in lieu of processing a larger, more electronically-cumbersome range image (e.g., range image electronic image data file).
The system also includes a gesture-recognition module 140. The gesture-recognition module 140 is configured for (i) receiving the gesture-data-point profile 135 from the range-image-analysis module 130, (ii) mapping the gesture-data-point profile 135 to a corresponding device command, and (iii) transmitting the corresponding device command to the mobile computing device for execution of the device command by the mobile computing device.
Typically, the system stores in non-volatile memory a library of gestures (e.g., a gesture library). The gesture library contains information regarding each of the gestures that are recognizable by the system. Typically, this information includes information regarding the relative position and relative motion, if any, of the gesture-data-points that make up a given gesture. To identify what gesture is represented by a given gesture-data-point profile 135, the gesture-recognition module 140 compares the gesture-data-point profile 135 to the gesture-data-point profiles stored in the gesture library. If a gesture-data-point profile stored in the gesture library matches the gesture-data-point profile 135 received from the range-image-analysis module 130, then the gesture-recognition module 140 maps the recognized gesture to the device command that corresponds with the matching gesture in the gesture library. The gesture-recognition module 140 then transmits the device command that corresponds to the recognized gesture to the mobile computing device 105 for execution of the device command by the mobile computing device 105. A device command may represent be a unitary command, or it may be a multi-step command. In other words, a device command may, in certain embodiments of the present disclosure, instruct the mobile computing device 105 to take a series of actions in a specified order. The system may require, for example, that a user perform a unique gesture (e.g., a security gesture or login gesture) to access the system. The login gesture may be in lieu of, or in addition to, traditional login techniques such as alphanumeric passwords.
In an exemplary embodiment, the system acquires a second-taken range image 124 to allow the system to determine the motion of the AGC and/or any constituent positionable segments 127. The gesture-capturing module 120 is configured for acquiring a second-taken range image 124 depicting the AGC 126. Using image-motion-processing techniques, the system compares aspects of the first-taken range image 122 with aspects of the second-taken range image 124 to determine what portions, if any, of the AGC 126 moved between the time of the first-taken range image 122 and the time of the second-taken range image 124. Typically, the comparison of the first-taken range image 122 with the second-taken range image 124 includes a comparison of the gesture-data-points 132 associated with each range image. The resulting information regarding the movement depicted in the range images is typically incorporated in the gesture-data-point profile 135 that the system uses to identify the gesture being presented to the range camera 107.
It will be appreciated by a person of ordinary skill in the art that the system may acquire multiple range images to assist in determining the particular kinetic motion that may be exhibited by the AGC 126. For example, the system may acquire and process twenty-four range images per second to determine the movements of the AGC 126 and/or its constituent positionable segments 127. It will also be appreciated that the system may through analysis acquire information regarding not only the direction of motion of the AGC 126 and/or the positionable segments 127, but also the velocity associated with the motion. In this way, the system can potentially distinguish between gestures (or gesture components) presented at differing speeds. For example, the system may identify a slower-waving hand as a gesture distinct from a gesture represented by a faster-waving hand.
The gesture-controlled computer system 100 according to the present invention may have a pre-populated gesture library. In other words, the system may be configured to recognize at least one gesture without the need for any initialization by the user. In one embodiment, the system is configured to allow the user to customize (e.g., through a “learning mode”) the gesture library. To customize the gesture library, the user performs the gesture (which may include a series of discrete positionings and/or movements of the AGC) to allow the gesture-control subsystem 110 to create a gesture-data-point profile that represents the gesture to be added to the gesture library. The user then associates the newly-added gesture with a device command, for example by manually executing the device command using another input on the mobile computing device (e.g., a keyboard, a touchscreen, a mouse, etc.). The system subsequently stores in the gesture library the association between the new gesture and the device command input by the user. This customization feature effectively allows a user to create a customized library of gestures (e.g., a language of gestures), each gesture associated with a different device command.
In addition to allowing the user to add new gestures to the gesture library, the system may also allow a user to dynamically change the association (e.g., mapping) between a particular gesture and a device command. If a given gesture is associated with a first device command, for example, the user may direct the system to associate that gesture with a second, different device command. Furthermore, the system may allow a user to map more than one device command to a given gesture. The device command that is actually executed is typically determined by the context in which the gesture is performed. In other words, in cases where more than one device command is mapped to a single gesture, when that gesture is performed the system executes the device command that is appropriate given the present status of the workflow. The system is typically configured to allow the user to map these device commands to gestures in the field, thereby providing significant flexibility to the user.
In another exemplary embodiment, the system 100 is configured to allow a user to create a digital signature by performing a signature gesture within the range camera's field of view. For example, to sign for the receipt of a package, a package recipient could be instructed by the system user to mimic the movements used in signing the recipient's name. The alternative embodiment of the system is configured to track the movements of at least a portion of the recipient's hand to acquire the recipient's electronic signature. Alternatively, the system is configured to track the movements of at least a portion of a stylus that the signing package recipient manipulates (e.g., holds while moving the hand) in a signature movement.
In another exemplary embodiment, the system 100 may permit the initiation of device commands via a combination of gestures and other input techniques. For example, a user may initiate a device command by performing a gesture before, during, or after providing a voice command received through the system's microphone.
In another exemplary embodiment, the gesture-control subsystem 110 is configured for initiating a device command associated with a given gesture upon the user's performance of an execution signal. Typically, an execution signal is a gesture (e.g., hand gesture), such as a stationary closed fist, or open hand closing into a closed fist. In other words, after recognizing a gesture associated with a particular device command, the system may not actually initiate the device command until the system recognizes that an execution signal has been performed. Typically, the execution signal is performed immediately subsequent to the gesture (e.g., command gesture). For example, after a user performs a gesture, she may perform an execution signal indicating that she now wishes the system to initiate the device command associated with the performed gesture. The execution signal may also be a voice command from the user. For example, the user may perform and gesture and then, when satisfied that the system has recognized the correct gesture, say “execute”, thereby causing the system to initiate the associated device command.
In this way, the system's ability to identify an execution signal allows the user to control whether or not the system performs the device command associated with the gesture as recognized by the system. This feature advantageously allows the user to have greater control over when the device command is initiated. For example, if the system indicates to the user that it has recognized the gesture to be something other than what the user expected (e.g., the user swiped two fingers to the left and the system indicated it recognized three fingers swiping to the left), the user can prevent the initiation of an undesired device command by not performing an execution signal. The user may repeat the gesture until the gesture is recognized by the system as the user intended, at which time the user may perform an execution signal to indicate to the system that system should execute the device command associated with the last-recognized gesture.
In another exemplary embodiment, the gesture-control subsystem 110 is configured for ignoring a gesture (or partial gesture) that precedes (e.g., immediately precedes) a user's performance of a cancellation signal. When the gesture-control subsystem detects that the user has performed a cancellation signal, the system disregards the gesture or partial gesture that the user was performing just prior to the cancellation signal. The cancellation signal may be a gesture (e.g., hand gesture), a user-spoken command, or other type of input (e.g., pressing a button). For example, if a user performs half of a desired gesture and then decides to initiate a different command or no command, the user can restart the process by performing the cancellation signal. The cancellation signal prompts the system to disregard the previous gesture attempt and prepare for a new gesture. A typical cancellation signal may include moving the AGC 126 away from the range camera 107 until it is out of the nominal range. In other words, where the system may typically recognize gestures performed within a certain distance range from the range camera (e.g., within a specified z-coordinate range), when the user moves the AGC 126 outside of that range, the system will recognize that as a cancellation signal.
In another exemplary embodiment, the system 100 is configured to recognize gestures only when certain conditions are met. For example, the user may toggle between a gesture mode and a standard input mode (e.g., entering device commands through other input devices such as a keyboard, a touchscreen, a microphone, etc.). Alternatively, the system 100 may automatically enter into gesture mode under certain conditions. For example, the system 100 may be configured to enter gesture mode only when a specific, triggering gesture is presented by the user. For example, when a user makes a gesture that includes extending three fingers, the system 100 enters gesture mode. When less than two fingers are extended, however, the system 100 stays in mouse mode. In mouse mode, the user's hand and/or finger movements can be used to control the mobile computing device's display screen in a manner similar to as if a touchscreen or trackpad were being used. Alternatively, the system 100 automatically enters gesture mode when the system detects the presence of an AGC at predetermined distance (e.g., greater than three inches) from the range camera. In this way, the system 100 can be configured to allow conventional touchscreen input when a user's hand is close to the mobile computing device's display screen, but to automatically switch to gesture mode when the user's hand is not determined to be close to the display screen.
In another exemplary embodiment, the system 100 may be configured for providing the user with a visual representation of the device command initiated by the user's performance of a recognized gesture. The mobile computing device's display (e.g., LCD display) may provide onscreen visualization of inputted device commands using a combination of dynamic cursors and/or backgrounds. The system may also provide the user with audio (e.g., aural) feedback to indicate events such as gesture start, gesture end, and change of modes (e.g., from gesture mode to mouse mode). In this regard, the system 100 may include a user-feedback module 145 for providing gesture-related feedback to a user in response to the performance of a gesture.
The gesture-related feedback provided by the user-feedback module may include an identification of the performed gesture. In other words, the gesture-related feedback may identify for the user the specific gesture recognized by the system when the user performed a particular gesture. By providing gesture-related feedback, the system enables the user to determine whether or not the system is appropriately recognizing the performed gesture and/or to determine whether the user is appropriately performing the desired gesture.
The gesture-related feedback may include a visual response (e.g., graphic response or output). For example, if a user holds up two fingers and swipes them to the right, the user-feedback module 145 may cause the system's display screen (e.g., LCD monitor) to display “2→”. By viewing this visual output, the user may confirm that the system properly understood (e.g., identified) the performed gesture. Alternatively, the visual response may include the illumination of a light or the illumination of a light in a particular manner (e.g., a pre-determined on-off sequence and/or duration, a pre-determined color or sequence of colors, etc.).
The gesture-related feedback may include an aural response (e.g., audio response or output). For example, if a user points her little finger toward the range camera and then moves her little finger in a circular manner, the user-feedback module may cause the system's speaker to emanate a particular sound associated with that gesture (e.g., three tones in series). The sound associated with a particular gesture may include a voice recording. The voice recording may state the nature of the performed gesture (e.g., “little finger in circular motion”) or, more typically, it may state the command associated with the performed gesture (e.g., “go to main menu”).
In an alternative embodiment, the gesture-controlled computer system according to the present disclosure includes a performance module 150. The gesture performance module 150 is configured to analyze the user's performance of a gesture to determine how well the user is performing the gesture. This analysis can allow the system to detect instances when the user is performing a gesture(s) inappropriately, thereby presenting an opportunity to correct the user's performance. For example, at the beginning of a work shift, a user may perform a given gesture appropriately, such as through the use of substantially precise movements. As the shift wears on, fatigue may cause the user to use less precise movements in the performance of a gesture. When a user performs a gesture improperly, it may result in unwanted delays as the system attempts to recognize the gesture. By detecting when a user is performing a gesture improperly (e.g., imprecisely, sloppily, etc.), the performance module 150 can either notify the user (e.g., through a graphic display or audio signal) to improve the performance of the gesture, or it can notify a third party that is responsible for the user's performance (e.g., a manager). Additionally, the performance module's analysis can provide insight into the nature and timing of worker fatigue. If the performance module 150 detects a pattern of gestures that are impeded by fatigue, it may prompt management to consider and potentially remedy the cause of the fatigue. For example, the fatigue may be caused by a gesture that requires too great a degree of dexterity and/or endurance to be performed repeatedly over the course of an entire work shift. An alternative, less taxing gesture may need to be considered. This ability for the performance module 150 to detect fatigue may be used generally to determine when a user (or entire workforce) becomes fatigues, and to take appropriate measures to reduce the impact of the fatigue (e.g., better timing of breaks).
In an alternative embodiment, the system 100 includes an articulating-gesture-controller-identifier module 155. The articulating-gesture-controller-identifier module 155 identifies an articulating-gesture-controller within a range image. Typically, the articulating-gesture-controller-identifier module 155 employs traditional image analysis algorithms (e.g., edge detection algorithms, curve detection algorithms, etc.) in determining whether a particular range image includes an image of an AGC, including a human hand. Typically, the articulating-gesture-controller-identifier module 155 includes a series of templates matching an AGC in various configurations (e.g., open hand seen from the top, closed hand seen from the side, partial palm, etc.). One technique involves identifying whether the range image includes fingertips. Although this technique is generally effective, it may be prone to false identifications because other objects may initially appear to be fingertips. This technique may be refined by applying a cascading system of AGC identification. Using this cascading technique, the articulating-gesture-controller-identifier module 155 first analyzes the range image to determine the presence of a shape consistent with a human hand. When a human hand is identified, the articulating-gesture-controller-identifier module 155 then looks for fingertips. By initiating a search for fingertips only after identifying, for example, a shape consistent with a hand, the system avoids devoting resources to a search for fingertips, which can consume substantial amounts of time and/or processing power.
In another aspect, the present disclosure embraces a gesture-controlled computer system that includes a wearable-gesture-controller. The wearable-gesture-controller is configured to detect a user's movements (e.g., through integrated accelerometers) in lieu of, or supplementary to, a range camera. The wearable-gesture-controller may, for example, take the form of a ring that can be worn on the user's finger. The ring can detect movements in the user's hand and/or finger. The user can, therefore, perform gestures using the hand bearing the ring, which gestures may be identified and acted upon according to the systems and methods disclosed herein.
In another aspect, the present disclosure embraces a method for controlling a mobile computing device. A gesture is presented (e.g., made, created, etc.) using an articulating-gesture-controller. The gesture is presented within a range camera's field-of-view. A first-taken range image of the articulating-gesture-controller is acquired. The first-taken range image depicts the articulating-gesture-controller during the performance of the gesture. A gesture-data-point profile is generated. The gesture-data-point profile is derived at least in part from the first-taken range image. The gesture-data-point profile includes information regarding a plurality of gesture data points, including information about the relative position of the gesture data points. The gesture-data-point profile is mapped to a corresponding device command. The corresponding device command is transmitted to the mobile computing device for execution of the device command by the mobile computing device.
In another aspect, the present disclosure embraces a gesture-controlled kitchen display system. The food industry, and in particular the restaurant industry, has in recent years largely moved toward the utilization of kitchen display systems (e.g., kitchen automation system). Traditionally, the provision of restaurant services relied heavily upon paper transactions. A food server would write down a patron's order and submit the written order to the food preparers (e.g., cooks). The written order, or a carbon copy of the written order, would also be used at the point-of-sale to determine the amount owed by the customer.
As technology evolved, the restaurant industry looked to ways to reduce reliance on written orders, which can be costly and inefficient. Ultimately the industry turned to kitchen display systems. Kitchen display systems are computer-based (e.g., processor-driven) systems that permit electronic processing of food orders and payment. Rather than providing the food preparers with a written order (e.g., a ticket), the server enters the order into the kitchen display system via a user input (e.g., keyboard, mouse, touchscreen, etc.). Food preparers view the orders on a display device (e.g., LCD monitor). When the food preparer finishes making the food, he or she can indicate that the order is ready by selecting an input on the kitchen display system.
Because traditional input devices (e.g., QWERTY keyboard, mouse, etc.) can be too tedious for the fast-paced conditions at a restaurant, alternative means of entering information into the kitchen display system were developed. The bump bar is one such device that has enjoyed widespread use. A bump bar is an input device that has a simplified interface (e.g., relatively few command buttons). A typical bump bar has a button(s) for selecting a particular food order ticket or chit. A bump bar typically also has a button for initiating a “bump,” which is an indication that a particular food order is ready to be served to a customer. A bump bar may have other functionality, including a recall button that allows for a previously-bumped order(s) to be returned to the display.
Bump bars may be stand-alone input device (e.g., breakaway boxes) or integrated components of a traditional QWERTY keyboard. Alternatively, and of greater prevalence in recent years, bump bars may be a virtual component of the kitchen display system's touchscreen display. In this configuration, typically the bottom portion of the touchscreen is devoted to a visual representation of a bump bar where the user can select a given function by touching the corresponding button image on the touchscreen.
Regardless of configuration, bump bars share the common feature that they require tactile input from the user (e.g., pushing a button, touching the screen, etc.). This presents a problem in the context of food services because of the importance of cleanliness. The bump bar, as well as other components of the kitchen display system, may become dirty over time, especially upon frequent use by multiple different workers performing different jobs. When a worker touches a bump bar, a touchscreen, or other physical component of the kitchen display system, the worker potentially compromises the cleanliness of his or her hands as well as the touched component. In addition to the cleanliness aspect, the requirement of tactile input in these kitchen display system can be time-consuming despite the fact that they are designed to be more efficient than traditional methods.
The present disclosure's gesture-controlled kitchen display system allows for servers, food preparers, and other users to input information into the kitchen display system without making physical contact with any component of the kitchen display system. Referring now to
The gesture-controlled kitchen display system 100A according to the present disclosure also includes a range camera 107 having a field of view comprising an area of three-dimensional space. The gesture-controlled kitchen display system 100A also includes a gesture control subsystem 110 in communication with the processor 103 and the range camera 107. The gesture-control subsystem includes a gesture-capturing module 120 and a range-image-analysis module 130. The gesture controlled kitchen display system 100A also includes a gesture-recognition module 140. The gesture-recognition module 140 is configured for (i) receiving the gesture-data-point profile from the range-image-analysis module, (ii) mapping the gesture-data-point profile to a corresponding kitchen-display-system command, and (iii) transmitting the corresponding kitchen-display-system command to the processor 103 for execution of the kitchen-display-system command by the processor 103. A kitchen-display-system command is any command (e.g., directive to the kitchen display system) that can be initiated by a user input.
The gesture-controlled kitchen display system 100A allows a user to interact with the kitchen display system with gestures in lieu of (or in addition to) actions that require tactile interaction. For example, rather than bumping the first three tickets on the display screen by touching all three tickets and then touching the “bump” button, a food preparer could make the gesture corresponding to the bump kitchen-display-system command (e.g., a fist, moving a first in a circular motion, etc.), followed by holding up three fingers. By way of further example, whereas a traditional bump bar may have arrow buttons allow the user to scroll the screen and/or selection indicator in various directions, a user may instead achieve the same input by pointing a finger up, down, left, right, etc.
In another aspect, the present disclosure embraces a gesture-controlled point-of-sale system. Point-of-sale systems (e.g., POS, electronic point-of-sale, EPOS) are typically computerized systems positioned at the location where a customer makes payment to a merchant in exchange for goods or services. Point-of-sale systems are intended to improve efficiency and accuracy in the purchase transaction, such as by utilizing indicia readers to record an item's purchase price (as opposed to a clerk reading a price tag and manually entering the price). In the restaurant industry, in particular, the use of point-of-sale systems can be problematic. For instance, because of potential contaminants on the surface of the point-of-sale system (e.g., on the touchscreen), a worker may be required to repeatedly wash his or her hands (or don new gloves) every time he or she touches the point-of-sale system. This can result in wasted time and/or wasted money (e.g., through purchasing multiple latex gloves per day). This problem can be alleviated at least in part by utilizing the gesture-controlled point-of-sale system according to the present disclosure.
Referring now to
The gesture-controlled point-of-sale system 100B according to the present disclosure also includes a range camera 107 having a field of view comprising an area of three-dimensional space. The gesture-controlled point-of-sale system 100A also includes a gesture control subsystem 110 in communication with the processor 103 and the range camera 107. The gesture-control subsystem includes a gesture-capturing module 120 and a range-image-analysis module 130. The gesture controlled kitchen display system 100A also includes a gesture-recognition module 140. The gesture-recognition module 140 is configured for (i) receiving the gesture-data-point profile from the range-image-analysis module, (ii) mapping the gesture-data-point profile to a corresponding point-of-sale-system command, and (iii) transmitting the corresponding point-of-sale-system command to the processor 103 for execution of the point-of-sale-system command by the processor 103. A point-of-sale-system command is any command (e.g., directive to the point-of-sale system) that can be initiated by a user input.
The gesture-controlled point-of-sale system 100B allows a user to interact with the point-of-sale system with gestures in lieu of (or in addition to) actions that require tactile interaction. For example, rather using a keypad to select menu items (e.g., cheeseburger) purchased by a patron, a user may make the gesture that corresponds to the menu item (e.g., one hand on top of the other to represent a hamburger). Upon making a gesture that causes the gesture-controlled point-of-sale system 100B to provide a total amount due, the patron may initiate payment (e.g., via credit card, NFC-enabled smartphone, etc.). In that scenario, by using gestures, the user does not come into physical contact with the equipment of the gesture-controlled point-of-sale system or the customer's payment method.
In another aspect, the present disclosure embraces a gesture-controlled food preparation system. Commercial food preparation occurs at many levels. Restaurant food preparers cook meals in kitchens, deli workers prepare sandwiches behind deli counters, butchers prepare cuts of meat for purchase by grocers, and so on. With respect to commercial food preparation, an importance is placed upon cleanliness of both the hands of the food preparers as well as the equipment involved in the food preparation (e.g., slicers, cooking utensils, containers, etc.). As such, when a worker comes into physical contact with equipment during food preparation, it may be necessary for the worker to wash his or her hands prior to continuing with food preparation to ensure cleanliness. For example, a deli worker who touches a slicer may have to wash his or her hands before continuing to make a sandwich. This can result in delays for the business and patron.
The present disclosure's gesture-controlled food preparation system allows food preparers other users to operate the food preparation system without making physical contact with any component of the food preparation system. In this context, food preparation system refers to any equipment or machinery utilized in food preparation or processing, such as scales, slicers, cookers, ovens, grinders, sausage makers, pasta makers, labelers, packagers (e.g., vacuum sealers), bread slicers, roasters, fryers, heat lamps, etc. Referring now to
The gesture-controlled food preparation system 100C according to the present disclosure also includes a range camera 107 having a field of view comprising an area of three-dimensional space. The gesture-controlled food preparation system 100C also includes a gesture control subsystem 110 in communication with the processor 103 and the range camera 107. The gesture-control subsystem includes a gesture-capturing module 120 and a range-image-analysis module 130. The gesture controlled food preparation system 100C also includes a gesture-recognition module 140. The gesture-recognition module 140 is configured for (i) receiving the gesture-data-point profile from the range-image-analysis module, (ii) mapping the gesture-data-point profile to a corresponding food-preparation-system command, and (iii) transmitting the corresponding food-preparation-system command to the processor 103 for execution of the kitchen-display-system command by the processor 103. A food-preparation-system command is any command (e.g., directive to the food preparation system) that can be initiated by a user input (e.g., increase oven temperature, open oven door, start/stop slicer, etc.).
The gesture-controlled food preparation system 100C allows a user to interact with the food preparation system with gestures in lieu of (or in addition to) actions that require tactile interaction.
In another aspect, the present disclosure embraces a gesture-controlled equipment-safety system. Particularly in industrial settings, life, limb, and property can be placed in jeopardy when equipment malfunctions or other anomalies are encountered (e.g., a worker becomes entrapped in equipment). To minimize injury to persons and property, many types of equipment (e.g., industrial equipment) incorporate safety systems. These safety systems may include cutoff switches and alarm-activators. These equipment-safety systems may be integral to the industrial equipment, or may be stand-alone systems often found in proximity to the equipment (e.g., halon systems in server rooms). In some instances, it may be difficult for a worker to activate an equipment-safety system in the event of an emergency. For example, the body position of the worker, or the worker's partial incapacitation, may make it difficult to press a cutoff switch.
The present disclosure's gesture-controlled equipment-safety system allows industrial workers and other users to operate the equipment-safety system without making physical contact with any component of the equipment-safety system. Referring now to
The gesture-controlled equipment-safety system 100D according to the present disclosure also includes a range camera 107 having a field of view comprising an area of three-dimensional space. The gesture-controlled equipment-safety system 100D also includes a gesture control subsystem 110 in communication with the processor 103 and the range camera 107. The gesture-control subsystem includes a gesture-capturing module 120 and a range-image-analysis module 130. The gesture controlled equipment-safety system 100D also includes a gesture-recognition module 140. The gesture-recognition module 140 is configured for (i) receiving the gesture-data-point profile from the range-image-analysis module, (ii) mapping the gesture-data-point profile to a corresponding equipment-safety-system command, and (iii) transmitting the corresponding equipment-safety-system command to the processor 103 for execution of the equipment-safety-system command by the processor 103. An equipment-safety-system command is any command (e.g., directive to the equipment-safety system) that can be initiated by a user input (e.g., initiate fire suppression, shut down power, activate safemeasure such as safety guards, etc.). The gesture-controlled equipment-safety system 100D allows a user to interact with the equipment-safety system with gestures in lieu of (or in addition to) actions that require tactile interaction.
Embodiments of the present disclosure that have been described as incorporating a mobile computing device alternatively may incorporate non-mobile (e.g., stationary) computing devices such as desktop computers, point-of-sale terminals, and appliances such as smart televisions or game consoles. The gesture-controlled computer system of the present disclosure may be integrated into another machine, device, and/or system, as in the case of a gesture-controlled point-of-sale system. Alternatively, the gesture-controlled computer system may be an add-on device or system that, when connected, coupled, or integrated with another machine, device, and/or system, provides the gesture-based functionality disclosed herein. In other words, the gesture-controlled computer system disclosed herein may be an independent system capable of being coupled (e.g., communicatively coupled) to a separate machine, device and/or system. The coupling may be achieved in any acceptable manner, including via a Universal Serial Bus interface or similar I/O interface.
Stand-alone embodiments of the present disclosure typically are advantageously configured with plug-and-play capability, thereby alleviating the need for a user to configure the host to interface with the system. For example, when the system is connected to the host, the system enumerates on the PC bus as a standard keyboard or standard mouse. In some embodiments, the present disclosure embraces a gesture software module that can be executed on various computer-based systems to facilitate the control of such computer-based system using gestures. Typically, the gesture software module is operating system agnostic, platform agnostic, and/or range camera agnostic. This allows the gesture software module to be employed substantially universally across varied types of computer-based systems.
To supplement the present disclosure, this application incorporates entirely by reference the following patents, patent application publications, and patent applications:
In the specification and/or figures, typical embodiments of the invention have been disclosed. The present invention is not limited to such exemplary embodiments. The use of the term “and/or” includes any and all combinations of one or more of the associated listed items. The figures are schematic representations and so are not necessarily drawn to scale. Unless otherwise noted, specific terms have been used in a generic and descriptive sense and not for purposes of limitation.
The present application claims the benefit of U.S. Patent Application No. 62/043,728 for a Gesture-Controlled Computer System filed Aug. 29, 2014. The present application claims the benefit of U.S. Patent Application No. 62/083,566 for a Gesture-Controlled Computer System filed Nov. 24, 2014. The present application claims the benefit of U.S. Patent Application No. 62/183,385 for a Gesture-Controlled Computer System filed Jun. 23, 2015. Each of the foregoing patent applications is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 62043728 | Aug 2014 | US | |
| 62083566 | Nov 2014 | US | |
| 62183385 | Jun 2015 | US |
