Patent Application
20160267755
The present application relates to object detection and localized extremity guidance.
Recently, smart watches have been a very active area of research and development and various companies have released capable wrist-worn computers. For the blind, these wearable smart devices can be used to communicate event-based knowledge. For example, a watch can vibrate on the hour to indicate the passage of time, or vibrate in response to an incoming phone call instead of ringing. Vibration around the wrist is generally un-intrusive, but still informative.
The type of vibration used to alert a user can also be varied to convey different information. In U.S. Pre-grant publication 2006/0129308 A1 (US308), a system is described where different vibrations convey limited information about different types of objects detected in the environment using RFID tag-readers, such as identification codes identifying the objects. This information is relayed to a computer which makes use of it, such as alerting the user to the existence of a dangerous condition (e.g., a fire) using vibration. However, while US308 discusses the possibility of using the information from the RFID modules to ascertain the direction of travel, speed, and path of the user, US308 does not disclose any particular methods for computing the speed and path of the user, or using the path computation to guide an extremity of a user to and/or around object(s). Rather, US308 is limited to generally describing using RFID tags to define a grid, which is used to track a user's general movements.
Some tactile belt systems employ a haptic interface around the waist of a user for communicating directions in new environments and thus guiding people, such as those who are visually impaired, along arbitrarily complex paths. These systems are designed to work with robot(s) as a guide, which can detect obstacles or potentially other locations of interest using existing techniques, and guide the user to designated areas. However, the directional feedback provided by such a system is not localized and thus lacks adequate directional definition in some cases. In addition, such a guide robot can be complex and require extensive setup, training, and maintenance.
A system described by J. Rempel, “Glasses That Alert Travelers to Objects Through Vibration? An Evaluation of iGlasses by RNIB and AmbuTech,” AFB AccessWorld Magazine, vol. 13, no. 9, Sep. 2012 (Rempel), uses glasses configured to alert the visually impaired about objects using vibration. These glasses detect objects that may be in the path of the user using ultrasound, and vibrate to indicate their proximity and left or right direction relative to the objects. However, the system described by Rempel is inadequate for localized guidance as it provides even less well-defined directional information than the above belt system.
Thus, the above-described systems are limited to providing coarse-grained navigational assistance which is unsuitable for localized guidance, such as guiding a hand to a particular target. Furthermore, they are not integrated with object detection to find specific objects of interest a user may want to grab, which is not a trivial task.
In a related area, some systems use vibrotactile feedback for human-robot interaction, such as leader-follower scenarios involving multiple robots as described by S. Scheggi, F. Chinello, and D. Prattichizzo, “Vibrotactile haptic feedback for human-robot interaction in leader-follower tasks,” in PETRA, Crete Island, Greece, 2012 (Scheggi). In particular, Scheggi demonstrates how bracelets equipped with three vibro motors worn by the human leader of a human robot team can be used to improve team cohesion. The robots track the human's path, velocity, and expected trajectory, and warn the human when his/her motion would make following impossible. However, the system described by Scheggi does not guide the human to a particular location using vibration, but rather constrains his/her motions based on robot feedback. In addition, the system is incapable of detecting objects and/or guiding a person to those objects, and is thus not suitable for localized guidance applications.
Various techniques also exist for teaching people new motor skills for use in sports training, dancing, fixing bad posture, or some forms of physical therapy, such as therapy provided after the occurrence of a stroke. Traditionally, a trainer would watch a given pupil and give the pupil feedback including spoken directions, visual demonstrations, and manually moving the pupil's limbs into the right location. But paying for such training or therapy is expensive and unattainable for many people. As a result, researchers have begun developing computerized haptic interfaces to guide a person's motion, and therefore increase the quality and consistency of the training, and the number of people who have access to it.
Currently, researchers are investigating which haptic interface is the most suitable for teaching motor skills. For instance, the system described in J. Lieberman and C. Breazeal, “TIKL: Development of a Wearable Vibrotactile Feedback Suit for Improved Human Motor Learning,” IEEE Transactions on Robotics, vol. 23, no. 5, pp. 919-926, October 2007, uses vibration motors placed about the wrist and upper arm of a given person to help that person achieve the desired positioning of his/her arm. The system uses a room-mounted computer vision system configured to track the person's arm relative to a given orientation and uses vibration to help position the arm along the appropriate axis. In this way, a person's arm could be pushed into the right location using vibration.
Another system described by F. Sergi, D. Accoto, D. Compolo, and E. Guglielmelli, “Forearm orientation guidance with a vibrotactile feedback bracelet: On the directionality of tactile motor communication,” in Proc. of the Int. Conf. on Biomedical Robotics and Biomechatronics, Scottsdale, Ariz., 2008, pp. 133-138, uses vibrotactile feedback with a single bracelet to guide an arm along a trajectory, and explores various configurations for such a bracelet. Other implementations, such as those described by A. L. Guinan, N. C. Hornbaker, M. N. Montandon, A. J. Doxon, and W. Provancher, “Back-to-back skin stretch feedback for communicating five degree-of-freedom direction cues,” in IEEE World Haptics Conference, Daejeon, Korea, 2013 and K. Bark, J. Wheeler, P. Shull, J. Savall, and M. Cutkosky, “Rotational Skin Stretch Feedback: A Wearable Haptic Display for Motion,” IEEE Transactions on Haptics, vol. 3, no. 3, pp. 166-176, July 2010, use skin stretch instead of vibration as another modality for guiding hands into a desired rotational form. The system described by M. F. Rotella, K. Guerin, Xingchi He, and A. M. Okamura, “HAPI Bands: A haptic augmented posture interface,” in HAPTICS Symposium, Vancouver, BC, 2012 uses a set of five bands placed around a person's wrists, elbows and waist, which guide the person to a correct posture (e.g., Yoga) in response to a computer vision-based analysis. This system uses a Kinect™ camera to estimate the person's body skeleton and generate vibrational error.
While some of above haptic systems may be designed to use vibratory feedback to achieve particular postures and/or actions, these systems lack the capability to detect objects in the environment and provide localized navigational assistance to the user to reach out and manipulate detected objects.
Technology for localized guidance of a body part of a user to specific objects within a physical environment using a vibration interface is described.
According to one innovative aspect of the subject matter described in this disclosure, a system includes a vibration interface wearable on an extremity by a user. The vibration interface includes a plurality of motors. The system includes sensor(s) coupled to the vibrotactile system and a sensing system coupled to the sensor(s) and the vibration interface. The sensing system is configured to analyze a physical environment in which the user is located for a tangible object using the sensor(s), to generate a trajectory for navigating the extremity of the user to the tangible object based on a relative position of the extremity of the user bearing the vibration interface to a position of the tangible object within the physical environment, and to guide the extremity of the user along the trajectory by vibrating the vibration interface.
The system and further implementations may each optionally include one or more of the following features. For instance, the system may include: an input device configured to receive input data from the user indicating the tangible object; that the input device is coupled to the sensing system to communicate data reflecting the tangible object to the sensing system; that the one or more sensors are further configured to receive transponder signals from the tangible object; that the transponder signals include identification data identifying the tangible object; that the sensing system is executable by the one or more processors to determine a unique identity of the tangible object based on the identification data; that the one or more sensors include a perceptual system configured to capture image data including images of the physical environment and any objects located within the physical environment; that the perceptual system is coupled to the sensing system to provide the image data including the images; that the sensing system is further configured to process the image data to determine a location of the tangible object; that the sensing system is further configured to determine the relative position of the extremity of the user bearing the vibration interface; that the sensing system is further configured to sense movement of the extremity by the user using the one or more sensors, to re-determine the relative position of the extremity of the user to the tangible object responsive to sensing the movement, to update the trajectory for navigating the extremity of the user to the tangible object based on a change to the relative position of the extremity of the user to the tangible object, and to guide the extremity of the user along the updated trajectory, as updated, using the vibration interface; that the vibration interface includes a plurality of motors; that the sensing system is further configured to determine a vibratory pattern for vibrating one or more of the motors of the vibration interface based on the trajectory generated for navigating the extremity of the user to the tangible object and to vibrate the vibration interface by vibrating the one or more motors according to the vibratory pattern to convey the direction for movement of the extremity to reach the tangible object; a motor control unit configured to vibrate the motors of the vibration interface according to the vibratory pattern determined by the sensing system; that the motor control unit is coupled to the sensing system to receive control signals.
In general, another innovative aspect of the subject matter described in this disclosure may be embodied in methods that include: analyzing a physical environment in which a user bearing a vibration interface is located for a tangible object using one or more sensors; determining a relative position of the extremity of the user bearing the vibration interface to a position of the tangible object within the physical environment; generating a trajectory for navigating the extremity of the user to the tangible object based on the relative position of the extremity to the position of tangible object; and guiding the extremity of the user along the trajectory by vibrating the vibration interface.
The method and further implementations may each optionally include one or more of the following features. For instance, the system may include: sensing movement of the extremity by the user using the one or more sensors; responsive to sensing the movement, re-determining the relative position of the extremity of the user to the tangible object; updating the trajectory for navigating the extremity of the user to the tangible object based on a change to the relative position of the extremity of the user to the tangible object; guiding the extremity of the user along the trajectory, as updated, by vibrating the vibration interface; that determining the relative position of the extremity includes determining an orientation of the extremity using sensing data captured by the one or more sensors, the sensing data reflecting movement of the extremity by the user, calculating a ray originating from a predetermined point of the extremity and extending through a predetermined point of the tangible object, and calculating an angular offset 8 between the orientation of the extremity and the ray; that generating the trajectory for navigating the extremity of the user to the tangible object is based on the angular offset; that the vibration interface includes a plurality of motors and guiding the extremity of the user along the trajectory includes vibrating one or more of the motors of the vibration interface; that determining a vibratory pattern for vibrating the one or more of the motors of the vibration interface based on the trajectory generated for navigating the extremity of the user to the tangible object; that guiding the extremity of the user along the trajectory by vibrating the vibration interface further includes vibrating the one or more motors according to the vibratory pattern to convey the direction for movement of the extremity to reach the tangible object; that the vibratory pattern includes one or more of linear motion and rotational dimensions and vibrating the one or more motors of the vibration interface includes vibrating the one or more motors based on the one or more of the directional and rotational dimensions of the vibratory pattern to convey the direction for movement of the extremity to reach the tangible object; identifying a bit sequence and vibration intensity value for the one or more motors; that the bit sequence and vibration intensity value reflect the one or more of the directional and rotational dimensions of the vibratory pattern; that vibrating the one or motors of the vibration interface includes vibrating the one or more motors using the bit sequence and vibration intensity value; determining that the extremity of the user has reached the position of the tangible object within the physical environment; terminating vibrating the vibration interface to cease guiding the extremity of the user; that the position of the tangible object is fixed or variable; that determining the relative position of the extremity of the user to the position of the tangible object within the physical environment includes determining a central position of the tangible object, determining a centroid of the vibration interface, and calculating the relative position based on a distance between the central position of object and the centroid of vibration interface; identifying an obstacle within the physical environment between the tangible object and the extremity of the user using the one or more sensors; that generating the trajectory for navigating the extremity of the user to the tangible object based on the relative position of the extremity to the position of tangible object is further based on a path that circumnavigates the obstacle; that analyzing the physical environment in which the user is located for the tangible object using one or more sensors includes locating the tangible object within the physical environment using a visual perception system; detecting an operational problem associated with the vibration interface; determining a unique vibratory pattern for the operational problem; vibrating the vibration interface based on the unique vibratory pattern; receiving input from user via an input device providing assistance to address the operational problem; and receiving input data from a user via an input device indicating the tangible object as an object of interest.
Other aspects include corresponding methods, systems, apparatus, and computer program products for these and other innovative aspects.
The technology described by the present disclosure may be particularly advantageous in a number of respects. For instance, the technology is capable of guiding an extremity of a user, such as the user's hand, to a given object in the environment. This is beneficial, and in some cases critical, when the target subject is blind, visually impaired, or otherwise incapable of discerning the surrounding environment or objects clearly. In a further example, the technology may guide the user's hand to detected objects out of the user's line-of-sight, providing guidance for difficult to see objects, or as a teaching guide for explaining manipulation tasks with new or unknown objects.
The above list of features and advantages is not all-inclusive and many additional features and advantages are within the scope of the present disclosure. Moreover, it should be noted that the language used in the present disclosure has been principally selected for readability and instructional purposes, and not to limit the scope of the subject matter disclosed herein.
The disclosure is illustrated by way of example, and not by way of limitation in the figures of the accompanying drawings in which like reference numerals are used to refer to similar elements.
Vision is an important aspect of reaching for, and grasping an object. A person depends on their eyes to provide feedback on relative positioning, and identify how to move their hand. For a visually impaired person, this feedback is missing either completely or to a significant degree. To find the small everyday objects that he or she uses, a visually impaired person generally depends on those objects being located in the same place. Dishes, for example, are always put back in the same spot, and it is important to not fall behind on putting them away. When the person is in a new environment or is looking for objects that get moved around by other people in the environment, the person is relegated to spend a significant amount of time groping around with his/her hands or asking for assistance.
This disclosure describes novel technology for assisting users to find objects using localized guidance. The technology combines real-time computational object detection with vibration (e.g., haptic) interface(s) worn on the extremit(ies) (e.g., wrist, ankle, neck, waist, etc.) that guide the extremit(ies) to the object location(s) using vibration. In an example, the vibration interface can guide the hand of a person to a detected object in the environment, and, although the person may not be able to perceive the object (e.g. they are blind, line-of-sight is obstructed, they are unfamiliar with the object, etc.), the technology detects the object relative to the person's hand and then it guides the hand to the object of interest by providing vibratory feedback through motors embedded in a wristband.
The technology includes a sensing system 100 configured to determine the relative position of vibration interface(s) worn by the user to target object(s).
The part(s) of the user's body on which the vibration interface(s) are worn are referred to herein as extremities. Example extremities include a wrist, ankle, knee, leg, arm, waist, neck, head, prosthetic, assistive device (e.g., cane), or other appendage, etc. The vibration interface(s) are configured to provide dynamic, real-time sensory feedback to the user. The implementations described herein use vibrational feedback produced by motors included in the vibration interface(s), although it should be understood that other types of feedback produce by suitable devices are also possible, such as electrostatic feedback, pressure-based feedback, etc.
The technology includes one or more perceptual systems for sensing the environment and detecting the objects within it. In some implementations, a perceptual system may include a vision system (e.g., depth-based skeletal tracking systems, range-based arm detection systems, and/or visual detection in RGB-D images) that can capture and process 2D and 3D depth images for objects and provide that information to the sensing system 109 (the object identifier 204), which can identify the objects by matching attributes of the objects in the images to corresponding pre-stored attributes in a data storage, such as the memory 237. In some implementations, the perceptual system may include an digital image capture device capable of capturing still and motion images, and may include a lens for gathering and focusing light, a photo sensor including pixel regions for capturing the focused light and a processor for generating image data and/or detecting objects based on signals provided by the pixel regions. The photo sensor may be any type of photo sensor including a charge-coupled device (CCD), a complementary metal-oxide-semiconductor (CMOS) sensor, a hybrid CCD/CMOS device, etc. In
The technology can indicate the direction of motion that will allow the user to most easily reach an object using vibrotactile feedback. In an example, sensors worn by the person (e.g., on one or more extremities) and/or mounted in the surrounding room, can be used to search for an object of interest when a spoken command is issued and captured by the sensing system 100. Once the object is found, and its relative position to the user's extremity (e.g., hand) is established, a vibration interface (e.g., bracelet/wristband) worn on the wrist of the individual vibrates using various patterns to indicate the direction the extremity needs to travel to reach the object.
A vibration interface may contain multiple vibration motors (also called vibrotactile motors or just motors) which it can control to convey various vibratory patterns corresponding with different movements to be undertaken by the bearer of the interface. For instance, the motors individually and/or cooperatively produce the vibratory patterns (also called signals) that convey the different movements. A vibratory pattern may include a magnitude and/or linear and/or rotational dimensions. By following the directions associated with the vibratory patterns, an individual can find and grasp/touch otherwise unseen objects in the environment. By way of example,
It should be understood that the above patterns are non-limiting and provided by way of illustration, and that other patterns are contemplated and encompassed by the scope of this disclosure, such as a lack of vibration, varying pulses, combinations of different motors to indicate complex motion (e.g. a combination of left and up), and varying the intensity of vibration to indicate distance to the detected object.
The types of trajectories that can be followed by the wrist worn vibration interface 100 based on the conveyed vibratory patterns are varied.
In some implementations, the sensing system 100 can adjust the vibration level (the amount of vibration the user feels) as a function of the position of the vibration interface relative to a target to indicate how much the user needs to move the extremity in a given direction. For instance, as the user's moves the extremity in response to the vibrational feedback, the vibration interface 800 can adapt the intensity of the vibration to indicate the amount of upward or forward motion is still necessary to reach to the target, although other suitable signals instructing the user about his/her progress are also possible (e.g., frequency of pulses, other signal types, etc.).
As a further example,
In addition to the sensing system 100 communicating different movements through vibratory motors mounted to one or more extremities, the motors can also be used to convey the internal state of the vibrotactile system 115 to the human bearer. For instance, if the sensing system 100 has lost the location of a human bearer or is otherwise unable to track objects and/or the bearer due to calibration, lost self-localization, etc., a unique vibratory pattern can be generated by the vibrotactile system 115 (e.g., the same motor set) so that the human knows that the sensing system 100 is in need of human assistance. In response, the human can act appropriately to aid in the track recovery. Further examples of internal states that can be conveyed to the user through alternative patterns of motion include processing issues or delays, configuration issues, communication delays, changes or updates to the current planned trajectory, certain communications from other nodes or objects, etc.
Returning to
The computing device 101 has data processing and communication capabilities and is coupled to the network 105 via signal line 104 for communication and interaction with the other entities of the system, such as the vibrotactile system 115 and/or the objects 117a . . . 117n. The computing device 101 may also be coupled to the vibrotactile system 115 via signal line 102 representing a direct connection, such as a wired connection and/or interface.
The computing device 101 is representative of various different possible computing devices and/or systems. Depending on the implementation, the computing device 101 may represent a client or server device. In addition, while a single computing device 101 is depicted, it should be understood that multiple computing devices 101 may be included in the system and coupled for communication with one another either directly or via the network 105. For instance, one computing device 101 may be a remote server accessible via the network from another local computing device 101, such as a consumer device. Examples of various different computing devices 101 include, but are not limited to, mobile phones, tablets, laptops, desktops, netbooks, kiosks, server appliances, servers, virtual machines, smart TVs, set-top boxes, media streaming devices, portable media players, navigation devices, personal digital assistants, custom electronic devices, embeddable/embedded computing systems, etc.
The vibrotactile system may be coupled to the network 105 for communication with the other entities of the system 100 via signal line 108. The vibrotactile system 115 includes a user-wearable electro-mechanical system configured to provide vibratory feedback to the user based on objects detected in the environment surrounding the user wearing the vibrotactile system 115. The vibrotactile system 115 includes a user-wearable portion about a body part. In some implementations, the user-wearable portion includes an encircling member that the user can don about the body part, such as a band, strap, belt, bracelet, etc. The body part may include an extremity, such as an hand, wrist, arm, ankle, knee, thigh, neck, head, prosthesis, or any other natural or artificial body part that can guided by the user using his/her motor skills. The user-wearable portion includes a set of vibration motors 112, also simply referred to as motors, controlled by a motor controller unit 110. The motors 112 singly and/or cooperatively produce signals (vibration patterns) to communicate various information to the bearer of the system 115, such as environmental and operational information. For example but not limitation, the motor(s) 112 may produce certain unique vibratory patterns, each of which signaling a particular direction in which the user should move the extremity bearing the user-wearable portion and the level in which the user should move in that direction, as discussed elsewhere herein. Other signals are also possible, as discussed elsewhere herein.
The vibrotactile system 115 also has a control portion including a sensing system 109, a calibrator 118, and the motor control unit 110. The sensing system 109 includes software and/or hardware logic executable to receive and process sensing data from the sensor(s) 114 and provide instructions to the motor control unit 110. In turn, the motor control unit 110 interprets the instructions and activates and deactivates the motor(s) 112 and/or the intensity of the motor vibration to produce the vibrational feedback corresponding to the instructions. The motor control unit 110 includes hardware and/or software logic to perform its functionality and is electrically coupled to the motor(s) to send and/or receive electrical signals to and/or from the motor(s). The sensing system 109 is coupled to the sensor(s) 114 via a wired and/or wireless connection to receive the sensing data.
The sensor(s) 114 are device(s) configured to capture, measure, receive, communicate, and/or respond to information. The sensor(s) 114 may be embedded in the user-wearable portion of the vibrotactile system 115 and/or included in another element of the system 100, such as the computing device 101 and/or another object in the environment. Example sensor(s) 114 include, but are not limited to, an accelerometer, a gyroscope, an IMU, a photo sensor capable of capturing graphical (still and/or moving image) data, a microphone, a data receiver (e.g., GPS, RFID, IrDA, WPAN, etc.), a data transponder (e.g., RFID, IrDA, WPAN, etc.), a touch sensor, a pressure sensor, a magnetic sensor, etc.
The calibrator 118 includes hardware and/or software logic executable to calibrate the motor(s) 112 to produce accurate vibratory feedback. The motor controller unit 110 is coupled to and interacts with the calibrator 118 to calibrate the motor(s) 112. The calibrator 118 may retrieve vibratory pattern parameters for a given vibratory pattern from the memory 127 and measure and compare corresponding aspects of the pattern as produced by the motor controller unit 110 in association with the motor(s) 112 with the parameters to determine compliance. For any aspects outside of the corresponding parameters, the calibrator 118 may adjust the current operational conditions of one or more of the motor(s) 112 so the vibratory pattern meets performance expectations.
The objects 117a . . . 117n include any tangible objects that may be included in an environment and that users can interact with and/or use. The objects may be everyday objects that a person would use or could include specialized objects that are intended to serve a particular purpose. For instance, as the technology discussed herein can be used as, but is not limited to, assistive technology for individuals that have various impairments, such as physical, visual, or hearing impairments, and one or more of the objects may represent assistive devices, such as a walking cane, hearing aid, prosthesis, etc. Other objects may represent everyday items the user may use, such as a coffee mug, cell phone, keys, table, chair, etc.
The objects 117a . . . 117n respectively include transponders 119a . . . 119n. The transponders 119 are configured to transmit information about the objects information to corresponding receivers included in the computing device 101 and/or the vibrotactile system 115. In an example implementation, a transponder 119 may be an active or passive RFID tag and the computing device 101 and/or the vibrotactile system 115 may include a corresponding RFID reader (sensor 114), which is configured to energize and/or read the information on the tag using an electromagnetic field. In further implementations, the transponder 119 may be configured to transmit information to corresponding sensors in the computing device 101 and/or the vibrotactile system 115 using other suitable protocols, such as Bluetooth™, IrDA, various other IEEE 802 protocols, such as IEEE 802.15.4, or other suitable means. As shown in
The processor 235 includes an arithmetic logic unit, a microprocessor, a general-purpose controller, or some other processor array to perform computations and provide electronic display signals to a display device. The processor 235 is coupled to the bus 220 for communication with the other components. Processor 235 processes data signals and may include various computing architectures, such as but not limited a complex instruction set computer (CISC) architecture, a reduced instruction set computer (RISC) architecture, other instruction sets, an architecture implementing a combination of various instruction sets, etc. The processor 235 may represent a single processor or multiple processors and may reflect a monolithic or distributed processing architecture. Other processors, operating systems, sensors, displays and physical configurations are possible and contemplated.
The memory 237 stores software instructions and/or data that may be executed and/or processed by the processor 235, such as code for performing the techniques described herein. Example software instructions may include but are not limited to instructions comprising at least a portion of the sensing system 109, the motor control unit 110, and/or the calibrator 118, etc. The memory 237 is coupled to the bus 220 for communication with the other components of the computing device 200.
In some instances, the memory 237 may store a camera engine including logic operable by the processor 237 to control/operate the perceptual system. For example, the camera engine is a software driver executable by the processor 237 for signaling the camera to capture and store a still or motion image, controlling the flash, aperture, focal length, etc., of the camera, provide image data, detect objects in the image data, etc.
The memory 237 may be volatile and/or non-volatile memory and may include may include any suitable memory device or system. Example devices include but are not limited to dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, hard disk drives, optical disc (e.g., CD, DVD, Blue-Ray™, etc.) devices, other mass storage devices for storing information on a more permanent basis, remote memory and/or storage systems, etc.
The communication unit 241 transmits and receives data to and from other nodes of the system 100, such as the computing device 101, the vibrotactile system 115, objects 117, etc., depending on which entity is represented. The communication unit 241 is coupled to the bus 220 for wireless and/or wired communication with the other components of the computing device 200. In some implementations, the communication unit 241 includes one or more wireless transceivers for exchanging data with the other entities of the system 100 using one or more wireless communication protocols, including IEEE 802.11, IEEE 802.16, BLUETOOTH®, or other suitable wireless communication protocols. In some implementations, the communication unit 241 includes port(s) for direct physical connection to the network 105 and/or other entities of the system 100 (e.g., objects 117, computing device 101, vibrotactile system 115), etc., depending on the configuration.
In some implementations, the communication unit 241 in a vibrotactile system 115 may be configured to communicate with the computing device 101 and/or objects 117 using various short, medium, and/or long-range communication protocols (RFID, NFC, Bluetooth®, Wi-Fi, Cellular, etc.). For instance, the communication unit 241 may include a sensor 114 for receiving data from the transponders 119 of the objects 117, as discussed elsewhere herein. As a further example, the sensor 114 may be an RFID reader configured to energize and receive tag ID data from the tag represented by a transponder 119 of an object 117, although other data exchange variations are also possible, as discussed elsewhere herein.
The sensing system 109 and/or other component so the system 100 can be implemented using hardware, software, and/or a combination thereof. For example, in some cases aspects of the system 109 may be implemented using hardware including a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC), may be implemented as software stored in the memory 237 and executable by the processor 235, a combination thereof, etc.
As shown in
In block 304, the method 300 may analyze the environment in which a user bearing a vibration interface is located for the object of interest using one or more sensors. In some implementations, the object identifier 204 may scan the environment using one or more sensors 114 of the vibrotactile system 115 or the computing device 101 for objects contained therein and determine whether any of those objects match the object of interest indicated by the user in block 302.
The objects in the environment may in some cases broadcast via radio frequency unique identifying information, which the sensor(s) 114 may receive and provide to the object identifier 204 for processing. The object identifier 204 may process the unique identifying information of each detected object by comparing it to unique identifying information of the object of interest, which may be retrievable from data storage (e.g., the memory 237, a database, remote data storage, etc.), to determine which object(s) in the environment match the indicated object of interest. The position determination module 208 may process the radio frequency signals broadcasted by the objects to determine their respective locations, for instance, using known micro-location techniques.
In some cases, the objects in the environment may be detected by one or more perceptual systems in cooperation with the sensing system 109. For example, as discussed elsewhere herein, the perceptual system (e.g., a sensor 114) may capture image data of the physical environment including objects included therein and the object identifier may receive the image data from the perceptual system and process it to detect objects included in the image data using standard image processing, object detection methods. Other variations and configurations are also contemplated and possible.
In block 306, the method 300 determines a relative position of the extremity of the user bearing the vibration interface to a position of the tangible object in the environment and generates in block 308 a trajectory for navigating the extremity of the user to the tangible object based on the relative position of the extremity to the position of tangible object.
In some implementations, the position determination module 208 determines the relative position of the extremity to the object and the trajectory generator 210 generates the trajectory based thereon. For instance, with a perceptual system, a form of object detection may be used to detect and determine the location of the object of interest. In addition, a method for detecting the position (location and/or orientation) of the extremity is also used if the sensor detecting the object is not embedded in or included on the vibration interface.
In some implementations, the position determination module 208 may execute operations for determining the relative positions between an extremity and objects including using a classifier engine (e.g., boosted Cascade), which may be included in the position determination module 208 or separate therefrom and executable by computing device 200, to detect objects in the image embodied by image data captured by the perceptual system, which may include physical objects in the environment and/or the user's extremity (e.g., hand). The position determine module 208 may use the image data and/or output from the classifier engine to identify the relative position of the extremity (whether detected or previously determined) to other detected object(s) in the image. For instance, the position determination module 208 could use depth data from an RGB—D camera or the like included in the perceptual system, could estimate the position(s) from the relative size(s) of the object(s) in the image, etc.
In some cases, the position determination module 208 can estimate the orientation of the extremity, for example, by determining a plurality of reference positions of the extremity and track the change in those positions over time to determine the current extremity orientation, although other variations are also possible and contemplated. The position determination module 208 may consider the motion of the extremity to determine the current and estimated future position of the extremity. The position determination module 208 may use the motion determination, the extremity orientation, position information of an object of interest, and/or other data to calculate the trajectory, as discussed elsewhere herein. For instance, the position denervation module may calculate a motion vector from the known positions of the object(s) of interest and the orientation and/or movement of the extremity.
In a further example, the sensing system 109 may search for a door handle in the environment using image data captured by a hand mounted camera. The sensing system 109 may analyze the images embodied by the image data from the camera using a classifier engine to find the door handle. The sensing system 109 would vibrate the vibration interface with the goal of situating the detected object in the center of the image. In this case, forward motion would move the extremity (e.g., hand) towards the door handle, and to that extent, the position determination module 208 may not be required to determine or estimate the actual distance to the object (e.g., at least not accurately), but can instead specify up, down, left, and right motions based on the distance of the center of the object in the image to the center of the image.
In some implementations, the position determination module 208 may determine the relative position of the extremity by determining an orientation of the extremity using sensing data captured by the one or more sensors. The sensing data reflects any movement of the extremity by the user. The position determination module 208 then calculates a ray originating from a predetermined point of the extremity and extending through a predetermined point of the tangible object and calculating the angular offset 8 between the ray and the orientation of the extremity. The trajectory generator 210 may then use the angular offset to generate the trajectory for navigating the extremity of the user to the tangible object. Further illustrative techniques for determine the relative position and generating the trajectory are discussed herein with reference to at least
Objects detected in the physical environment may have a fixed or variable location. When variable, the system 100 may be configured to track the change in location of those objects and dynamically guide the user to those objects using the techniques discussed herein. For instance, the position determination module 208 may detect movement in the environment by comparing a series of frames and detecting a change in the position of the object, and the position determination module 208 may process the image data to determine the current position of the objects within the environment, for example, using a Cartesian coordinate system and known reference points, such as its own position within the environment or other reference points included in the environment and reflected in the image data.
The method 300 may then guide the extremity of the user along the trajectory by vibrating the vibration interface. For example, the method 300 determines, in block 310, a vibratory pattern for vibrating one or more of the vibrotactile motors included in the vibration interface based on the trajectory generated in block 308 and vibrates the vibrotactile motor(s) 112 according to the vibratory pattern to convey the direction for movement by the user of the extremity to reach the tangible object. In some implementations, the vibrotactile feedback (VF) module 212 determines the vibratory pattern based on the trajectory received from the trajectory generator 210 and/or other signals, and interacts with the motor control unit 110 to vibrate the vibrotactile motor(s) 112, as discussed in further detail with reference to at least
In block 314, the method 300 determines whether the object of interest has been reached, and if not, may repeat one or more of the preceding blocks 306, 308, 310, and/or 312 is needed to guide the extremity of the user to the object of interest. For example, the position determination module 208 may determine that the extremity of the user has reached the position of the tangible object within the physical environment and may signal the VF module 212 to terminate vibrating the vibration interface to cease guiding the extremity of the user. In response, the VF module 212 may signal the motor control unit 110 to stop vibrating the motor(s).
In some cases, the position determination module 28 may continuously (re)determine the relative position of the object to the extremity of the user as the position of the object and/or the extremity of the user changes due to movement. For example, the position determination module 208 may sense movement of the extremity by the user using the one or more sensors. For instance, the position determination module 208 may receive signals from a gyroscope, IMU, accelerometer, or other movement sensors included in the vibration interface configured to detect vertical, horizontal, and/or rotational movement of the extremity of the user, and may process those signals to determine a current position (e.g., orientation and location) of the extremity and whether the position is consistent with the trajectory.
Responsive to sensing the movement, the position determination module 208 may re-determine the relative position of the extremity of the user to the tangible object and update the trajectory for navigating the extremity of the user to the tangible object based on a change to the relative position of the extremity of the user to the tangible object. For instance, if the position of the extremity is not consistent with the trajectory, the position determination module 208 may signal the trajectory generator 210 to regenerate the trajectory using the updated position. The VF module 212 may then guiding the extremity of the user along the trajectory, as updated, by vibrating the vibration interface according to the updated trajectory.
Thus, in response to detected movement, the trajectory generator 210 may regenerate the trajectory if a different trajectory is needed based on a change in the relative position. Consequently, the vibrotactile feedback generated by the VF module 212 and provided to the user via the motor control unit 110 and the motor(s) 112 may be continuously adapted based on the user's movements to accurately guide the user's extremity to the object of interest.
In block 402, the method 400 stores predefined identifiers for objects in the physical environment. For example, a user or administrator using a computing device 101 may register objects within the environment with the sensing system 109. The sensing system 109 may generate and display a corresponding interface for inputting information about the objects, and the sensing system 109 may store that information in a data store, such as a remote storage system coupled to the network, the memory 237, or another storage device, for access and/or retrieval by the sensing system 109, such as the object identifier 204. In further embodiments, the sensing system 109 may automatically identify the objects within the environment (e.g., using information broadcasted by the objects, objects identified from image data, etc.) and store information about those objects in the data store. Other variations are also possible.
In blocks 404-410, the method 400 selects a means for locating the object of interest, such as a perceptual system or transponder. For instance, in block 404, the method 400 selects whether to use an external camera located in the physical environment; in block 406, the method 400 selects whether to use a necklace camera worn by the user; in block 408, the method 400 selects whether to use a camera included in a user's vibration interface; and in block 410, the method 400 selects whether to use a transponder associated with the tangible object. If, in blocks 404, 406, and 408, the selection is affirmative, the method 400 proceeds to use the selected camera to locate the object in block 412, as discussed elsewhere herein. Conversely, if in block 410, the selecting is affirmative, the method 400 proceeds to determine in block 418 the relative position of the user's extremity to the tangible object based on the transponder signal, as discussed elsewhere herein.
In block 414, the method 400 determines the relative position of the extremity of the user to the position of the tangible object within the physical environment. For instance, in doing so, the position determination module 208 determines a central position of the tangible object, determines a centroid of the vibration interface, and calculates the relative position based on a distance between the central position of object and the centroid of vibration interface.
In block 416, the method 400 identifies whether any obstacles exist within the physical environment between the tangible object and the extremity of the user using the one or more sensors. In some embodiments, the obstacle determination module 206 analyzes the image data captured by the camera to identify the obstacles, determines the position of the obstacles relative to the position of the object of interest and the position of the vibration interface, and provides that information to the trajectory generator 210 for use in generating the trajectory.
In block 420, the method 400 generates the trajectory for navigating the extremity of the user to the tangible object based on the relative position of the extremity to the position of tangible object and the position(s) of any obstacle(s) within the physical environment. By way of example, the trajectory may be based on a path that circumnavigates any detected obstacles.
In block 422, the method 400 determines a vibratory pattern including vibrational dimension(s) for vibrating one or more vibrotactile motors based on the trajectory generated in block 420. In some embodiments, the vibratory pattern includes one or more of linear motion and rotational dimensions that correspond to the movements that the user should perform to move his/her extremity toward the object. The VF module 212, via the motor control unit 110, then vibrates in block 424 the one or more motors based on the one or more of the directional and rotational dimensions of the vibratory pattern to convey the direction for movement of the extremity to reach the tangible object. As shown in block 426, the method 400 can iterate until the object has been successfully reached. For instance, if the object has not yet been reached, the method 400 may return to block 414 or 418 (depending on which operations are being used to locate the object). Otherwise, the method terminates or proceeds to another set of operations.
In block 504, the method 500 vibrates the one or motors of the vibration interface using the bit sequence and vibration intensity value(s). For example, the VF module 212 sends the bit sequence and the vibration intensity value(s) to the motor controller unit 110, which then uses the bit sequence and the vibration intensity value(s) to control/turn on/off the motors.
In block 606, the VF module 212 vibrates the vibration interface based on the unique vibratory pattern. Responsive to the vibration, the bearer of the interface provides input via an input device providing assistance to address the operational problem, which the sensing system 109 receives in block 608 and uses to resolve the operation problem (e.g., resets the vibratory interface, clears the memory 237, receives a location, receives identification of an object, etc.). If the problem is not resolved, the method 600 may return to block 604 and repeat. Otherwise, the method 600 may end or proceed to perform other operations, such as those discussed elsewhere herein.
For reference,
In the above description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it should be understood that the technology described herein can be practiced without these specific details. Further, various systems, devices, and structures are shown in block diagram form in order to avoid obscuring the description. For instance, various embodiments are described as having particular hardware, software, and user interfaces. However, the present disclosure applies to any type of computing device that can receive data and commands, and to any peripheral devices providing services.
In some instances, various embodiments may be presented herein in terms of algorithms and symbolic representations of operations on data bits within a computer memory. An algorithm is here, and generally, conceived to be a self-consistent set of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout this disclosure, discussions utilizing terms including “processing,” “computing,” “calculating,” “determining,” “displaying,” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
Various embodiments described herein may relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, including, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, flash memories including USB keys with non-volatile memory or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.
The technology may be implemented in hardware and/or software, which includes but is not limited to firmware, resident software, microcode, etc. Furthermore, the technology can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer readable medium can be any non-transitory storage apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
A data processing system suitable for storing and/or executing program code may include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories that provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution. Input/output or I/O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I/O controllers.
Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems, storage devices, remote printers, etc., through intervening private and/or public networks. Wireless (e.g., Wi-Fi™) transceivers, Ethernet adapters, and modems, are just a few examples of network adapters. The private and public networks may have any number of configurations and/or topologies. Data may be transmitted between these devices via the networks using a variety of different communication protocols including, for example, various Internet layer, transport layer, or application layer protocols. For example, data may be transmitted via the networks using transmission control protocol/Internet protocol (TCP/IP), user datagram protocol (UDP), transmission control protocol (TCP), hypertext transfer protocol (HTTP), secure hypertext transfer protocol (HTTPS), dynamic adaptive streaming over HTTP (DASH), real-time streaming protocol (RTSP), real-time transport protocol (RTP) and the real-time transport control protocol (RTCP), voice over Internet protocol (VOIP), file transfer protocol (FTP), WebSocket (WS), wireless access protocol (WAP), various messaging protocols (SMS, MMS, XMS, IMAP, SMTP, POP, WebDAV, etc.), or other known protocols.
Finally, the structure, algorithms, and/or interfaces presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method blocks. The required structure for a variety of these systems will appear from the description above. In addition, the specification is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the specification as described herein.
The foregoing description has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the specification to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the disclosure be limited not by this detailed description, but rather by the claims of this application. As will be understood by those familiar with the art, the specification may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Likewise, the particular naming and division of the modules, routines, features, attributes, methodologies and other aspects are not mandatory or significant, and the mechanisms that implement the specification or its features may have different names, divisions and/or formats.
Furthermore, the modules, routines, features, attributes, methodologies and other aspects of the disclosure can be implemented as software, hardware, firmware, or any combination of the foregoing. Also, wherever a component, an example of which is a module, of the specification is implemented as software, the component can be implemented as a standalone program, as part of a larger program, as a plurality of separate programs, as a statically or dynamically linked library, as a kernel loadable module, as a device driver, and/or in every and any other way known now or in the future. Additionally, the disclosure is in no way limited to implementation in any specific programming language, or for any specific operating system or environment. Accordingly, the disclosure is intended to be illustrative, but not limiting, of the scope of the subject matter set forth in the following claims.
