Patent Grant
12044541
The invention is related to the field of assistance of the movement of visually impaired persons. In particular the invention is related to a method and a wearable device for assisting the movement of a visually impaired person, which throughout the invention shall be called visually impaired user. The term “visually impaired” shall encompass throughout the invention the moderate impairment as well as severe impairment (blindness).
Studies have shown that 36 million people were affected by severe impairment in 2015 while 216.6 million had moderate to severe visual impairment. While these numbers are increasing, people become more aware of their needs and solutions targeted at aiding visual disability emerge. For example, cities are becoming more accessible to blind individuals using classic navigation methods such as the walking cane or the guide dog. Technological solutions also emerge and begin being accepted by the blind and visually impaired community. Solutions such as the ones proposed by OrCam Inc. or Microsoft Inc. have seen various levels of adoption. However, despite the advancements in technology, the most used solution for the visually impaired persons is still the walking cane.
Technological solutions for assisting the visually impaired persons are not new. Early efforts can be attributed to Paul Bach-Y-Ritta et al. in U.S. Pat. No. 3,594,823A. There, visual signal has been translated to haptic feedback on the back of blind individuals.
Generally speaking, the technological solutions for assisting the visually impaired persons emerged in one or more of the following categories:
Various solutions address one or more of the above-captioned categories.
U.S. Pat. No. 9,915,545 published 26 Jul. 2015 discloses a method for providing directions to a blind person using a smart device. The method includes detecting, by at least two sensors and in response to a selection of a find mode of the smart device, image data corresponding to a surrounding environment of the smart device and positioning data corresponding to a positioning of the smart device; receiving by an input device the desired object or desired location; determining by a processor the initial location of the smart device based on the image data, the positioning data and map data stored in a memory of the smart device; providing by the output device of directions to the desired object based on the initial location of the smart device and the map data.
U.S. Pat. No. 9,629,774 published 16 Jul. 2015 discloses a smart necklace that includes a body defining at least one cavity and having a neck portion and first and a second side portions. The necklace includes a pair of stereo cameras that is configured to detect image data including depth information corresponding to a surrounding environment of the smart necklace. The necklace further includes a positioning sensor configured to detect positioning data corresponding to a positioning of the smart necklace. The necklace includes a non-transitory memory positioned in the at least one cavity and configured to store map data and object data. The smart necklace also includes a processor positioned in the at least one cavity, coupled to the pair of stereo cameras, the positioning sensor and the non-transitory memory. The processor is configured to determine output data based on the image data, the positioning data, the map data and the object data. In an academic article published 15 Nov. 2016, “Wearable Indoor Navigation System with Context Based Decision Making for Visually Impaired” , the authors Xiaochen Zhang et al. present a wearable indoor navigation system for the visually impaired. The system uses Simultaneous Localisation and Mapping SLAM and semantic path planning for the localisation and navigation, integrating m sensors and feedback devices such as an RGB-D camera, an Inertial Measurement Unit IMU and a web camera. The system applies the RGB-D based visual odometry algorithm to estimate the user's location and orientation and the Inertial Measurement Unit IMU to refine the orientation error. Major landmarks such as room numbers and corridor corners are detected by the web camera and the RGB-D camera and matched to the digitalized floor map so as to localize the user. The path and motion guidance are generated to guide the user to a desired destination. The article suggests a way to improve the fitting between the rigid commands and optimal machine decisions for human beings in a context-based decision-making mechanism on path planning to resolve user's confusions caused by incorrect observations.
In respect to the sensorial substitution, the known solutions generally propose an improper replacement of the bandwidth of eyesight which is known to be far greater than the one embedded in the auditory and/or the haptic senses, providing either too scarce information or too much non-essential information, confusing or annoying the user.
In respect to the ways of generating the map as well as the ways of generating paths for the visually impaired person and communicating said paths to him/her, the Navigation/GPS-based methods of prior art offer a general path but do nothing to avoid obstacles or living beings.
In respect to the ways of localizing the position of the visually impaired person in its environment and/or localizing various objects that can be considered obstacles and/or target destinations, methods such as Microsoft Seeing AI or OrCam recognize only the objects that are present in their limited field of view but do not offer sufficient information about how to reach the target destinations. In general, the known solutions have no or reduced possibility to store the position of the objects detected by the sensors and, consequently no or reduced possibilities to recognize objects detected in the past. The known solutions have no possibilities to provide full information about the objects, information that may be of use for the visually impaired person such as: if a chair is occupied by other person, the sizes and other physical and chemical characteristics of the objects which leads to the need that the visually impaired person touches said objects in order to perceive said physical and chemical characteristics, which is not very hygienic, can be dangerous and may take too much time.
In respect to the ways of communicating the information regarding the environment, paths or obstacles to the visually impaired person and receiving feedback from him/her, most of current solutions communicate the information too slowly to the visually impaired person and/or receive from him/her the feedback too slow, which leads to difficult and delayed modification of the initial path.
For example, in an embodiment of U.S. Pat. No. 9,629,774, although the smart necklace may recognize stairs, exits restrooms or empty seats, said necklace is not configured to provide more in-depth information such as characteristics of the objects: orientation, where is the latch or doorknob or if the empty seat is dirty.
U.S. Pat. No. 9,629,774 teaches a limited number of types of commands that the smart necklace can transmit to the visually impaired user: different degrees of turns, such as 45-degree turn, a 90-degree turn, left turn, right turn. Using this way of communicating the information, U.S. Pat. No. 9,629,774 is rigid when compared with the natural path generation of the non-visually impaired person.
U.S. Pat. No. 9,629,774 teaches about a map that contains only location of various objects without any other characteristics.
U.S. Pat. No. 9,915,545 teaches a method for providing directions to a blind user of an electronic device, the directions being from a current location of the electronic device to a location of a desired object. In this method there is no differentiation between the types of areas on which the paths are generated, there are no relationships built between various objects, some important of the characteristics of the objects and living beings are left out, such as the detection of the emotional status of the living beings or the degree of cleanliness of a surface, and the content of the map is reduced to the information from the field of view of the sensors.
The problem to be solved by the invention is to provide for a method for assisting the movement of the visually impaired user that allows the user to navigate indoor and outdoor in a manner closer to the navigation of a non-visually impaired user. In particular, the problem to be solved by the invention is:
In order to solve the problem, the inventors conceived in a first aspect of the invention a method for assisting the movement of a visually impaired user by means of a wearable device, comprising the following steps:
S1
Acquiring data from the environment of the visually impaired user by means of a sensory unit 2 of the wearable device 1, sensing from a field of view 20, sending said acquired data to a Sensory Fusion sub-unit 30 of a Processing and control unit 3 of the wearable device 1,
S2
Fusing the acquired data by the Sensory fusion sub-unit 30, sending the fused data to a Live Map sub-unit 31 of the Processing and control unit 3,
Creating, repeatedly updating and storing by the Live Map sub-unit 31 of a Live Map 310, the Live Map 310 comprising the following data:
1. Live Map determinations based on fused data received from the Sensory Fusion sub-unit 30, including:
2. Live Map determinations that are generated based on a plurality of relationships Rn between the plurality of objects On and/or the plurality of living beings Ln received from a Relationship Manager sub-unit 32 of the Processing and control unit 3,
3. Live Map determinations that are generated based on a free area A defined as an ensemble of areas on a ground not occupied by the plurality of objects On and the plurality of living beings Ln, the free area A including:
Automatically or in response to a request from the visually impaired user determining, by a Navigation Manager sub-unit 33 of the Processing and control unit 3, repeatedly updating and storing, of at least one navigation path Pn and associated navigation guiding instructions for the visually impaired user to navigate from the current position of the sensory unit 2 to a point of interest PI selected among the plurality of objects (On) and/or the plurality of living beings Ln, Automatically or in response to a request from the visually impaired user repeatedly selecting a preferred navigation path SP from the at least one navigation path Pn,
that satisfies at least two navigation path requirements:
i) Passes through the walkable area WA and/or on the conditional walkable area CWA, and
ii) Meets a set of safety requirements including a non-collision requirement, and a non-aggressivity requirement.
Wherein any request from the visually impaired user is made by using haptic means 51 or audio means 52 of a User commands interface 5, said requests being received by the Navigation Manager sub-unit 33 via a User commands interface Manager sub-unit 34 of the Processing and control unit 3,
and transmitting by the Navigation Manager sub-unit (33) to a Feedback Manager sub-unit 35 of the Processing and control unit (3)
i) of the preferred navigation path SP and
ii) of the associated navigation guiding instructions,
wherein, when the preferred navigation path SP passes through the conditional walkable area CWA, the Navigation Manager sub-unit) sends to the Feedback Manager sub-unit the associated navigation guiding instruction corresponding to said at least one predictable conditional walkable area requirement
S4
Providing, by the Feedback Manager sub-unit 35 guidance to the visually impaired user, along the preferred navigation path SP, by using guiding modes for transmitting each associated navigation guiding instruction, each navigation instruction comprising haptic and/or auditory cues sent by the Feedback Manager sub-unit 35 to a Feedback unit 4 of the Processing and control unit (3), said Feedback unit 4 comprising:
wherein the guiding modes for each associated navigation guiding instruction are selected by the visually impaired user by the User commands interface 5 and through user commands that are received by the Feedback Manager sub-unit 35 via the User commands interface Manager sub-unit 34.
In a second aspect of the invention, it is provided a wearable device 1 for assisting the movement of a visually impaired user, comprising:
electronic communications means between the Sensory unit 2, the Processing and control unit 3, the Feedback unit 4, the User commands interface 5 , the Power storage unit 6 and the memory M, by communication protocols,
In a third aspect of the invention, it is provided a computer program comprising instructions which, when the program is executed by the wearable device causes the wearable device 1 to carry out the steps of the computer-implemented method for assisting the movement of a visually impaired user, in ny of the preferred embodiments, including combinations thereof. In a fourth aspect of the invention, it is provided a computer readable medium having stored thereon instructions which, when executed by the wearable device 1, causes the wearable device 1 to carry out the steps of the computer-implemented method, in any of the preferred embodiments, including combinations thereof.
In a fifth aspect of the invention, it is provided a non-transitory computer-readable storage device storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations of the computer-implemented method, in any of the preferred embodiments, including combinations thereof. In a sixth aspect of the invention, it is provided a system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations of the computer-implemented method, in any of the preferred embodiments, including combinations thereof. According to one example implementation, a computer-implemented method comprising includes acquiring data from an environment of a visually impaired user, comprising a sensory unit of a wearable device sensing from a field of view, sending the acquired data to a sensory fusion sub-unit of a processing and control unit of the wearable device, fusing the acquired data by the sensory fusion sub-unit, sending the fused data to a live map sub-unit of the processing and control unit, and creating, repeatedly updating, and storing, by the live map sub-unit, a live map. The live map includes one or more live map determinations that are generated based on the fused data received at the processing and control unit from the sensory fusion sub-unit, including a position and an orientation of the sensory unit, a plurality of objects, and a plurality of living beings, one or more live map determinations that are generated based on a plurality of relationships between the plurality of objects or the plurality of living beings or between the plurality of objects and the plurality of living beings that are received from a relationship manager sub-unit of the processing and control unit, one or more live map determinations that are generated based on a free area that is defined as an ensemble of areas on a ground not occupied by the plurality of objects and the plurality of living beings, the free area including a walkable area that satisfies a set of permanent predetermined walkable area requirements, and a conditional walkable area that satisfies the set of permanent predetermined walkable area requirements, and at least one predictable conditional walkable area requirement. The method includes automatically or in response to a first request from the visually impaired user, determining, by a navigation manager sub-unit of the processing and control unit, repeatedly updating and storing, at least one navigation path and associated navigation guiding instructions for the visually impaired user to navigate from a current position of the sensory unit to a point of interest selected among the plurality of objects or the plurality of living beings or the plurality of objects and the plurality of living beings, automatically or in response to a second request from the visually impaired user, repeatedly selecting a preferred navigation path from the at least one navigation path that (i) passes through the walkable area or on the conditional walkable area or on the walkable area and on the conditional walkable area, and (ii) meets a set of safety requirements including a non-collision requirement, and a non-aggressivity requirement, where any request from the visually impaired user is made by using haptic means or audio means of a user commands interface the requests being received by the navigation manager sub-unit via a user commands interface manager sub-unit of the processing and control unit, transmitting, by the navigation manager sub-unit to a feedback manager sub-unit of the processing and control unit, the preferred navigation path and the associated navigation guiding instructions, where, when the preferred navigation path passes through the conditional walkable area, the navigation manager sub-unit sends to the feedback manager sub-unit the associated navigation guiding instruction corresponding to the at least one predictable conditional walkable area requirement, providing, by the feedback manager sub-unit, guidance to the visually impaired user, along the preferred navigation path , using guiding modes for transmitting each associated navigation guiding instruction, each navigation instruction comprising haptic or auditory cues sent by the feedback manager sub-unit to a feedback unit of the processing and control unit, the feedback unit including haptic feedback actuators configured for placement on the head of the visually impaired user, or auditory feedback actuators configured for placement to one or both ears of the visually impaired user, or haptic feedback actuators configured for placement on the head of the visually impaired user and auditory feedback actuators configured for placement to one or both ears of the visually impaired user, where the guiding modes for each associated navigation guiding instruction are selected by the visually impaired user by the user commands interface and through user commands that are received by the feedback manager sub-unit via the user commands interface manager sub-unit.
Some implementations may include one or more of the following features. For example, the method may include creating and updating the live map, including repeatedly determining the position and orientation of the sensory unit, a position, orientation and characteristics of the plurality of objects and of the plurality of living beings, based on the fused data received from the sensory fusion sub-unit, and repeatedly sending the created and updated live map to a localization module of the sensory fusion sub-unit, repeatedly generating and updating, by the relationship manager sub-unit, the plurality of relationships between the plurality of objects or the plurality of living beings or the plurality of objects and the plurality of living beings based on the data acquired from the live map including applying a set of the predetermined relations requirements, and repeatedly sending the updated plurality of relationships to the live map, repeatedly localizing, by a localization module the position and orientation of the sensory unit with respect to the plurality of the objects, and, to the plurality of living beings of the live map using localization algorithms applied to the data received from the sensory unit and data from the of the live map and repeatedly sending the localization data of the position and orientation of the sensory unit to a walkable area detection module of the sensory fusion sub-unit, repeatedly determining, by the walkable area detection module, the free area based on the data received from the sensory unit, the data received from the localization module, the set of permanent predetermined walkable area requirements, and the at least one predictable conditional walkable area requirement calculated and stored in the memory, and repeatedly sending the updated free area to the live map, and repeatedly storing the updated live map in the memory. The live map may be updated by the sensory fusion sub-unit using simultaneous localization and mapping (SLAM) algorithms. The method may include sending an information request by the visually impaired user to a sound representation sub-unit of the processing and control unit regarding at least one object selected from the plurality of objects or at least one living being selected from the plurality of living beings, extracting by a sound representation sub-unit of the processing and control unit from the live map the information regarding the selected at least one particular object or at least one particular living being; representing the extracted information as corresponding spatialized sounds, transmitting the spatialized sounds to the visually impaired user by the feedback unit, selecting, by the visually impaired user of the point of interest from the plurality of objects or from the plurality of living beings, and transmitting the corresponding selection request to the navigation manager sub-unit. The method may include determining by the navigation manager wandering path together with the associated navigation guiding instructions for the visually impaired user, and sending the wandering path and the associated navigation guiding instructions to the feedback manager sub-unit. The haptic cues may vary in duration, periodicity, intensity or frequency of the vibration according to predetermined preferred navigation path complexity criteria, and the audio cues may vary in frequencies, duration, repetition intensity, or 3d spatial virtualization according to the predetermined preferred navigation path complexity criteria. A three-dimensional walkable tunnel may be defined as a virtual tunnel of predetermined cross-section, having as horizontal longitudinal axis the preferred navigation path, and wherein the guiding mode further comprises specific haptic cues sent to the visually impaired user when the visually impaired user is approaching the virtual walls of the walkable tunnel. The preferred navigation path may be divided into predetermined segments delimited by a plurality of milestones, and the guiding mode may include haptic cues or auditory cues signaling the position of a next at least one milestone providing associated navigation guiding instructions to the visually impaired user from a current milestone to a subsequent milestone, and the length of the predetermined segments may vary depending on the complexity and length of the preferred navigation path. The guiding mode may include haptic cues or auditory cues or haptic and auditory cues signaling a direction on the preferred navigation path. The direction on the preferred navigation path may be determined by the line defined by an origin of the sensory unit and an intersection of the preferred navigation path with a circle having an origin at the position of the sensory unit and a radius with a predetermined length, and the auditory cues signaling the direction on the preferred navigation path may originate from a spatialized sound source placed at a predetermined first distance of the spatialized sound source s with respect to the sensory unit. The auditory cues may be spatialized sounds originating from a spatialized sound source that virtually travels along a predetermined second distance on the preferred navigation path from the position of the sensory unit until the spatialized sound source reaches the end of the predetermined second distance and back to the position of the sensory unit.
In another general implementation, a wearable device for assisting the movement of a visually impaired user includes a sensory unit configured to be placed on the head of the visually impaired user, including a camera, a depth sensor, an inertial measurement unit, and a sound localization sensor. The device includes a processing and control unit including a sensory fusion sub-unit including a localization module, a walkable area detection module, an orientation computation module, a sound direction localization module, a sound classification module, an object 2d characteristics extraction module, an object 3d characteristics fusion module, and an object sound characteristics fusion module. The device includes a live map sub-unit, a relationship manager sub-unit, a navigation manager sub-unit, a user commands interface manager sub-unit, a feedback manager sub-unit, and a sound representation sub-unit. The device includes a feedback unit configured to be placed on the head of the visually impaired user, including a plurality of haptic feedback actuators including left haptic feedback actuators, right haptic feedback actuators, center haptic feedback actuators, a plurality of auditory feedback actuators including left auditory feedback actuators, and right auditory feedback actuators The device includes a user commands interface configured to be placed on the head of the visually impaired user including a plurality of user commands haptic means, and a plurality of user commands audio means. The device includes a power storage unit, a memory, and electronic communications component between the sensory unit, the processing and control unit, the feedback unit, the user commands interface, the power storage unit and the memory. Example implementations may include one or more of the following features. The sensory unit may include at least one additional sensor, from among a global positioning sensor configured to determine the absolute position of the sensory unit, or a temperature sensor configured to determine the temperature of the objects and of the living beings. The sensory fusion sub-unit may include a relative-to-absolute conversion module that is configured to fuse the data from the object sound characteristics fusion module with the data regarding the absolute position of the sensory unit, and an object temperature characteristics fusion module that is configured to fuse the data from the object sound characteristics fusion module with the data regarding the temperature of the objects and of the living beings, and to send the fused data to the live map sub-unit.
Other example implementations may include a system including one or more processors; and one or more non-transitory machine-readable storage devices storing instructions that are executable by the one or more processors to perform operations corresponding to the disclosed methods, or a non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that when executed by one or more processors cause the one or more processors to perform operations corresponding to the disclosed methods. Any feature in one aspect of the invention may be applied to other aspects of the invention, in any appropriate combination. In particular, method features may be applied to device features, and vice versa.
Wherever applicable, means—plus—function features may be expressed alternatively in terms of their corresponding structure, such as a suitably programmed processor and associated memory.
Particular combinations of the various features of the invention can be implemented and/or supplied and/or used independently.
The main advantages of this invention are the following:
1 and
This list includes references to components, parameters or criteria presents in the description and/or drawings. It is created to ease the reading of the invention.
Live Map determinations based on the fused data received from the Sensory Fusion sub-unit 30
Live Map determinations based on the data received from the Relationship Manager sub-unit 32:
Guiding modes for transmitting the corresponding navigation guiding instructions
t0 starting point, and tfinal end point in time of spatialized sound sources virtually moving on the contour of the window 85 detailed description and examples of realization
With reference to
The wearable device 1 comprises two hardware units not represented graphically: a Power storage unit 6, and a memory M.
Throughout the invention, it shall be understood that the visually impaired person is wearing the wearable device 1 and that the wearable device 1 is switched on. Therefore, any reference in the description, claims and drawings to the wearable device 1 or to the Sensory unit 2 shall be understood as including a reference to the position of the visually impaired person. For simplicity, throughout the invention, the visually impaired user shall be referred to as “he”, encompassing all gender situations.
Details about the configuration and location of the hardware units will be given in the section of the description that relates to the configurations of the wearable device 1.
For a better understanding of the method, the basic components of the hardware units are briefly described keeping the pace together with the disclosure of the method.
The Sensory unit 2 is placed on the head of the visually impaired user and comprises basic sensors:
The Processing and control unit 3 comprises:
The Sensory fusion sub-unit 30 comprises:
The method according to the invention includes 4 steps. The four steps will be firstly described briefly in their succession. Then, steps 2, 3 and 4 will be detailed.
S1 The sensory unit 2 of the wearable device 1, placed on the head of visually impaired user, acquires data from the environment of the visually impaired user.
For this purpose, the sensory unit 2 senses from a field of view 20 having as origin the position of the sensory unit 2.
S2 The data sensed by the sensory unit 2 is sent to the Sensory fusion sub-unit 30.
The Sensory fusion sub-unit 30 fuses the data acquired from the sensory unit 2 by data processing algorithms that include filtering, smoothing, and artificial intelligence-based algorithms, and then sends the fused data to the Live map sub-unit 31 of the Processing and control unit 3.
Further on, the Live Map sub-unit 31 creates, repeatedly updates and stores a Live Map 310. The Live Map 310 comprises three categories of data:
1. Live Map determinations that are generated based on fused data received from the Sensory Fusion sub-unit 30,
2. Live Map determinations that are generated based on a plurality of relationships Rn between the plurality of objects On and/or the plurality of living beings Ln,
3. Live Map determinations based on a free area A.
The Live Map 310 is a database stored in the memory M. Throughout the invention, the update and store of the data in the Live Map 310 shall include the update and store of the Live Map 310 in the memory M. The way the Live Map 310 is stored is outside the scope of the invention.
In an embodiment of the present invention, the Live map 310 already exists in the memory M. In this case, the territorial range of the Live Map 310 is determined by the content stored in the past in the Live Map 310. As it can be seen from
1. The Live Map determinations based on the fused data received from the Sensory Fusion sub-unit 30 include the following determinations:
2. The Live Map determinations based on the plurality of relationships Rn between the plurality of objects On and/or the plurality of living beings Ln are received from a Relationship Manager sub-unit 32 of the Processing and control unit 3.
The Relationship Manager sub-unit 32 imports the most recent updates from the Live Map 310 by querying the Live Map sub-unit 31 for updates in the Live Map 310.
Computations are based on predetermined relations requirements, comprising at least:
For simplicity, throughout the invention:
After carrying out the computations, the Relationship Manager sub-unit 32 sends the updated relations as result of the computations to the Live map sub-unit 31 to store same in the Live Map 310.
Details regarding the generation of the plurality of relationships Rn are given in the section related to S.2.2 below.
3. The Live Map determinations based on the free area A
The free area A is defined as an ensemble of areas on a ground not occupied by the plurality of objects On and the plurality of living beings Ln.
Said free area A is divided into three categories:
In S3, automatically or in response to a request from the visually impaired user, a Navigation Manager sub-unit 33 of the Processing and control unit 3 determines, repeatedly updates and stores in the memory M, one or more navigation paths Pn for the visually impaired user to navigate from the current position of the sensory unit 2 to a point of interest PI selected among the plurality of objects On and/or the plurality of living beings Ln.
The term “navigation” shall be understood in this invention as encompassing:
Non-limiting examples of navigation paths Pn include:
The Navigation Manager sub-unit 33 repeatedly selects, automatically or in response to the request from the visually impaired user, one preferred navigation path SP. If only one navigation path Pn was determined, then the preferred navigation path SP is the navigation path Pn. If two or more navigation paths Pn were determined, the Navigation Manager sub-unit 33 repeatedly selects one of them as the preferred navigation path SP.
The preferred navigation path SP is repeatedly sent by the Navigation Manager sub-unit 33 together with associated navigation guiding instructions, to a Feedback Manager sub-unit 35 of the Processing and control unit 3.
In order to determine one or more navigation paths Pn, the Navigation Manager sub-unit 33 queries the Live Map 310 in order to check if at least two navigation path requirements are met. The first navigation path requirement is that all navigation paths Pn—thus including the preferred navigation path SP, must pass through the walkable area WA and/or on the conditional walkable area CWA.
The second navigation path requirement is to meet a set of safety requirements in respect to the plurality of objects On and/or the plurality of living beings Ln positioned or predicted to be positioned in the proximity of the at least one navigation path Pn in the predetermined unit of time. The proximity is predetermined, for example at 0.3 m from the position of the wearable device 1.
The set of safety requirements includes at least one non-collision requirement and at least one non-aggressivity requirement. Other safety requirements may be defined for various specific needs arising either from the needs of the visually impaired person e.g., elderly person, or from the characteristics of the environment where the visually impaired person usually lives, e.g., a densely populated urban area or from both.
The non-collision requirement means that the individual paths of plurality of objects On and/or the plurality of living beings Ln must not collide with the at least one navigation path Pn.
The non-aggressivity requirement means that the mood of the plurality of living beings Ln must not anticipate an aggressive action directed against the visually impaired user.
Other navigation path requirements may be defined by the user such as but not limited to the requirement to avoid crowded areas or to avoid passing through zones with slopes higher than a predetermined value.
The navigation path requirements are predetermined and stored in the memory M. They are applied by the Navigation Manager sub-unit 33. The visually impaired user can set other predetermined navigation path requirements by means of the User commands interface Manager sub-unit 34.
When the selected navigation path SP passes through the conditional walkable area CWA, the Navigation Manager sub-unit 33 sends to the Feedback Manager sub-unit 35 an associated navigation guiding instruction associated to said at least one predictable conditional walkable area requirement.
The determination of the at least one navigation path Pn is initiated either automatically by the Navigation Manager sub-unit 33 or by receiving from the visually impaired user of an initiation request,
In case the Navigation Manager sub-unit 33 determines two or more navigation paths Pn, the selection of the preferred navigation path SP is carried out either automatically by the Navigation Manager sub-unit 33 or by receiving by said Navigation Manager sub-unit 33 of a selection request from the visually impaired user.
The Navigation Manager sub-unit 33 can be configured such that, by default, the selection of the preferred navigation path SP be carried out either automatically by the Navigation Manager sub-unit 33, or according to the selection request from the visually impaired user.
When carried out automatically by the Navigation Manager sub-unit 33, the selection of the preferred navigation path SP is based on applying a set of pre-determined path selection criteria including cost criteria, cost-time to destination criteria, comfort criteria. The application of the path selection criteria is carried out according to prior art.
The requests made by the visually impaired user are made by using haptic means 51 or audio means 52 of a User commands interface 5. These requests are received by the Navigation Manager sub-unit 33 via a User commands interface Manager sub-unit 34 of the Processing and control unit 3.
In S4 the Feedback Manager sub-unit 35 guides the visually impaired user along the preferred navigation path SP, by using guiding modes for transmitting each associated navigation guiding instruction as received from the Navigation Manager sub-unit 33.
The guiding modes are sent by the Feedback Manager sub-unit 35 to a Feedback unit 4 of the Processing and control unit 3. Each navigation guiding instruction comprises haptic and/or auditory cues.
The guiding modes are:
The selection of the guiding modes for each associated navigation guiding instruction is carried out by the visually impaired user by the User commands interface 5 and through user commands that are received by the Feedback Manager sub-unit 35 via the User commands interface Manager sub-unit 34.
Details Regarding S2—With Reference to
The Live Map 310 can be compared with a multi-layer cake, as several layers of information are added from the first sub-step until the last sub-step as described below. With each layer, the Live Map 310 acquires a higher level of detail and accuracy. The creation is continuous, having as result the continuous update and continuous storage of the Live Map 310.
The advantage of creating multiple information layers in the Live Map 310 is related to the ease of use of understanding and accessing the data. As each individual layer contains specific information which is relevant to certain other components of the system, this facilitates faster access to the information.
S 2.1. The Live Map sub-unit 31 creates and updates the Live Map 310 by repeatedly determining the position and orientation of the sensory unit 2, the position and orientation, and characteristics of the plurality of objects On, of the plurality of living beings Ln, based on the fused data received from the Sensory Fusion sub-unit 30, and repeatedly sends the created and updated Live Map 310 to a Localisation module 301 of the Sensory Fusion sub-unit 30,
S 2.2. The Relationship Manager sub-unit 32 repeatedly generates and updates a plurality of relationships Rn between the plurality of objects On and/or the plurality of living beings Ln based on the data acquired from the Live Map 310 comprising applying a set of the predetermined relations requirements. The plurality of relationships Rn, repeatedly updated, are repeatedly sent to the Live Map 310, thus updating the Live Map 310 content as outputted from S 2.1. with the layer referring to the content of the plurality of relationships Rn,
S 2.3. The Localisation module 301 repeatedly localizes the position and orientation of the sensory unit 2 with respect to the plurality of the objects On, and, respectively to the plurality of living beings Ln of the Live Map 310 using localisation algorithms applied to the data received from the sensory unit 2 and data from the of the Live Map 310. The localisation of the position and orientation of the sensory unit 2 is repeatedly sent to a Walkable Area Detection module 302 of the Sensory fusion sub-unit 30, thus updating the Live Map 310 content as outputted from S 2.2 with the layer referring to the localisation data of the position and orientation of the sensory unit 2 in respect to the plurality of the objects On, and, respectively to the plurality of living beings Ln.
S 2.4. The Walkable Area Detection module 302 repeatedly determines the free area A, based on:
i) the data received from the sensory unit 2,
ii) the data received from the Localisation module 301,
iii) the set of permanent predetermined walkable area requirements, and
iv) the at least one predictable conditional walkable area requirements calculated and stored in the memory M of the Walkable Area Detection module 302.
The components of the free area A repeatedly updated, are repeatedly sent to the Live Map 310, thus updating the Live Map 310 content as outputted from S 2.3 with the layer referring to the components of the free area A.
S.2.5. The updated Live Map 310 is repeatedly stored in the memory M.
S.2.1. Details regarding the Live map determinations based on fused data
The Orientation Computation module 303 determines the current position and orientation of the Sensory unit 2 of the wearable device 1, of the plurality of the objects On and the plurality of living beings Ln in respect to the sensory unit 2 based on the inertial movement data provided by the Inertial Measurement unit 23. For this purpose, the Orientation Computation module 303 applies an orientation computation algorithm that calculates the orientation of the system on the 3 axes (pitch, roll, yaw) and for the 3D positioning of objects since the Camera 21 and Depth sensor 22 reveal where are the detected objects On in respect to the Camera 21, but not how they are oriented in respect to the ground.
The Object 2D Characteristics Extraction module 306 provides the pixel-wise segmentation of the 2D images acquired from the Camera 21, and detects in the pixel-wise segmented 2D images each object On of the plurality of the objects On, and each living being Ln of the plurality of living beings Ln placed in the field of view 20, and determines their respective position in 2D coordinates, and their respective physical characteristics.
The Object 2D Characteristics Extraction module 306 uses an Object 2D Characteristics Extraction Algorithm that combines several actions:
Thus, the Object 3D Characteristics Fusion module 307 determines the position each of the objects On in respect to the Sensory unit 2 in 3D coordinates, their physical characteristics, such as dimensions, composition, structure, colour, shape, humidity, temperature, degree of occupancy, degree of cleanliness, degree of usage, degree of wear, degree of stability, degree of fullness, degree of danger, and their orientation in respect to the Sensory unit 2, and the future position at predetermined moments in time in 3D coordinates based on the vector of movements, respectively.
The Object 3D Characteristics Fusion module 307 also determines data regarding position of each of the living beings Ln in 3D coordinates, their physical characteristics, like height, their skeleton pose orientation, and the prediction of its future position in the predetermined unit of time, respectively. Based on skeleton pose orientation, facial expression and their physical characteristics the Object 3D Characteristics Fusion module 307 determines the current activity and mood status of each of the living beings Ln.
The Sound Direction Localisation module 304 determines the direction of the plurality of sound streams expressed in 3D coordinates emitted respectively by each of the plurality of the objects On and the plurality of living beings Ln based on the data received from the Sound localisation sensor 24.
In one embodiment of the method, the direction of the plurality of sound streams is determined by comparing the differences of a sound stream between microphones of the Sound localisation sensor 24 while knowing the position of the microphones. The Sound Direction Localisation module 304 triangulates the source of the sound stream coming, detecting the direction from which the sound stream comes.
Each of the plurality of sound streams whose direction has been determined by the Sound Direction Localisation module 304 is classified into sound types by means of the Sound Classification module 305.
The Object Sound Characteristics Fusion module 308 adds acoustical characteristics to each of the plurality of the objects On and the living beings Ln for which the 3D coordinates have been determined based on the classified sound types determined by the Sound Classification module 305.
Then, the Object Sound Characteristics Fusion module 308 sends all fused data to the Live Map sub-unit 31 in order to be stored in the Live Map 310.
S.2.2. Details regarding the generation of the plurality of relationships Rn
The Live Map determinations based on the data received from the Relationship Manager sub-unit 32 provide further detailed information defining the environment of the visually impaired user. In this way the Relationship Manager sub-unit 32 of provides more accurate and detailed information about the objects On and the living beings Ln fulfilling the invention's objective of a safer navigation of the visually impaired user and a more concrete navigation goal, the latter being defined in the invention as the point of interest PI.
The algorithms used by the Processing and control unit 3 include but are not limited to: Object Detection, Object Pose Detection, Object Characteristics Determination. The algorithms used by the processing and control unit 3 define as item:
For example, in case of the object On is a four-leg chair, the chair is defined as a separate item from each one of its four legs.
The degree of itemization is predetermined being outside the scope of the invention.
The processing and control unit 3 creates clusters of objects based on their physical relationships.
Thus, predetermined parent-child relations connect the separate items so that they can form objects On, living beings Ln or ensembles between more than two objects On, more than two living beings Ln or objects On and living beings Ln. For example: the door handle belongs to the door. Both the door handle and the door are items. The main difference between the items on one hand, and the objects On and living beings Ln on the other hand is that the objects On and living beings Ln correspond to the usual expectation of the people about what an object and a living is, whereas for the algorithms all the objects On and the living beings Ln as well as their components are treated as items.
The predetermined conditional relations refer to connecting the separate items only if a condition is satisfied, for example the pedestrian crossing is a conditional walkable area, conditioned on the colour of the traffic light.
The Relationship Manager sub-unit 32 uses the data from the Live Map 310 to compute possible relations using specific algorithms.
For parent-child relations, non-limiting examples of algorithms are as follows:
Creation of new items based on detected relations:
For conditional relationships non-limiting examples of algorithms are as follows:
Depending on the type of object On or living being Ln, certain properties are transmissible from a parent to a child, for example:
All parameters used in the algorithms for establishing the plurality of relationships Rn are pre-determined: for example, for determining physical proximity predetermined ranges of distances are used.
S.2.3. Details regarding the localisation of the position and orientation of the sensory unit 2
The Localisation module 301 repeatedly determines current position and orientation of the sensory unit 2 of the wearable device 1 and of the plurality of the objects On and living beings Ln in respect to the sensory unit 2, in the 3D coordinates, on the current Live Map 310 by means of localisation algorithms applied to the data acquired from a Camera 21, the Depth sensor 22, an Inertial Measurement unit 23 of the Sensory unit 2.
The results of the localisation are sent to the Walkable Area Detection module 302 which determines the components of free area A.
S.2.4. Details regarding the determination of the components of the free area A.
The set of permanent predetermined walkable area requirements comprises categories that are predetermined for each visually impaired user, taking into consideration various general safety and comfort requirements.
The set comprises at least the following two categories:
The conditional walkable area CWA does satisfy the set of permanent predetermined walkable area requirements and must satisfy in addition the at least one predictable conditional walkable area requirement.
The set of permanent walkable area requirements as well as the at least one predictable conditional walkable area requirement are predetermined for each visually impaired user and stored in the memory M. The Walkable Area Determination module 302 applies said requirements to the data it receives from the Camera 21 and Depth sensor 22 on one hand and from the Localisation module 301 on the other hand, said data received from the Localisation module 301 including the updates of the relations as received from the Relationship Manager sub-unit 32 and stored in the Live Map 310.
In another preferred embodiment, parts of the Live Map 310 are downloadable form the internet from any geographical maps site, said parts referring to the layers described in S 2.1 to S.2.4 and taking into account that, depending on the geographical maps site from where map is downloaded, the information of each layer can be partial or complete. The download from the internet is carried out using a Communication unit 7, not represented graphically, connected to the internet. In this case, the Localisation module 301 localizes the position and orientation of the sensory unit 2 on the downloaded Live map 310.
In case there is a previously stored Live Map 310 in the memory of the Live map sub-unit 31, the Live Map 310 is created based on the Live Map determinations of the previously stored Live Map 310.
In case a previously stored Live Map 310 exists in the memory M of the wearable device 1, either because it was determined by the Live Map sub-unit 31 previously or because it was downloaded from the internet or both of them, the determinations based on the data received from the sensory unit 2 start with step 2.3 by the identification within previously stored Live Map 310 of the current position and orientation of the sensory unit 2 by means of localisation module 301, and identification of the free area A, including the walkable area WA and the conditional walkable area CWA by the Walkable Area Detection module 302 by means of localisation algorithms applied to the data received form the sensory unit 2.
Further on, the Live map 310 is repeatedly updated with additional information described in S 2.1 to S.2.2 and the remainder of step 2.3 and steps 2.4 and steps 2.5. are carried out as described above.
In a preferred embodiment, the Live Map (310) is updated by the Sensory fusion sub-unit (30) using Simultaneous Localisation and Mapping SLAM algorithms.
The SLAM algorithms are in particular advantageous since they use an iterative process to improve the estimated position with the new positional information. The higher the iteration process, the higher the positional accuracy. This cost more time for computation and high-configuration hardware with parallel processing capabilities of the processing units.
In another preferred embodiment the SLAM algorithms used are visual SLAM algorithms which have the benefits of providing vast information, being cheap and easy to implement since may be used passive sensors and components having extremely low size, weight, and power SWaP footprint.
The invention, as disclosed so far, refers to the cases where the point of interest PI is known to the visually impaired user before sending the initiation request.
In other cases, the visually impaired user has not sufficient information about the point of interest PI before sending the initiation request. Typical examples are when he arrives in a new environment, or when something has changed in the known environment, such as the usual places of the seats.
One example is when the visually impaired user enters a new room that has four windows. He wants to open a window. But which one of the four windows to select as point of interest PI? Or the visually impaired user enters a conference room where there are, say 30 occupied seats and 10 free seats. Which of the 10 free seats to choose as point of interest PI?
To encompass these cases where the visually impaired user needs additional information from his environment in order to select the point of interest PI before sending the initiation request, in another preferred embodiment, when the point of interest PI is not known by the visually impaired user, a sub-step 3-0 is carried out before all the other sub-steps of step 3:
In S.3-0.1. the visually impaired user sends an information request to a Sound representation sub-unit 36 of the Processing and control unit 3 regarding at least one object On selected from the plurality of objects On or at least one living being Ln selected from the plurality of living beings Ln, said at least one object On or at least one living being Ln as a potential point of interest PPI for the visually impaired user. An example of at least one object On selected from the plurality of objects On is a group of windows from a selected room, which may be named “window”.
The term “potential” means that any of the objects On from the group of objects On may be selected as initial point of interest PI.
The Sound representation sub-unit 36 is:
Taking the example of the room with four windows, the visually impaired user sends an information request named “window” through the User commands interface 5 to the Sound representation sub-unit 36 that he is interested to learn how many windows are in the room, their position in the room, the size or the shape of the windows, the position of their handles. The window is in this example the potential point of interest PPI. The information request refers to a predetermined area of interest which is in the proximity of the place where the visually impaired user stands at the moment when he sends the information request, which in this case is the room.
The information request is transmitted by the User commands interface 5 to the Sound representation sub-unit 36 via the User commands interface Manager sub-unit 34, just like the initiation request and the selection request.
In S.3-0.2. the Sound representation sub-unit 36 extracts from the Live Map 310 the information regarding the selected at least one particular object On or at least one particular living being Ln and represents said at least one particular object On or at least one particular living being Ln, respectively, as corresponding spatialized sounds and transmits same to the Feedback Unit 4, via the Feedback Manager sub-unit 35, when the Sound representation sub-unit 36 is not part of said Feedback Manager sub-unit 35.
The representation in spatialized sounds is generated by means of the Sound representation sub-unit 36 by encoding the classified sound types of the selected objects On or, respectively, selected living beings Ln based on predetermined spatialized sounds criteria.
The non-limiting and non-exhaustive examples of the predetermined spatialized sounds criteria are:
The type of encodings of the classified sound types of the selected objects On or, respectively, the selected living beings Ln based on predetermined spatialized sounds criteria is chosen based on testing procedures determining the ability of the user to distinguish various technical features of the sounds.
The visually impaired user is able to localize each spatialized sound using natural capabilities of the human beings to process sounds emanating from sound sources and following adequate training with the wearable device 1.
The localization of the spatialized sounds is carried out in three spatial dimensions:
In S. 3-0.3, the visually impaired user selects the point of interest PI from said specific plurality of objects On or, respectively, from said plurality of living beings Ln and transmits the corresponding selection to the Navigation Manager sub-unit 33.
The group of examples No. 2 details the matter of the sound representation.
In some situations, the point of interest PI is not in the Live Map 310, for example, when the visually impaired person arrives to a new destination.
In this case, the Live Map unit 31 sends to the Navigation Manager sub-unit 33 and to the User commands interface Manager sub-unit 34 the confirmation that the point of interest PI is not in the Live Map 310. The method has an additional sub-step in S3 before determining, repeatedly updating and storing the at least one navigation path (Pn):
S 3-1 The Navigation Manager sub-unit 33 determines a wandering path WP -not represented graphically, while S1 and S2 of the method are repeated until the point of interest PI is found and stored in the Live Map 310, said wandering path WP satisfying the at least two navigation path requirements.
It is possible to determine the wandering path WP while the Navigation Manager sub-unit 33 represents as corresponding spatialized sounds specific category of objects On or said specific category of living beings Ln. Once the decision as to the selection of the point of interest PI is taken, the remainder of step 3 and the step 4 of the method are carried out as disclosed.
Details regarding S4
All the guiding modes have the purpose to keep the visually impaired user, when navigating, on the preferred navigation path SP. Each preferred navigation path SP has its own degree of complexity that corresponds to the variety of navigating situations arising from real life. The inventors thought to quantify the degree of complexity of the preferred navigation paths SP by using scores corresponding to predetermined preferred navigation path SP complexity criteria, which include both objective criteria and subjective criteria, the latter being the own interpretation of the visual impaired user of the objective criteria: e.g. what is perceived as a long distance for a specific visually impaired user is not perceived as long for other visually impaired user, the same with noise or temperature of the immediate environment.
Below are presented some non-limiting and non-exhaustive examples of the predetermined preferred navigation path complexity criteria:
The haptic cues vary in duration, periodicity, intensity or frequency of the vibration according to predetermined preferred navigation path complexity criteria.
The audio cues vary in frequencies, duration, repetition, intensity, or 3D spatial virtualization according to the predetermined preferred navigation path complexity criteria.
The variation of the haptic cues and, respectively audio cues, has the advantage of adapting the guidance of the visually impaired user to the degree of complexity of each preferred navigation path as quantified by the predetermined preferred navigation path SP complexity criteria.
The advantages of the variation of the characteristics of the haptic cues and of the auditory cues as well as the possibility to combine haptic and auditory cues are as follows:
The haptic cues are received through the haptic feedback actuators 41. The visually impaired user receives training before use of the wearable device 1 in order to associate each type of haptic cue with the specific guiding instruction.
With reference to
Said haptic feedback actuators 41 include vibrating actuators and close-range remote haptics such as ultrasonic haptic feedback actuators.
Vibrating actuators comprise a plurality of resonant actuators converting the electric signals received from the Feedback Manager 35 into forced vibrations felt on the forehead of the visually impaired user, said vibrations associated with a specific guiding instruction.
A non-limiting example of vibrating actuator used in the invention is a linear resonant actuator. Each of the left haptic feedback actuators 411, right haptic feedback actuators 412 centre haptic feedback actuators 413 can comprise one or more linear resonant actuators.
Using the linear resonant actuators is advantageous for the invention because of their known good haptic performance, their improved efficiency at resonance compared with other vibrating actuators, their capacity of optimizing power consumption and their small size which allows configuring them for example in the form of a matrix, if more than three directions of guiding are envisaged.
There are two types of haptic cues:
Non-limiting examples of applying predetermined preferred navigation path complexity criteria are given below:
The types of haptic cues are predetermined for each case depending on the needs of the visually impaired user. An example of predetermination of haptic cues is presented below for a better understanding of the teaching of the invention, and not for limiting same:
Further haptic cues can be defined to accommodate other navigation situations or requirements of the visually impaired user.
To ensure a more accurate guidance and to avoid at the same time unnecessary overloading of the visually impaired user with haptic cues, it is possible to combine the types of haptic cues.
E.g.:
The assignment of each type of haptic cue to one or more from the feedback actuators 41 used is predetermined.
Auditory Cues
Auditory cues are sounds perceptible by humans received through the auditory feedback actuators 42 in the ears of the visually impaired user.
The auditory feedback actuators 42 are speakers, headphones or bone-conduction speakers converting the electric signals received from the Feedback Manager sub-unit 35 into sounds. The associated navigation guiding instructions received through the auditory feedback actuators 42 are based on the principle of assigning a specific sound to each associated navigation guiding instruction.
With reference to
Each of the left auditory feedback actuators 421 and right auditory feedback actuators 422 can comprise a plurality of speakers, headphones or bone-conduction speakers placed on the same azimuth.
The types of auditory cues are predetermined for each case depending on the needs of the visually impaired user. An example of predetermination of auditory cues is presented below for a better understanding of the teaching of the invention, and not for limiting same:
Further types of auditory cues can be defined to accommodate navigation situations or requirements of the visually impaired user.
The assignment of each type of auditory cue to one or more from the auditory feedback actuators 42 is predetermined.
Considering the origin of the sounds, there are two types of sounds:
In one preferred embodiment, depicted in
The three-dimensional walkable tunnel T is determined by the Navigation Manager sub-unit 33 at the same time with the preferred navigation path SP, and then sent to the Feedback Manager sub-unit 35 together with the haptic cues.
The advantage of the walkable tunnel T is that it allows a more comfortable navigation of the visually impaired user with a larger degree of liberty to the left and to the right defined by the virtual walls of the walkable tunnel T.
The guiding cues are transmitted when the visually impaired user is reaching the virtual walls of the walkable tunnel T so that he returns within the space defined virtual walls of the walkable tunnel T. In some embodiments, apart from the guiding cues signalling the virtual walls of the walkable tunnel T, other guiding cues are transmitted to confirm that the visually impaired user is navigating safely within the virtual walls of the walkable tunnel T.
The cross-section of the walkable tunnel T is predetermined depending on the plurality of the possible cross-sections along the preferred navigation path SP and on the visually impaired user's preferences.
The example No. 1 details the guiding modes using the walkable tunnel T.
In another preferred embodiment, with reference to
The guiding mode of S4 comprises haptic cues and/or auditory cues signalling the position of a next milestone 93 providing associated navigation guiding instructions to the visually impaired user from a current milestone 93 to the subsequent milestone 93. When the visually impaired user has already passed the subsequent milestone 93, said subsequent milestone 93 becomes the current milestone 93 and so on.
The length of the predetermined segments varies depending on the complexity and length of the preferred navigation path SP.
The length of each segment between two consecutive milestones 93 is inversely proportional with the predetermined preferred navigation path complexity criteria: the more complex the preferred navigation path SP, the shorter each segment. The milestones 93 are more frequent in the portions that contain change of direction in either horizontal or vertical plane than in the portions of going straight.
The length of each segment between two consecutive milestones 93 is determined by applying the predetermined preferred navigation path SP complexity criteria, which means that the length of the segments along the preferred navigation path SP is not necessarily equal, as seen in FIG. 9.
The length of each segment can be calculated using scores corresponding to said predetermined preferred navigation path complexity criteria or can be adapted dynamically for using artificial intelligence-based learning methods. For example, if the visually impaired user has some preferred navigation paths SP that are repetitive and he selects guiding method by using milestones as favourite, it is convenient to use said learning methods to adapt dynamically the length of the milestones.
The cues used in the guiding mode from the current milestone 93 to the subsequent milestone 93 are:
A non-limiting example of using the haptic cues is as follows:
The variation of the duration, periodicity, intensity or frequency of the vibration of the haptic pattern cues is directly proportional to the predetermined preferred navigation path complexity criteria and at the same time they vary inversely proportional to the distance left until the subsequent milestone 93.
A non-limiting example of using the auditory cues is as follows:
The spatialized sounds vary directly proportional in frequencies, duration, repetition, intensity, and 3D spatial virtualization according to the predetermined preferred navigation path complexity criteria and at the same time they vary inversely proportional to the distance left until the subsequent milestone 93.
Using only auditory cues is advantageous in the situation when there is only one subsequent milestone 93 that coincides with the point of interest PI: for example, if the visually impaired user needs to go from the sofa to the kitchen, in this case the kitchen being the only one subsequent milestone 93. The spatialized auditory cue corresponds in this case to the kitchen. Using auditory cues has the advantage of simplicity and predictability, because it provides the visually impaired user the possibility to associate the distance left to be navigated until the subsequent milestone 93 with the corresponding auditory cue heard from the position of said subsequent milestone 93, which improves his degree of orientation and feeling of safety when navigating. Using only auditory cues is preferred when the point of interest PI is known to the visually impaired user and the distance to be travelled until the point of interest PI is short, for example for the navigation paths inside the house.
When the guiding mode from the current milestone 93 to the subsequent milestone 93 is by haptic and auditory cues, one between said haptic and auditory cues may be defined as primary and the other one as secondary, the secondary outputting cues only in special predetermined situation, such as for example the seventh cue instructing to stop and resume.
In another preferred embodiment, with reference to
Combining the two guiding modes has the advantage of combining the advantage of each of the guiding modes: the comfort and the safety of the walking tunnel with the simplicity and predictability of the guiding mode from the current milestone 93 to the subsequent milestone 93.
In another preferred embodiment, the guiding mode of S4 consists in haptic cues or auditory cues or haptic and auditory cues signalling the direction on the preferred navigation path SP.
A non-limiting example of using the haptic cues is as follows:
The haptic pattern cues are predetermined such that they give the impression to the visually impaired user to be dragged by his forehead constantly towards the direction in which he moves by a person standing in front of him.
A non-limiting example of using the auditory cues is as follows:
The spatialized sounds vary directly proportional in frequencies, duration, repetition, intensity, or 3D spatial virtualization according to the predetermined preferred navigation path complexity criteria. The visually impaired person, when navigating, follows the direction of the spatialized sound source S.
The main difference between the guiding mode based on signalling the direction on the preferred navigation path SP and based on the guiding mode from the current milestone 93 to the subsequent milestone 93 refers to the variation of the features of the haptic pattern cues, and respectively spatialized sounds:
The use of haptic cues or auditory cues signalling the direction on the preferred navigation path SP is advantageous to be used in situations when the degree of complexity of the preferred navigation path SP is lower than in the case of using the guiding mode from the current milestone 93 to the subsequent milestone 93 or the guiding mode of the walking tunnel T. One such example is when the same preferred navigation paths SP are used frequently. The advantage of the use haptic cues or auditory cues signalling the direction on the preferred navigation path SP is that they produce less fatigue to the visually impaired user.
One non-limiting example of using haptic cues signalling the direction on the preferred navigation path SP is given in
In another non-limiting example of using auditory cues signalling the direction on the preferred navigation path SP is given in
The spatialized sound source S is placed at a predetermined first distance d1 of the spatialized sound source S in respect to the Sensory unit 2.
In order to obtain flexibility in the guiding modes and to adapt said guiding modes to the degree of complexity of the preferred navigation path SP, the predetermined first distance d1 of the spatialized sound source S in respect to the Sensory unit 2 can be either smaller than the predetermined length of the radius r—as depicted in
It is possible to combine haptic cues with auditory cues, the combination not being represented graphically.
In another preferred embodiment, with reference to
A non-limiting example of using the auditory cues is as follows:
The auditory feedback actuators 42 repeatedly output the spatialized sound source S by means of variation of the frequencies, duration, repetition, intensity, and 3D spatial virtualization directly proportional to the predetermined preferred navigation path complexity criteria.
The predetermined second distance d2 is inversely proportional to the predetermined preferred navigation path SP complexity criteria, that is the more complex the preferred navigation path SP is, the smaller the predetermined second distance d2.
The predetermined second distance d2 typically varies between 0.2 m and 5 m. If the preferred navigation path SP is very complex, the predetermined second distance d2 typically varies between 0.2 and 1 m. The examples of the values for the predetermined second distance d2 are given for illustration purpose only and shall not be considered as limiting.
Example: the predetermined second distance d2 is 1.2 m. This means that the spatialized sound source S is virtually travelling at 0.2 m from the position of the sensory unit 2. The spatialized sounds travel back and forth from the position of the sensory unit 2 until they reach 1,2 m in the direction of navigation and then they come back to the position of the sensory unit 2. As the speed of the sound is significantly higher than the speed of human walk, the visually impaired user receives the navigating guiding instructions in more detail than in any other guiding mode disclosed in this invention, because in the guiding mode using the virtual travel of the spatialized sounds the sounds travel independently from the visually impaired user.
The features of the sounds, namely any between frequencies, duration, repetition, intensity, and 3D spatial virtualization or combinations of them, vary inversely proportional with the distance left until the predetermined second distance d2. For example, the auditory cues are more frequent and/or more intense and/or more 3D spatially virtualized or last longer when the spatialized sound source S is at 0.1 m than when the spatialized sound source S is at 0.2 m.
The advantage of this guiding mode is that it allows a fine tuning of the navigation which makes it advantageous in environments where the walkable area WA is very narrow and, consequently, the preferred navigation path SP looks like a slalom between the objects On and the living beings Ln.
In a second aspect of the invention, the wearable device 1 comprises the Sensory unit 2, the Processing and control unit 3, the Feedback unit 4, the User commands interface 5.
The wearable device 1 comprises two hardware units not represented graphically: the Power storage unit 6, and the memory M.
The term “memory M” shall be understood as designating a plurality of non-volatile memories either grouped together in a single distinctive hardware unit or spread in each of the other hardware units.
The memory M is configured to store at least the Live Map 310, all the algorithms, all the criteria and requirements and the preferences of the visually impaired user such as but not limited to the type of cues he prefers for receiving the guiding instructions. The storage is carried out according to prior art.
The wearable device 1 is, in a preferred embodiment, a single-component device, whereas in other preferred embodiments is a multi-component device.
In case of the single-component device 1, all the hardware units are included in the wearable device 1 as shown in
In case of the preferred embodiments of the multi-component device 1, with reference to
Two non-limiting examples of the preferred embodiments of the multi-component device 1 depict two components:
In this case, the belt-worn component 12, or, respectively, the wrist component 12 comprises the processing and control unit 3, the User commands interface 5, and the power storage unit 6. The memory M can be comprised in any of the two components or spread among them.
1 and
The division of the components among the headset component 11 and the belt-worn component 12, or, respectively, the wrist component 12 is mainly based on the size and weights of the units.
The advantage of using the single-component device 1 is that its preferred location on the head produces a sensorial experience for the visually impaired user of the wearable device 1 very close to the sensorial experience of the non-visually impaired person, being close to the position of the ears which enables hearing the auditory cues.
However, in some cases, some hardware units, such as the Processing and control unit 3 and/or the Power storage unit 6 may be heavy and bulky. In these cases, the multiple-component device 1 has the advantage of placing the heavy and bulky hardware units in other locations of the body such as but not limited to the belt or the wrist.
As the technology evolves in general towards miniaturization of hardware units, this will lead to increase the possibility of using the single-component device 1 without placing too much burden on the head of the visually impaired user.
In another preferred embodiment, not represented graphically, there are three components:
The memory M can be comprised in any of the headset component 11 or the belt-worn component 12, or, respectively, the wrist component 12 or spread among the two.
The configuration of the various units composing the wearable device 1 in order to work the invention is not influenced by the positioning of said hardware units in the one—or, respectively multiple-component device to the various parts of the human body.
The hardware units communicate between themselves either by wired communication protocols or by wireless communication protocol, or by a combination of wired and wireless protocols, said communication taking place according to prior art.
The Sensory Unit 2
The Sensory unit 2 has means configured to collect data regarding the environment of the visually impaired user.
The data collected by the Sensory unit 2 refers to multiple characteristics of objects On and living beings Ln that are generally identified by a human of good sensory capabilities including good vision. The data, as collected by the Sensory unit 2, reflects the complexity of the environment with more accuracy than in the state of art.
To satisfy the aim of collecting more accurate data, the Sensory unit 2 requires a combination of sensors of multiple types that will be described in detail. It shall be understood that all examples of sensors are for a better understanding of the teaching of the invention and shall not limit the invention.
The Sensory unit 2 comprises four basic sensors: a Camera 21, a Depth sensor 22, a Inertial Measurement unit 23 and a Sound localisation sensor 24.
The best position of the Camera 21, the Depth sensor 22, and the Inertial Measurement unit 23—irrespective of whether the wearable device 1 is a single-component or a multi-component device, is on the forehead as shown in
The configuration of the positioning of the Sensory unit 2 on the forehead of the visually impaired user must ensure that the field of view 20 includes:
The first sensor is the Camera 21. The term “Camera 21” designates throughout the invention, one or several digital video cameras. The invention requires to have at least digital video camera.
The Camera 21 is configured to acquire 2D images from a Camera field of view, and to send the acquired 2D images to the Localisation module 301, to the Walkable Area Detection module 302, and to the Object 2D Characteristics Extraction module 306.
The term “images” encompasses the static images as well as the videos, depending on the frame rate of acquisition of the images of the Camera 21.
The images acquired by the Camera 21 refer to the visual characteristics of the plurality of objects On and of the plurality of living beings Ln such as aspect; category—e.g. trees cars; colour, shape, dimensions as well as the components of the free area A.
Non-limiting examples of Camera 21 include: HD Camera, having minimum video resolution 1280 pixels×720 pixels, VGA Camera, having minimum video resolution 320 pixels×240 pixels,
The minimum requirements of the Camera 21 are as follows:
The Camera 21 can be RGB Camera or not. The RGB features help to provide more accurate information from the Camera field of view.
The more complex the Camera is, the more information will contain the 2D images acquired by the Camera.
The second sensor is the Depth sensor 22. The term “Depth sensor 22” designates throughout the invention one or several depth sensors. The invention requires to have at least one depth sensor.
The Depth sensor 22 is configured to acquire 3D point clouds data corresponding to 3D distance position and dimension for each of the objects On and each of the living beings Ln placed in the Depth sensor field of view as a continuous point cloud, and to send them to the Localisation module 301, to the Walkable Area Detection module 302, and to the Object 3D Characteristics Fusion module 307.
The 3D point cloud data acquired by the Depth sensor 22 refers to the 3-D physical characteristics of the objects On and the living beings Ln such as density, volume, etc.
Non-limiting examples of Depth sensor 22 are stereoscopic camera, radar, Lidar, ultrasonic sensor, mmWave radar sensor. Using mmWave radar sensor is advantageous because it is able to sense the pulse or the breath of the living beings Ln, even when the living beings Ln are moving which brings additional information for the visually impaired user.
It is possible to combine the Camera 21 and the Depth Sensor 22 in a single sensor Camera and Depth Sensor 21-22. The advantage is reducing the size and weight of the two afore-mentioned sensors by using only one sensor configured to carry out the tasks of the two sensors. One non-limiting example of Camera and Depth Sensor 21-22 would be a time of flight TOF camera. The third sensor is the Inertial Measurement unit 23. The term “Inertial Measurement unit 23” designates throughout the invention an ensemble made of at least one accelerometer and at least one gyroscope and, either as separate sensors, or combined sensor. It is preferable to add at least one magnetometer for better accuracy, either as a separate sensor or combining it with the at least accelerometer and/or the at least gyroscope. It is better to use combined sensors because of the need to reduce the size and weight of the ensemble. The invention requires to have at least one inertial measurement unit.
The Inertial Measurement unit 23 is configured to determine the orientation of the Sensory unit 2, and to send the determined orientation to the Localisation module 301, and to the Characteristics Fusion module 307 by means of the Orientation Computation module 303.
Since the Sensory unit 2 is placed on the forehead of the visually impaired user, the information acquired by the Inertial Measurement unit 23 implicitly refers to orientation of the head of the visually impaired user in respect to the ground.
The fourth sensor is the Sound localisation sensor 24.
The term “Sound localisation sensor 24” designates throughout the invention an ensemble of one or several sensors used to determine the source of various sounds in the three-dimensional space usually by the direction of the incoming sound waves and the distance between the source and sensor(s).
The Sound localisation sensor 24 is configured to acquire a plurality of sound streams in the three-dimensional space emitted by the objects On and the living beings Ln, and to send them to the Sound Direction Localisation module 304.
The information acquired by the Sound localisation sensor 24 refers to the sounds emitted by the objects On and the living beings Ln, including the directionality of said sounds. The coverage of the environment by the Sound localisation sensor 24 is defined by its beam pattern.
A non-limiting example of sound localisation sensor is a microphone array. The minimum number of microphone arrays used for the Sound localisation sensor 24 must be such that the sum of the beam pattern equals to the angle of the field of view 20. The maximum number of microphone arrays used for the Sound localisation sensor 24 covers 360°. The microphone arrays are positioned within the headset such that the sum of their beam pattern be comprised between the angle of the field of view 20 and 360°.
The basic sensors receive from the Sensory fusion sub-unit 30 of the Processing and control unit 3 specific configurations, including the correlation of the respective field of views of the Camera 21, Depth sensor 22, with the range of measurement of the Inertial Measurement unit 23 and the beam pattern of the Sound localisation sensor 24.
Said correlation has as result the field of view of the basic sensors 20, depicted schematically in
However, the Sound localisation sensor 24 may have a wider range that the field of view of the basic sensors 20, for example when the number of microphone arrays is such that the sum of the beam pattern equals to 360°. This is advantageous because it allows gathering sound information originating from the back of the visually impaired user.
In another preferred embodiment, depicted in
Any combination of each of the additional sensors with the group of basic sensors has the advantage of providing additional information to the Processing and control unit 3 which leads to a more accurate and detailed Live Map 310.
Each of the two additional sensors has a corresponding module in the Sensory Fusion sub-unit 30, as follows:
The Global positioning sensor 25 is configured to determine the absolute position of the Sensory unit 2 and to send the determination to a Relative to Absolute Conversion module 309-1 that converts the relative position of the Sensory Unit 2 into absolute position, thus the position of the objects On and the position of the living beings Ln is expressed as absolute position.
The best position of the Global positioning sensor 25 is on the top of the headset component 11 of the wearable device 1 in case of multi-component device, respectively on the top of the wearable device 1 in case of single component device.
In the absence of the Global positioning sensor 25, the Sensory Fusion sub-unit 30 determines the relative position of the wearable device 1 in respect to each of the objects On and to each of the living beings Ln.
The Temperature sensor 26 is configured to determine the temperature of the objects On and of the living beings Ln, and to send the determined temperature to an Object Temperature Characteristics fusion module 309-2.
In case of using either of the additional sensors, the data outputted by the Object Sound Characteristics Fusion module 308 is sent to either the Relative to Absolute Conversion module 309-1 or the Object Temperature Characteristics fusion module 309-2 respectively, fused with the data sent by the respective sensor and the outcome is sent to the Live Map sub-unit 31.
In case of using both additional sensors, as depicted in
The Processing and Control Unit 3
The Processing and control unit 3 is a computing unit, comprising at least one processor and at least one non-volatile memory, such as but not limited to a microcontroller, a computer, a supercomputer. The term “computing unit” encompasses a single computing unit or a plurality of computing units located remotely from one another communicating within a computer communication system.
The Processing and control unit 3 comprises: the Sensory fusion sub-unit 30, the Live Map sub-unit 31, the Relationship Manager sub-unit 32, the Navigation Manager sub-unit 33, the User commands interface Manager sub-unit 34, the Feedback Manager sub-unit 35, and the Sound representation sub-unit 36.
With reference to
With reference to
The Localisation module 301 comprises means configured to localize the current position and orientation of the sensory unit 2 of the wearable device 1 and of the plurality of the objects On and living beings Ln in respect to the sensory unit 2, in 3D coordinates, on the current Live Map 310 by means of localisation algorithms applied to the data acquired from the Camera 21, the Depth sensor 22, the Inertial Measurement unit 23 of the Sensory unit 2.
The Localisation module 301 further comprises means configured to send the localisation of the position and orientation of the sensory unit 2 to the Walkable Area Detection module 302, thus updating the Live Map 310 content as outputted from S 2.2 with the layer referring to the localisation data of the position and orientation of the sensory unit 2 in respect to the plurality of the objects On, and, respectively to the plurality of living beings Ln.
The Walkable Area Detection module 302 comprises means configured to receive the data acquired from the Camera 21, the Depth sensor 22, and means configured to receive data from the Localisation module 301, and, based on both sources of data, means configured to define the walkable area WA, and the conditional walkable area CWA, and send them to the Live Map sub-unit 31, by applying the set of permanent predetermined walkable area requirements and predictable conditional walkable area requirements, stored in the memory M.
The Orientation Computation module 303 comprises means configured to determine the orientation of the wearable device 1 based on the inertial data provided by the Inertial Measurement unit 23, and to sends the determinations to Object 3D Characteristics Fusion module 307.
The Sound Direction Localisation module 304 comprises means configured to determine the direction of the plurality of sound streams expressed in 3D coordinates emitted respectively by each of the plurality of the objects On and the plurality of living beings Ln based on the data received from the Sound localisation sensor 24 and means configured to send the determined direction to the Sound Classification module 305.
The Sound Classification module 305 comprises means configured to classify into sound types the plurality of sound streams received from the Sound Direction Localisation module 304 and to send the classified sound types to the Object Sound Characteristics Fusion module 308. The means configured to classify into sound types the plurality of sound streams typically use artificial intelligence algorithms.
The Object 2D Characteristics Extraction module 306 comprises means configured to provide the pixel-wise segmentation of the 2D images acquired from the Camera 21, to detect in the pixel-wise segmented 2D images each object On of the plurality of the objects On, and each living being Ln of the plurality of living beings Ln placed in the field of view 20, to determine their respective position in 2D coordinates, and their respective physical characteristics and to send the determinations to the Object 3D Characteristics Fusion module 307.
The Object 3D Characteristics Fusion module 307 comprises means configured to receive data from the Object 2D Characteristics Extraction module 306, from the Orientation Computation module 303 and from the Depth sensor 22, and to determine:
The Object Sound Characteristics Fusion module 308 comprises means configured to add acoustical characteristics to each of the plurality of the objects On and the living beings Ln for which the 3D coordinates have been determined based on the classified sound streams types determined by the Sound Classification module 305 by associating with the detected objects On and the living beings Ln and send all data to the Live Map sub-unit 31.
In an embodiment of the present invention, the Sensory fusion sub-unit 30 further comprises the Relative to Absolute Conversion module 309-1. This module comprises means configured to convert the relative position of the Sensory Unit 2 into absolute position, to fuse the data from the Object Sound Characteristics Fusion module 308 with the data regarding absolute position of the Sensory unit 2 and to send the determinations to the Live Map sub-unit 31 either directly or by means of the Object Temperature Characteristics fusion module 309-2.
In another embodiment of the present invention, the Sensory fusion sub-unit 30 further comprises the Object Temperature Characteristics fusion module 309-2. This module comprises means configured to determine the temperature of the detected objects On and the living beings Ln, to fuse the data from the Object Sound Characteristics Fusion module 308 with the data regarding the temperature of the objects On and of the living beings Ln and to send the fused data to the Live map sub-unit 31. If the Relative to Absolute Conversion module 309-1 is used, it sends the data to the Object Temperature Characteristics fusion module 309-2 and finally fuses the data with the data regarding the temperature of the objects On and of the living beings Ln.
According to the invention, the Live Map sub-unit 31 comprises means configured to create, repeatedly update and store the Live Map 310 and means to receive data referring the components of the free area A and the updated Live Map 310 content as outputted from S 2.2 with the layer referring to the localisation of the position and orientation of the sensory unit 2 from the Walkable Area Detection module 302, data regarding each of the plurality of the objects On and the living beings Ln in 3D coordinates including acoustical characteristics from the Object Sound Characteristics Fusion module 308, and to send all Live map determinations to the Localisation module 301.
The Live Map sub-unit 31 comprises means configured to receive:
The Live Map sub-unit 31 comprises means configured to send:
Further on, the Relationship Manager sub-unit 32 comprises means configured to carry out computations for determining and updating the relations between the plurality of objects On and/or the plurality of living beings Ln, and to send the updated relations as result of the computations to the Live map sub-unit 31 to store same in the Live Map 310.
The Navigation Manager sub-unit 33 comprises means configured to:
The User commands interface Manager sub-unit 34 comprises means configured to receive requests and selections that the visually impaired user makes by means of the User commands interface 5 and to transmit them to the Navigation Manager sub-unit 33 and means configured to send selected guiding modes to The Feedback Manager sub-unit 35.
The User commands interface Manager sub-unit 34 further comprises means for receiving requests from the visually impaired user for sound representation of a specific category of objects On or a specific category of living beings Ln from the Live Map 310.
The Feedback Manager sub-unit 35 comprises means configured to guide the visually impaired person along the preferred navigation path SP by receiving the guiding instructions from the Navigation Manager sub-unit 33 together with selected guiding modes from the User commands interface Manager sub-unit 34 and means configured to transmit the corresponding associated guiding instructions to the Feedback unit 4, and further comprises means for sending the sound representation regarding a specific category of objects On or a specific category of living beings Ln.
In the embodiments where the Sound representation sub-unit 36 is a self-contained sub-unit and a sub-unit of the Navigation Manager sub-unit 33, the Feedback Manager sub-unit 35 further comprises means for receiving sound representation of the specific category of objects On or a specific category of living beings Ln from the Sound representation sub-unit 36.
The Sound representation sub-unit 36 comprises means configured to receive requests from the visually impaired and to extract from the Live Map 310 of the corresponding information regarding a specific category of objects On or a specific category of living beings Ln and means for representing the extracted information as corresponding spatialized sounds and transmitting same to the Feedback Unit 4.
The Feedback unit 4, configured to be placed on the head of the visually impaired user, comprises means configured to guide the visually impaired user along the preferred navigation path SP by receiving the associated guiding instructions from the Feedback Manager sub-unit 35 and by sending the haptic and/or auditory cues to the visually impaired person as it was described in detail in the section regarding the details of the step 4 of the method, and comprises means for sending to the visually impaired user the sound representation the specific category of objects On or a specific category of living beings Ln.
The User commands interface 5, configured to be placed on the head of the visually impaired user, comprises means configured to receive from the visually impaired user the requests, namely the initiation request, the selection request and the information request and the selections of the guiding modes and to send them to the User commands interface Manager sub-unit 34.
Non limiting examples of the User commands interface 5 are as follows:
The communication of the User commands interface 5 with the visually impaired person and with User commands interface Manager sub-unit 34 is according to prior art.
The Power Storage Unit 6
The term “Power storage unit 6” shall be understood as designating one or several batteries configured to power the other hardware units of the wearable device 1. The way the Power storage unit 6 powers said other hardware units of the wearable device 1 is carried out according to prior art.
The Communication unit 7 comprises means configured to download maps from the Internet, such as but not limited to the downloadable maps.
In a third aspect of the invention, it is provided a computer program comprising instructions which, when the program is executed by the wearable device 1 causes the wearable device 1 to carry out the steps of the computer-implemented method for assisting the movement of a visually impaired user, in any of the preferred embodiments, including combinations thereof.
In a fourth aspect of the invention, it is provided a computer readable medium having stored thereon instructions which, when executed by the wearable device 1, causes the wearable device 1 to carry out the steps of the computer-implemented method, in any of the preferred embodiments, including combinations thereof.
In a fifth aspect of the invention, it is provided a non-transitory computer-readable storage device storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations of the computer-implemented method, in any of the preferred embodiments, including combinations thereof.
In a sixth aspect of the invention, it is provided a system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations of the computer-implemented method, in any of the preferred embodiments, including combinations thereof. The terms “computers” of the fifth and sixth aspects refer to a computing unit, comprising at least one processor and at least one non-volatile memory, such as but not limited to a microcontroller, a computer, a supercomputer. The term “computing unit” encompasses a single computing unit or a plurality of computing units located remotely from one another communicating within a computer communication system.
The detailed description of the method is exemplified in a real-life scenario, with reference to the
In the real-life scenario, the visually impaired person 1 is on the sidewalk of a street in the close proximity of the entrance to a building. He wants to get into the building thus has to navigate from his standpoint until the entrance door of the building and has also to find the doorbell of the entrance door.
This is a non-limiting example when the visually impaired user sends the initiation request in order to be guided to the entrance door of the building.
In
An example of the geometric predetermined walkable area requirements include: the height of the sidewalk must not exceed 7 cm, the distance to the fences must not exceed 0.5 m, the distance to the margins of the sidewalk must not exceed 0.5 m, the height of the virtual cuboid is 2.20 m, that is 40 cm more than the height of the visually impaired person that is 1.80 m. An example of static and dynamical physical relationships Rn is the relation created in the live map 310 by the Relationship Manager sub-unit 32 of the Processing and control unit 3 with respect to associating the colour of the traffic lights 831 to the conditional status of the conditional walkable area 943: if the colour is green, the area 943 is walkable whereas if the colour is red, the area 943 is non-walkable.
A non-limiting example for the conditional walkable area CWA is represented by the two pedestrian crossings 832 provided with traffic lights 831. The streets are defined as non-walkable area NA in the permanent predetermined walkable area requirements. When it comes to the pedestrian crossings 832, in case there are no traffic lights, they are predefined as walkable area 942, whereas in case there are traffic lights, they are predefined as conditional walkable area 943 that is they are walkable only when the colour of the traffic lights 831 is green. This is an example of at least one predictable conditional walkable area requirement, as colour of the traffic lights changes predictably changing from red to green and from green to red.
The visually impaired user 1 is on the sidewalk of the building when he sends the initiation request.
In several embodiments of the invention, the entrance door 84 is already in the Live Map 310 because it was added to it in the step 2 of the method in the past.
In the embodiment of the invention where said entrance door 84 is not yet in the Live Map 310 at the moment of sending the initiation request because the visually impaired user has just got off from a taxi to a completely new place and consequently the entrance door 84 was never added before to the Live Map 310, the Navigation Manager sub-unit 33 determines the wandering path WP to repeatedly refocus the field of view 20, while S1 and S2 of the method are repeated until said entrance door 84 is found and stored in the Live Map 310.
In the embodiment of the invention where the entrance door 84 is not known by the visually impaired user, because the visually impaired user has just got off from a taxi to a completely new place where are two entrance doors 84-01 and 84-02 one close to another, the visually impaired user sends an information request to the Navigation Manager sub-unit 33 for finding “entrance door”. Then the Navigation Manager sub-unit 33 queries the live Map 310 for the entrance doors in the area of interest from the proximity of the visually impaired user and finds that there are two entrance doors 84-01 and respectively 84-02.
If the two entrance doors 84-01 and 84-02 are not already stored in the Live Map 310, the Navigation Manager sub-unit 33 determines the wandering path WP until said entrance doors 84-01 and 84-02 are found and stored in the Live Map 310.
Once the two entrance doors 84-01 and 84-02 are found and stored in the Live Map 310 the Navigation Manager sub-unit 33 represents each of them as corresponding spatialized sounds and transmits same to the Feedback Unit 4 via the Feedback Manager sub-unit 35. Then the visually impaired user selects one among the entrance doors 84-01 and 84-02 as the entrance door 84 that constitutes his initial point of interest.
The Navigation Manager sub-unit 33 determines in S3 a single navigation path Pn, namely, an initial navigation path 911 for the visually impaired user to navigate from his standpoint to the entrance door 84. The preferred navigation path SP is thus the initial navigation path 911.
When the visually impaired user 1 navigates along the initial navigation path 911, the dog 83 is sensed by the Sensory unit 2.
The aggressivity of the dog is sensed as follows:
Since the data sensed by the basic sensors and, where applicable, by the additional sensors is fused and then sent to the Live Map sub-unit 31 such that to be included in the Live Map 310, the Navigation Manager sub-unit 33, when querying the Live Map 310, checks the at least two navigation path requirements and detects that the non-aggressivity requirement is not met. For this reason, the Navigation Manager sub-unit 33 it determines a secondary navigation path 912 towards the same initial point of interest PI 84. The preferred navigation path SP is now the secondary navigation path 912, which avoids the dog 83 having an adverse reaction.
With reference to
When the visually impaired user 1 approaches the first pedestrian crossing 832, the Relationship Manager sub-unit 32 determine that the conditional area 943 is conditioned by the colour of the first traffic light 831.
Therefore, a conditional relation is built in the Live Map 310, by the Relationship Manager sub-unit 32, relating the colour of the first traffic light 831 to the conditional status of the first pedestrian crossing 832.
When the traffic light 831 turns green, the conditional walkable area 943 is considered walkable and the visually impaired user 1 receives the associated navigation guiding instruction to continue the navigation on the secondary path 912.
The same repeats on the second pedestrian crossing 832.
With reference to
In
If the preferred navigation path SP passes through an indoor space, such as an apartment, the cross-section is usually smaller, for example around 0.5 that is around 0.25 m to the left and around 0.25 m to the right of said preferred navigation path SP.
The details of the guiding of the visually impaired user through the walkable tunnel 922 are exemplified below in relation to
In this example, the three-dimensional walkable tunnel T is selected for receiving the associated navigation guiding instructions.
The visually impaired user receives the start command by the first haptic cue—which is temporal, and the visually impaired user begins navigating.
The Feedback Manager sub-unit 35 will attempt to keep the visually impaired user on the preferred navigation path SP and within the limits of the walkable tunnel 922 by giving directional haptic cues.
If the visually impaired user, when navigating, is too close to the left side of the walkable tunnel 922, the second haptic cue—which is spatiotemporal, is received by the left feedback actuators 411. The linear resonant actuators of the left feedback actuators 411 output vibrations in rapid succession, one linear resonant actuator vibrating after another, in the direction in which the visually impaired user must reorient, that is to the right, giving the visually impaired user the tactile sensation of having the forehead dragged by someone to the right. The variation of the duration, periodicity, intensity or frequency of the vibration of the second haptic cue is proportional to the degree of closeness to the left side of the the walkable tunnel 922.
If the visually impaired user, when navigating, is too close to the right side of the walkable tunnel 922, the third haptic cue is received by the right feedback actuators 412—which is spatiotemporal, having identical configuration with the one of the second haptic cue except that it indicates as direction of reorientation the left instead of the right. The variation of the duration, periodicity, intensity or frequency of the vibration of the third haptic cue is proportional to the degree of closeness to the right side of the the walkable tunnel 922. Guiding the user forwards is by the fourth haptic cue, —which is spatiotemporal. The fourth haptic cue is received by the centre feedback actuators 413. The variation of the duration, periodicity, intensity or frequency of the vibration of the fourth haptic cue is proportional to the speed that the visually impaired user should have when navigating.
If the visually impaired user, when navigating, must reorient his direction of movement, on the horizontal plane, for example turn right when he arrives to the pedestrian crossroad 943 shown in
If the visually impaired user, when navigating, must reorient his direction of movement on the vertical plane, for example when the visually impaired user has already crossed the pedestrian road 943 and is approaching the stairs of the building and has to climb some stairs, the sixth haptic cue is received—which is spatiotemporal, by the centre feedback actuators 413. The variation of the duration, periodicity, intensity or frequency of the vibration of the sixth haptic cue is proportional to the amount of movement required to the visually impaired user.
When the visually impaired user, arrives to the pedestrian crossroad 832 shown in
The eighth haptic pattern cue—which is temporal, signals the end of the navigation as the point of interest PI is reached, being received from the centre feedback actuators 413.
Further types of haptic pattern cues can be defined to accommodate navigation situations or requirements of the user. For example, if the visually impaired user, when navigating, is centered within the walkable tunnel 922 of the secondary navigation path 912, the right feedback haptic actuators 412 and the left feedback haptic actuators 411 can either not present any type of haptic pattern cues, or present a ninth type of haptic pattern cue on both sides of the forehead, to signal the visually impaired user that he is navigating centered within the walkable tunnel 922.
Taking the example from the description when the visually impaired user enters a new room that has four windows 85, the first 85-1, the second window 85-2, the third window 85-3, and the fourth window 85-4, and he wants to open one of the four windows 85, the potential point of interest PPI is the group of the four windows as at least one object On selected from the plurality of objects On.
The term “potential” signifies that any of the windows 85 of the room may be selected as initial point of interest PI.
With reference to
The person skilled in the art shall understand that the examples described apply to any kind of Objects On, and mutatis mutandis to the categories of living beings Ln.
In sub-step S.3.-0.2 the Sound representation sub-unit 36 represents each of the four windows 85, as corresponding spatialized sounds: the first spatialized sound S86-1, the second spatialized sound S86-2, the third spatialized sound S86-3, and the spatialized sound fourth S86-4 and transmits the four spatialized sounds to the Feedback Unit 4 via Feedback Manager sub-unit 35, when the Sound representation sub-unit 36 is not part of said Feedback Manager sub-unit 35.
In sub-step S.3-0.3 the visually impaired user selects as initial point of interest PI one from the four windows 85-1, 85-2, 85-3, 85-4, and transmits the corresponding selection request just like any other selection request.
Representation in sounds of the sub-step S.3-0.2 is exemplified below with reference to the
In example 2-1 with reference to
In order to illustrate additional features of the window 85 representing said potential point of interest PPI, such as its dimensions, or density of the material, the Navigation Manager 33 will further encode the spatialized sound S86 into a spatialized sound having a particular frequency S86f, a particular time period S86t and/or a particular pulse S86p, not represented graphically, said frequency S86f, particular time period S86t and/or pulse S86p corresponding to said additional features.
In examples 2-2 and 2-3 with reference to
In example 2-4, with reference to
The Sound representation sub-unit 36 encodes the specific information of the selected windows 85-1, and 85-2 from the Live map 310 into the spatialized sounds S86f-1, and S86f-2 having different frequency features depending on the distance of the windows 85-1, and 85-2 to the visually impaired user, and sends the encoded spatialized sounds S86f-1, and S86f-2 to the visually impaired user.
Thus, for example, the corresponding audio cues of the spatialized sounds S86f-1, and S86f-2 corresponding to the additional features of the window 85 sent to the visually impaired user vary in frequencies: the cues last longer and/or the degree of repetition is higher for the window 85-2 than the one that is nearer to the visually impaired user, 85-1 respectively.
In example 2-5, with reference to
The Sound representation sub-unit 36 extracts the specific information of the selected window 85 from the Live map 310, encodes it into spatialized sounds S86P-E1, and S86P-E2 corresponding to the window extremities 85-1E and 85-E2, the spatialized sounds S86P-E1, and S86P-E2 having different encoding characteristics depending on the distance of each of the two chosen extremities relative to the visually impaired user. The distance can be measured either on the azimuth, on the elevation or on the range of the window 85, or in any combination of the aforementioned.
In example 2-6, with reference to
The Sound representation sub-unit 36 encodes the specific information of the dimensions of the selected window 85 extracted from the Live Map 310 into temporal spatialized sound S86P representing punctiform sounds along one of the three spatial dimensions between chosen extremities of the window 85 or a linear sound S86L moving on a straight-line path from the extremity 85-E1 to the extremity 85-E2, and sends them to the visually impaired user by means of auditory Feedback actuators 42. The same operation is carried out for the others extremities of the window 85, specifically 85-E3, and 85-E4 in case the window 85 is rectangular (not represented graphically).
The dimensions of the window 85 are measured between the extremities 85-E1, and 85-E2, 85-E3, and 85-E4 of the window 85 along the corresponding spatial dimensions by means known from the prior art.
In examples from 2-7 to 2-10 with reference to
The Sound representation sub-unit 36 extracts the specific information from the Live map 310, encodes it into temporal spatialized sounds S86 representing the shape of the window 85.
In example 2-7, with reference to
In example 2-8, with reference to
In example 2-9, with reference to
In example 2-10, with reference to
In examples 2.11, and 2.12 with reference to
In example 2.11 with reference to
The two windows 85-1 and 85-2 are separated by open space of various dimensions (e.g.: 5-10 cm in case of windows or 1-2 meters in case of the doors). The visually impaired user is placed closer to the window 851.
The visually impaired user sends the request for the Sound representation of the open space distance between the two windows 85-1 and 85-2 as well as for the shape of the interior frame of the two windows 85-1 and 85-2.
The Sound representation sub-unit 36 extracts the specific information from the Live map 310 of the two windows 85-1 and 85-2, encodes it into spatialized sounds S86t-1, and S86t-2 having different time characteristics for representing the shape of the interior frames of the two windows 85-1 and 85-2.
The window 85-1 placed closer to the visually impaired user, as it is shown in the
Because of the open space between the two windows 85-1 and 85-2, the window 85-1 placed closer to the visually impaired user acts like a barrier for detecting detailed information regarding the second window 85-2, consequently the The Sound representation sub-unit 36 is able only to output a simplified spatialized sound S86t-2 corresponding to the three-dimensional position of the window 85-2 and its vertical dimension.
In example 2-12, with reference to
For simplicity,
The Sound representation sub-unit 36 extracts the specific information from the Live map 310, encodes it into two spatialized sounds S861, and S862 starting at the same time, virtually moving in an angled pattern within the space between the contour of the interior frame, and the exterior contour of the window 85.
While the description of the method and the system was disclosed in detail in connection to preferred embodiments, those skilled in the art will appreciate that modifications may be made to adapt a particular situation without departing from the essential scope to the teaching of the invention.
Reference numerals appearing in the claims are by way of illustration only and shall have no limiting effect on the scope of the claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 21154462 | Jan 2021 | EP | regional |
This application is a continuation of U.S. application Ser. No. 17/574,189, filed Jan. 12, 2022, which claims prior to European Pat. App. No. 21154462.2, filed Jan. 29, 2021, both of which is incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3594823 | Bach-Y-Ritta et al. | Jul 1971 | A |
| 5973618 | Ellis | Oct 1999 | A |
| 6356210 | Ellis | Mar 2002 | B1 |
| 8271449 | Jung | Sep 2012 | B2 |
| 9429446 | Searle | Aug 2016 | B1 |
| 9629774 | Dayal et al. | Apr 2017 | B2 |
| 9770382 | Ellis | Sep 2017 | B1 |
| 9915545 | Chen et al. | Mar 2018 | B2 |
| 11112261 | Kimia | Sep 2021 | B2 |
| 11371859 | Amariei | Jun 2022 | B1 |
| 20030057973 | Nojima | Mar 2003 | A1 |
| 20040083035 | Ellis | Apr 2004 | A1 |
| 20070005245 | Ellis | Jan 2007 | A1 |
| 20080040951 | Kates | Feb 2008 | A1 |
| 20080266106 | Lim | Oct 2008 | A1 |
| 20090005973 | Salo | Jan 2009 | A1 |
| 20090032590 | Hopkins | Feb 2009 | A1 |
| 20090215536 | Yee | Aug 2009 | A1 |
| 20100004860 | Chernoguz | Jan 2010 | A1 |
| 20100007474 | Behm | Jan 2010 | A1 |
| 20100176952 | Bajcsy et al. | Jul 2010 | A1 |
| 20110148652 | Kim | Jun 2011 | A1 |
| 20110172907 | Rui Da Silva Freitas | Jul 2011 | A1 |
| 20140100771 | Ekpar | Apr 2014 | A1 |
| 20140315570 | Yun | Oct 2014 | A1 |
| 20140331131 | DeSellem | Nov 2014 | A1 |
| 20160021435 | Topchy | Jan 2016 | A1 |
| 20180283891 | Andrew | Oct 2018 | A1 |
| 20190113774 | Anderson | Apr 2019 | A1 |
| 20200043368 | Brathwaite | Feb 2020 | A1 |
| 20200326199 | Hendrix | Oct 2020 | A1 |
| 20200341563 | Poore | Oct 2020 | A1 |
| 20220026990 | Mani | Jan 2022 | A1 |
| Entry |
|---|
| Zhang et al., “Wearable Indoor Navigation System with Context Based Decision making for Visually Impaired”, International Journal of Advanced Robotics and Automation, 2016, 1:1-11. |
| Number | Date | Country | |
|---|---|---|---|
| 20220282985 A1 | Sep 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17574189 | Jan 2022 | US |
| Child | 17825487 | US |
