The field of the present disclosure relates generally to an augmented reality system, more particularly, to an augmented reality system integrated with a system that tracks assets.
A number of technological advances have been realized regarding the tracking of assets. In separate technical areas from the tracking of assets, many advances and improvements have been made in the areas of augmented reality (AR). Some of the advances in AR provide mechanisms for a user to partake in enhanced, immersive, and interactive experiences with their environment. A number of such experiences focus on entertainment aspects and outcomes.
In some respects, a need exists for methods and systems that provide an integration of AR systems and devices and asset tracking systems such that an efficient, reliable, and responsive platform including the benefits of both systems may be realized.
Embodying systems and methods herein relate to an Augmented Reality (AR) system that visualizes asset-related location data in combination with contextual directions relative to the asset and an AR device within a current field of view (FOV) of the AR device. In some embodiments, the AR device might display an indication of the contextual directions to direct a user to the location of the asset, particularly when the asset is not located in the current FOV of the AR device.
In some aspects, the present disclosure relates to an AR system integrated with an indoor positioning system (IPS) that supports and provides mechanisms to dynamically track assets within an environment covered by the IPS and to further visualize the assets in an AR device registered or otherwise associated with the AR system. In some aspects, the methods and systems of the present disclosure provide a visualization of the current, actual relative location of an asset tracked within an IPS in a current FOV of an AR device. In some instances and use-cases, a user of the AR device presented with a display of this combination of visualized information, may be made contextually aware of their current location relative to the tracked assets within their FOV and, in an instance the tracked assets are not within their FOV, be presented with contextual 3D directions from their current location to the tracked assets.
In some embodiments, a system disclosed herein may comprise an AR system and an IPS. Furthermore, the system may include an AR device including a display device/system that is registered to the AR system, an IPS based map of an environment of interest (e.g., a warehouse, factory, office building, or other area/facility covered by the IPS), a tracking system to locate assets within the IPS mapped environment, a system to track the AR device within the IPS mapped environment, a data management system (e.g., a server, a database, and combinations thereof) to store asset-related location information, and a communication network to facilitate and support communication between the IPS, the AR device, and the data management system.
Communication between the plurality of location beacons 110, 115, 120, and 125 and tag 105 may be via a wireless communication protocol. In some embodiments, the communication technology protocol used by location beacons 110, 115, 120, and 125 and tag 105 may be an ultra-wide band (UWB) technology. In some instances, UWB herein refers to technologies that transmit information spread over a large bandwidth (e.g., >500 MHz). Specific characteristics of the UWB communication signals used by the devices disclosed herein may vary.
The location beacons and tag(s) of system 100 may form at least a portion of the IPS of system 100. The plurality of location beacons 110, 115, 120, and 125 (may include fewer or more location beacons but, in some systems, at least n, where n>3) and tag(s) 105 may correspond to and encompass or cover an environment of interest, where each such environment can be fully mapped using the multiple location beacons and tags.
In some embodiments, each asset (e.g., 130) has an associated tag (e.g., 105) and the tag's three-dimensional location (x, y, z) and orientation (α, β, γ) are calculated using an UWB based radio communication system. In some aspects, UWB technology communication protocols are used herein to achieve centimeter level tracking accuracy that can be useful in tracking and locating assets in a multitude of different scenarios and applications, including but not limited to hospitals, warehouses, factories, powerplants, etc.
A location of a tag 105 in system 100 may be calculated based a determination of a distance between a particular tag and one or more of the location beacons in system 100. The determined location of tag 105 may be transmitted via a UWB signal to server 145. Data management system or server 145 may be implemented as a central server and/or database, a distributed server and/or database, and a cloud-based service.
An image marker 135 is physically included in system 100 at an origin of a coordinate system 140. Image marker 135 may be a visual image configured according to a design specification to include information, where the information may be coded per the design specification to convey, for example, an orientation of the image marker, a location of the image marker, a reference to other information, and other data. Image marker 135 may be a barcode, a Quick Response (QR) code, and other machine-readable two-dimension labelling schemes. Image marker 135 has its own coordinate system, as illustrated at 140. in some embodiments, coordinate system 140 includes a positive x-axis along a longer axis of the image marker, a positive y-axis along a shorter axis of the image marker, and a z-axis that is orthonormal to the plane of the image marker. In some aspects, the coordinate system of the IPS comprising the location beacons 110, 115, 120, and 125 and tag(s) 105 is aligned with the coordinate system 140 of the image marker by virtue of the image marker being physically located or attached to the origin of the IPS coordinate system.
In some aspects, image marker 135 may be optically scanned by AR device 150. In some embodiments, AR device 150 includes an image capture device/system (e.g., a camera) to optically scan image marker 135. The AR system's coordinate system or universe may be aligned with the marker coordinate system 140 by scanning image marker 135 by AR device 150.
In some respects, since the image marker coordinate system 140 is aligned with the IPS coordinate system by virtue of the image marker being positioned at the origin of the IPS' known coordinate system, the AR system's coordinate system is also aligned with the IPS' coordinate system. That is, the AR system's and the IPS' coordinate systems may be aligned with each other when configured as disclosed herein. With both the AR system and the IPS being referenced to the same coordinate system, the locations of the tag(s) 105 calculated by the UWB IPS of system 100 may be aligned and coordinated with the AR system of system 100 that includes AR device 150. Accordingly, the locations of tag(s) 105 and the asset(s) 130 associated with and affixed adjacent to the tag(s) herein may be accurately represented and included in a field of view of AR device 150. In this manner, a consolidated FOV might be displayed by AR device 150 that includes the AR device's actual, current FOV and the assets (e.g., 130) tracked in the environment of system 100 as they come into view within the current FOV.
In some aspects, server 145 may maintain and manage a searchable database including, for example, of all the tags of system 100, as well as their associated assets and the locations of those assets where the location for each location may be continually updated. In some aspects, the database of server 145 may be queried (e.g., via a voice or text search submitted by a user of AR device 150) for any asset's location maintained and updated in the database.
In some aspects, a tag 105 associated with an asset 130 herein may operate to communicate with the location beacons 110, 115, 120, and 125 in its environment (e.g., hospital, warehouse, factory, school, etc.), calculate its own position based on its relative distances from the plurality of location beacons, and further communicate its position to a server (e.g., 145) which can relay such information to the AR device 150 when the server is queried by the AR device.
In some aspects, after an asset's location is determined, the AR device's display presents visual cues overlaid on the display of an actual object(s) within the FOV of the user. In the instance the object is outside the FOV of the AR device/user, then the system may provide visual cues and/or directions from the AR device/user' current location to the location of the asset on the AR device (e.g., a visor, headset, helmet, glasses, etc.) guide the user to the asset.
In some embodiments, the determined location and orientation of the assets may be live streamed to one or more (i.e., multiple) AR device via a server that delivers the location and orientation information within the FOV of the user. In some respects, system 100 provides a mechanism and platform for a user of AR device 150 to query the current location of an asset using a combination of voice, text and gesture commands, as supported by AR device 150 and the search capabilities of server 145. In some aspects, system 100 including integrated AR system aspects and asset system tracking features provides the technical benefits of, for example, quickly and accurately locating and tracking high cost and/or critical item assets. Such benefits provided by the technical systems and methods disclosed herein might contribute to fewer lost items, reductions in the cycle time to locate items, increased productivity and efficiency, and other advantages.
Further regarding system 200, system 200 includes a plurality of beacon locations 210, 215, 220, and 225 that function similar to location beacons 110, 115, 120, and 125; a location tag 205 that operates similar to tag 105; an asset 230 similar to asset 130; an image marker 235 similar to image marker 135; and an image marker coordinate system 240 similar image marker coordinate system 140. As such, system 200 operates and provides many of the same or similar functions and benefits as system 100, except AR device 245 also performs the functions previously reserved for server 145 in system 100.
In some aspects, system 100 provides some technical benefit(s) by having AR device 245 onboard the functionalities of server 145, including, for example, a more compact system (fewer components) and not having to have a separate (e.g., central) storage facility to store asset-related location data. In this scenario, queries submitted by a user may be processed by AR device 245. In some embodiments, system 100 may be used in an instance where multiple AR devices are deployed. However, system 200 might also be implemented in some instances with multiple AR device, wherein the AR devices might directly communicate with each other.
Referring to
Input 325 includes (UWB) information indicative of a distance of a tag (e.g., 105, 205) relative to the one or more location beacons referenced in input 320. The relative distance of each tag from multiple different location beacons can be used to accurately determine the location of the tag, and by association, an asset located coincident with a particular tag. In some embodiments, input 325 may be continually (or at least periodically) updated so as to provide an accurate basis for the calculation of a current or real-time location for an asset associated with a tag herein.
Input 330 includes information indicative of a location of an image marker. Since the image marker is physically and statically located at the origin of an IPS as disclosed herein, input 330 may only need to be received once, so long as the position of the image marker is unchanged. In some embodiments, input 330 may be received periodically.
Input 335 includes information indicative of a location of an AR device in a system herein. The location of the AR device may be referenced to the image marker of input 330, as discussed hereinabove. In some embodiments, input 335 may be continually (or at least periodically) updated so as to provide a current or real-time location for an AR device herein.
In some aspects, the inputs 320, 325, 330, and 335 may be transmitted to inference engine 310. In some embodiments, inference engine 310 might include artificial intelligence (AI) aspects, such as machine learning, artificial neural networks, statistical learning, and other techniques. In some aspects, inference engine 310 may use one or more predetermined rules, lookup tables, and other stored data references to process inputs 305 to produce outputs 315. In some instances, inference engine 310 might use a combination of AI aspects and one or more other techniques in processing inputs 305. Operationally, inference engine 310 processes inputs 305 to determine and generate outputs 315. Outputs 315 might include the two outputs shown in
In some embodiments, inference engine 310 (or the functions thereof) may be embodied in AR device 350, where AR device 350 includes sufficient processing power and memory (if needed) to execute the functions disclosed herein as being performed by inference engine 310. That is, in some embodiments, inference engine 310 may be integrated into or at least functionally be performed, at least in part, by AR device 350.
In some embodiments, a combination of an asset's location and contextual directions from the AR device's current location to an asset may be included in a consolidated FOV of an AR device display. In some instances, the FOV may include both a representation of the asset's determined location and a representation of the contextual directions when the FOV includes a view of the asset's current location. In an instance the asset is not (yet) in the AR device's FOV, then the AR device's display of the FOV might include the contextual direction to the asset, without a representation of the asset since the asset in not (yet) in the FOV.
In some embodiments, inference engine 310 can be implemented by a remotely located central or distributed server system or integrated with the AR device 350. In some embodiments including a remotely located inference engine, the inputs from a location beacon and/or tag may be communicated wirelessly (e.g., UWB) to a central (or distributed) computing and database server. In this scenario, inference engine 310 calculates the outputs and sends the determined outputs to an AR device 350 where the determined outputs may be displayed in a current, contemporaneous consolidated field of vision (FOV) of the user with respect to the AR device. In an embodiment including an inference engine incorporated into an AR device, the inputs from location beacons and tags may be communicated wirelessly (e.g., UWB) to the AR device, where the AR device processes the location and other information with its known location. In this scenario, inference engine 310, as implemented in the AR device, calculates and displays the outputs in a current, contemporaneous consolidated FOV of the user with respect to the AR device.
At operation 410, an indication of a location of an AR device associated with or otherwise registered with a system to produce visualization(s) related to the tracked asset(s) as determined based on the tag locations referenced in operation 405 may be received. The location information might include 3D coordinate information with respect to the same coordinate system defining the location of the tag.
At operation 415, a location of the tracked asset is determined relative to the AR device, based on the asset location information and AR device location information received at operations 410 and 415, respectively.
Proceeding to operation 420, a determination of contextual directions from the determined location of the AR device to the indicated location of the tracked asset is performed. The contextual directions may be continually updated as a location of the AR device changes as a wearer or user of the AR device changes.
Continuing to operation 430, a consolidated current FOV display on the AR device is presented that includes, at least one of (i) a representation of the determined location of the asset and a representation of the determined contextual directions, and (ii) a representation of the determined contextual directions. In some aspects, the displayed FOV will include aspect (i) when the asset is within the AR device's FOV but the AR device (user) is not directly on or next to the asset. In some aspects, the displayed FOV will include aspect (ii) when the asset is not (yet) within the AR device's FOV.
As another example,
System 1300 includes processor(s) 1310 operatively coupled to communication device 1320, data storage device 1330, one or more input devices 1340, one or more output devices 1350, and memory 1360. Communication device 1320 may facilitate communication with external devices, such as a data server and other data sources. Input device(s) 1340 may comprise, for example, a keyboard, a keypad, a mouse or other pointing device, a microphone, knob or a switch, an infrared (IR) port, a docking station, and/or a touch screen. Input device(s) 1340 may be used, for example, to enter information into system 1300. Output device(s) 1350 may comprise, for example, a display (e.g., a display screen) a speaker, and/or a printer.
Data storage device 1330 may comprise any appropriate persistent storage device, including combinations of magnetic storage devices (e.g., magnetic tape, hard disk drives and flash memory), optical storage devices, Read Only Memory (ROM) devices, etc., while memory 1360 may comprise Random Access Memory (RAM), Storage Class Memory (SCM) or any other fast-access memory.
Inference engine 1332 may comprise program code executed by processor(s) 1310 (and within the execution engine) to cause system 1300 to perform any one or more of the processes described herein. Embodiments are not limited to execution by a single apparatus. Data storage device 1330 may also store data and other program code 1336 for providing additional functionality and/or which are necessary for operation of system 1300, such as device drivers, operating system files, etc.
In accordance with some embodiments, a computer program application stored in non-volatile memory or computer-readable medium (e.g., register memory, processor cache, RAM, ROM, hard drive, flash memory, CD ROM, magnetic media, etc.) may include code or executable instructions that when executed may instruct and/or cause a controller or processor to perform methods disclosed herein, such as a method of processing multivariable sensor data, sensor location, and AR device location to generate AR device visualization related to same (i.e., process 400,
The computer-readable medium may be a non-transitory computer-readable media including all forms and types of memory and all computer-readable media except for a transitory, propagating signal. In one implementation, the non-volatile memory or computer-readable medium may be external memory.
Although specific hardware and methods have been described herein, note that any number of other configurations may be provided in accordance with embodiments of the invention. Thus, while there have been shown, described, and pointed out fundamental novel features of the invention, it will be understood that various omissions, substitutions, and changes in the form and details of the illustrated embodiments, and in their operation, may be made by those skilled in the art without departing from the spirit and scope of the invention. Substitutions of elements from one embodiment to another are also fully intended and contemplated. The invention is defined solely with regard to the claims appended hereto, and equivalents of the recitations therein.
Number | Name | Date | Kind |
---|---|---|---|
8761811 | Alonzo | Jun 2014 | B2 |
8924548 | Frost et al. | Dec 2014 | B2 |
9395195 | Ihara et al. | Jul 2016 | B2 |
9448072 | Bandyopadhyay et al. | Sep 2016 | B2 |
9689688 | Sasao et al. | Jun 2017 | B2 |
10212553 | Ambha Madhusudhana | Feb 2019 | B1 |
20070290924 | McCoy | Dec 2007 | A1 |
20120249588 | Tison et al. | Oct 2012 | A1 |
20160140868 | Lovett | May 2016 | A1 |
20160171777 | Todeschini et al. | Jun 2016 | A1 |
20170076504 | Oar et al. | Mar 2017 | A1 |
20170168135 | Want | Jun 2017 | A1 |
20170367766 | Mahfouz | Dec 2017 | A1 |
20180131907 | Schmirler et al. | May 2018 | A1 |
20180247259 | Borders | Aug 2018 | A1 |
20180247421 | DeAngelis | Aug 2018 | A1 |
20190170845 | Stroiescu | Jun 2019 | A1 |
Number | Date | Country |
---|---|---|
107782314 | Mar 2018 | CN |
Entry |
---|
Xu, Jinsong et al., “Design and Implementation of Channel Estimation and Equalization of Indoor Positioning System Based on UWB”, 2009 Second International Conference on Intelligent Computation Technology and Automation, China , Oct. 10-11, 2009, DOI: 10.1109/ICICTA.2009.730, (pp. 57-61, 5 total pages). |
Kuhn, Michael J. et al., “Ultra wideband 3-D Tracking of Multiple Tags for Indoor Positioning in Medical Applications Requiring Millimeter Accuracy”, 2012 IEEE Topical Conference on Biomedical Wireless Technologies, Networks, and Sensing Systems (BioWireleSS), USA, Jan. 15-18, 2012, (pp. 57-60, 4 total pages). |
Pena-Rios, Anasol et al., “A Fuzzy Logic Based System for Geolocated Augmented Reality Field Service Support”, 2017 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Italy, Jul. 9-12, 2017, (pp. 1-6, 6 total pages). |
Number | Date | Country | |
---|---|---|---|
20200110934 A1 | Apr 2020 | US |