The field of the present disclosure relates generally to an augmented reality system, more particularly, to an augmented reality system integrated with one or more multivariable sensors.
A number of technological advances have been realized regarding environmental sensors. In separate technical areas from environmental sensors, many advances and improvements have been made in the areas of augmented reality (AR). Some of the advances in AR provide mechanisms for a user to partake enhanced, immersive, and interactive experiences with their environment. A number of such experiences focus on entertainment aspects and outcomes.
In some respects, a need exists for methods and systems that provide an integration of AR systems and devices and environmental sensors such that an efficient, reliable, and responsive platform for determining and mitigating environmental risks and other meaningful applications may be realized.
Embodying systems and methods herein relate to an Augmented Reality (AR) system that visualizes sensor data in conjunction with actionable information associated with the sensor data simultaneously with a presentation of the locations of the one or more sensors generating the sensor data. In some embodiments, the AR system might display an indication of potential risks and/or other information related to the sensor data. In some embodiments, the AR device may display actions and/or corrective procedures in response to the sensor data of the one or more sensors.
In some aspects, the present disclosure relates to an AR system integrated with one or more sensors. The one or more sensors may each include a multivariable sensor, capable of detecting more than one variable attribute in its environment. In some instances, a multivariable sensor herein might be configured to detect an environmental hazard or condition. The environmental hazard or condition may be a high temperature/fire condition, a gas presence, a high voltage/live wire condition, and other hazards and conditions. In some instances, a multivariable sensor herein might be configured to detect a state of an industrial asset. A nonlimiting example of the state of an industrial asset is the health of lubricating oil in an engine or in a gearbox, and an isolating oil in a transformer. A nonlimiting example of an engine is an internal combustion engine of a car, truck, marine vessel, locomotive. A nonlimiting example of a gearbox is a wind turbine gearbox.
In some embodiments, a multivariable sensor herein might be a gas-selective multidimensional detector. However, a multivariable sensor herein might detect other “events” other than the presence (or lack thereof) of one or more gasses in an environment occupied by the sensor. In some instances, the multivariable sensor might detect and make distinctions between different temperatures, different electrical conditions, different pressures, different lighting conditions, the presence (or lack thereof) of different entities (e.g., based on acquired images), etc.
Some examples of the present disclosure will be discussed in the context of a system and process including AR device(s) integrated with one or more multivariable sensors embodying a gas-selective multidimensional detector to quantify multiple individual gases and their mixtures, wherein the multivariable sensors might accurately detect gases in the presence of numerous interferences to the detection of gases. In some aspects, each of the gas-selective multidimensional detectors include detector-design criteria including, in some instances, a sensing material with diverse responses to different gases, a multivariable transducer with excitation conditions to provide independent output responses and to recognize these different gas responses, and data analytics to provide multi-analyte quantitation and rejection of interferences. Design criteria of some of these sensors may include a sensing material with multi-response mechanisms to different gases and a multivariable transducer with independent outputs to represent these different gas responses. Sensing materials can be inorganic, organic, polymeric, biological, composite, and formulated sensing materials that can be applicable for multivariable gas sensing because they have different response mechanism to different gases. These sensing materials, when coupled with electrical, optical, and electromechanical transducers designed for operation in a multivariable mode, provide performance capabilities previously unavailable from conventional sensor systems. In some aspects, a sensor herein might quantify individual components in gas mixtures, reject interferences thereto, and offer a more stable response over conventional single-output sensors and arrays thereof.
In some aspects, a sensor herein may require little to no consumables, have relatively low power requirements, be procured for a low cost, and embody unobtrusive form factors suitable for, for example, Industrial Internet and other applications.
In some regards, a performance of sensor(s) herein may be beneficial and improved when compared to classic mature analytical instruments such as gas chromatography, ion mobility, vibrational spectroscopy, and mass spectrometry instruments, particularly in Industrial Internet and some other applications.
Some improvements may include the ability to detect multiple gases similar to the classic mature analytical instruments but with the advantage of smaller power demands and smaller size. These advantages can allow deployments of the multivariable sensors herein in scenarios where deployment of classic mature analytical instruments is prohibitive because of their size, cost, power demands, and frequent maintenance needs.
A multivariable sensor can herein might be designed for detection of liquids and their properties. Such a sensor can include a liquid-selective multidimensional detector that can have a multivariable transducer that provides independent outputs about electromagnetic properties of the liquid and data analytics to provide multi-analyte quantification of the different components of interest in the liquid and rejection of response to interferences in the liquid. Details of such a multivariable sensor for detection of liquids and their properties are described in US Patent Application 20170138876, incorporated herein by reference.
A multivariable sensor may refer to a single sensor capable of producing multiple responses, such as response signals, that are not substantially correlated with each other and where these individual response signals from the multivariable sensor are further analyzed using multivariate analysis tools to construct response patterns of sensor exposure to different analytes at different concentrations. The multiple responses from the sensor may differ by sensor operating frequency, sensor operating wavelength, sensor operating temperature, sensor operating voltage, sensor operating power, sensor operating polarization, or the like.
In one or more embodiments, multivariable and/or multivariate signal transduction is performed on the multiple response signals using multivariate analysis tools to construct a multivariable sensor response. A multivariate analysis can refer to a mathematical procedure that may be used to analyze two or more variables from the sensor response and to provide information at the type of at least one gas from the measured sensor parameters and/or quantitative information about the concentration of at least one gas from the measured sensor parameters. For example, the multivariate analysis tools may include canonical correlation analysis, regression analysis, nonlinear regression analysis, principal components analysis, discriminate function analysis, multidimensional scaling, linear discriminate analysis, logistic regression, neural network analysis, or another tool.
Referring the
In some past systems, gas sensors were used as a part of electronic devices distributed densely or sparsely and might have been combined with a camera. Each gas sensor was non-selectively displaying the response to different gases.
Contrary to past, conventional systems, the present disclosure includes methods and systems that, in some embodiments, utilize a single gas sensor operating in its multivariable mode (i.e., a single multivariable sensor) to replace multiple conventional sensors.
In some aspects, gas-selective (or other types of) multidimensional detectors herein may provide improved selectivity of gas (or other event) detection in an unobtrusive format that may have numerous applications, including, for example, installation on mobile unmanned platforms. In some aspects, a multivariable multidimensional sensor herein may exhibit a high selectivity (e.g., to produce no/low false alarms), low cost (e.g., applicable to a disposable implementation), low power consumption (e.g., suitable for ubiquitous installations), a small form factor (e.g., conducive to unobtrusive embedded solutions), and a high sensitivity (e.g., early detection of an alarm situation).
Referring to
In some aspects, multivariable gas (and other) sensors may include built-in wireless connectivity capabilities to connect and communicate with other devices via the Internet. In some regards, a sensor herein having a functionality to communicate with other devices over the Internet may be referred to as an Internet of Things (JOT) device that may communicate with other devices (e.g., an AR device) over the “cloud”, or over a local on-premise server or communicate directly to the AR device.
In some aspects, the inputs 320, 325, and 330 may be transmitted to inference engine 310. In some embodiments, inference engine 310 might include artificial intelligence (AI) aspects, such as machine learning, artificial neural networks, statistical learning, and other techniques. In some aspects, inference engine 310 may use one or more predetermined rules, lookup tables, and other stored data references to process inputs 305 to produce outputs 315. In some instances, inference engine 310 might use a combination of AI aspects and one or more other techniques in processing inputs 305. Operationally, inference engine 310 processes inputs 305 to determine and generate outputs 315. Outputs 315 might include the four outputs shown in
In some embodiments, inference engine 310 (or the functions thereof) may be embodied in AR device 355, where AR device 355 includes sufficient processing power and memory (if needed) to execute the functions disclosed herein as being performed by inference engine 310. That is, in some embodiments, inference engine 310 may be integrated into or at least functionally be performed, at least in part, by AR device 355.
In some embodiments, alarm level 335 can be determined by inference engine 310 based on the location and/or quantity of the detected gas (or other event for other types of sensors) detected by a sensor. The alarm level might not be displayed in terms of raw data numbers indicative of how much gas is leaking and what type of gas is leaking, as detected by a sensor. Rather, in some instances inference engine 310 may use one or more factors, such as, for example, a location of the sensor, an output from neighboring sensors, information about the environment where the gas leak is occurring (e.g., open vs closed, presence of people, the type of gas sensor, etc.) and where the AR device is located (and hence, where a user of the AR device is located) to determine the alarm level output 335 for display on AR device 355. In some instances, an alarm level herein might be expressed or represented by a color code (e.g., red, yellow, and green), text, coded alpha-numerics, a graphical scale such as stars, etc. For example, a detection of a toxic gas in an open field versus detection in a closed environment would lead to different alarm levels and different actionable responses to be deployed on the AR devices. In another example, a detection of an inflammable gas near a thermal/fire source might lead to a higher alarm level as opposed to detection of an inflammable gas without any fire source nearby.
In some aspects, inference engine 310 may produce an indication of an alarm's location 340 (i.e., the location of the sensor that produced the alarm). In some embodiments, an indication of a location for an alarm might include 3D contextual directions to navigate from a current position of the AR device 355 (as indicated by input 330) to the “Alarm Location” (e.g., input 325) where the gas leak (or other event) is detected as occurring. In some embodiments, directions to the location of an alarm might be configured in terms of directional arrows pointing to the specific place or a direction for the AR device 355 user to go to or towards. In some instances, directions to an alarm's location might be configured as a text-based direction, such as, for example, “go to second floor, bay 1”, “go to room 1B East”, etc.
In some aspects, AR device 355 displays actions 345 produced by inference engine 310, where an action refers to a concrete, actionable solution and/or directive in response to an alarm level 335. Action 345, as well as other outputs 335, 340, and 350 may be provided to AR device 355 for consumption by a user (not shown in
Regarding the solution location or direction to solution output 345, this particular output provides a location of a solution in response to a given alarm level 335. The solution location/direction to the solution might include directions to the solution to address the detected alarm level. In some aspects, the solution location and/or direction to the solution might provide a location of a solution to the detected alarm level and/or directions to the area, locale, switch, valve, control, sensor, etc. to effectuate or invoke a determined solution to an alarm situation in response to the detected alarm level 335. The solution location/direction to the solution 350 may be sent to AR device 355, where the AR device may display the determined solution location and/or directions to the solution(s) 350 on a display of the AR device (e.g., display screen). Solutions in one context herein may refer to any entity or action that may tend to mitigate or resolve a situation indicated by an alarm level 335. Examples of a solution, in some instances herein might include, a user exiting an area, closing a valve, sounding an alarm horn, etc.
In some embodiments, inference engine 310 can be implemented by a remotely located central or distributed server system or integrated with the AR device 355. In some embodiments including a remotely located inference engine, the inputs from a sensor (e.g., raw sensor data 320 and a location of the sensor 325) and the AR device (e.g., a location of the AR device 330) may be communicated wirelessly to a central (or distributed) computing and database server. In this scenario, inference engine 310 calculates the outputs (e.g., 335, 340, 345, and 350) and sends the determined outputs to the AR device 355 where the determined outputs may be displayed in a current, contemporaneous field of vision (FOV) of the user with respect to the AR device. In an embodiment including an inference engine incorporated into an AR device, the inputs from a sensor (e.g., raw sensor data 320 and a location of the sensor 325) may be communicated wirelessly to the AR device 355, where the AR device processes the sensor data and sensor location information with its known location. In this scenario, inference engine 310, as implemented in the AR device, calculates and displays the outputs (e.g., 335, 340, 345, and 350) in a current, contemporaneous FOV of the user with respect to the AR device.
The coordinate system for the AR device and the one or more sensors may be aligned and locked in association with each other using an image marker. In one embodiment, an image marker may be fixed to a known and invariant location in the environment of the sensor(s) and have an asymmetric picture. The image marker may be scanned by or otherwise registered/associated with the AR device, wherein a center location of the image marker may represent the origin of the AR device's coordinate system and wherein the orientation of the image marker determines the orientation of the AR coordinate system. In this manner, the world coordinate system may be aligned with the coordinate system of the AR device (i.e., the AR coordinate system).
In some aspects, the sensors producing the raw data 320 and associated with sensor location 325 may be either fixed to the infrastructure of the sensor's environment or mobile. If the sensors are fixed, their locations may be stored in a database (or other data storage device/system) and their relative locations may be sent to AR device 355 as the user moves AR device 355 in the environment. In the instance the sensors are mobile, then accurate and long-range tracking modalities such as, for example, Ultrawideband (UWB) may be used to locate and track the sensors. As the sensors move in the environment, their location information may be updated via a server onto the AR device and/or via processor(s) onboard the AR device. The AR devices can be located in the air, above ground, underground, above water, underwater, in space, in an industrial setting, a residential setting, environmental setting, military setting, indoor setting, outdoor setting, and other settings.
At operation 410, an indication of a location of the multivariable sensor generating the information of operation 405 is received. The location information for a sensor may be determined by one or a combination of techniques, including but not limited to tags or other devices adjacent to, in, or on the sensor where the tag might determine its exact 3D location relative to a device (e.g., image marker) at a known, fixed location defined in a particular coordinate system. The location information for the sensor generating the raw data of operation 405 may be received via a wireless transmission from the respective sensor.
At operation 415, an indication of a location of an AR device associated with or otherwise registered to produce visualization(s) related to the sensor(s) generating the raw data of operation 405 may be received. The location information might include 3D coordinate information with respect to the same coordinate system defining the location of the sensor generating the raw data received of operation 405. In this manner, the location of the sensor and the location of the AR are the same (i.e., aligned with each other).
Proceeding to operation 420, a determination of an alarm is executed. The alarm may be determined based on the sensor information received from the at least one multivariable sensor at operation 405. The particular procedures, routines, and processes to determine the “alarm” may depend on a configuration of the sensor, a desired representation of an alarm indication, a desired or required input for an AR device integrated with the multivariable sensor(s) involved in the execution of process 400, other considerations, and combinations thereof.
At operation 425, a determination of the location of the alarm determined at operation 420 and a location of a solution associated with the determined alarm are made. The determination of the alarm's location may be based on the location of the multivariable sensor associated with or generating the information forming a basis for the alarm.
Continuing to operation 430, a visualization of the determined alarm and at least one of the determined location for the alarm and the determined location for the solution associated with the alarm is presented in a display of the AR device coincident or contemporaneously with a display of a current field of view (FOV) of the AR device. In some embodiments, both the determined location for the alarm and the determined location for the solution associated with the alarm may be presented in the display of the AR device coincident with or contemporaneously with a display of a current FOV of the AR device. For example, at least one of the determined location for the alarm and the determined location for the solution associated with the alarm may be presented as a graphical overlay on a current FOV displayed on the AR device. In some embodiments, an inference engine may receive the stated inputs received at operations 405, 410, and 415 and further perform operations 420 and 425 to produce data and information to have the visualizations of operation 430 presented by an AR device. In some instances, an AR device herein might include or integrate at least some of the function of an inference engine disclosed herein.
As another example,
System 1400 includes processor(s) 1410 operatively coupled to communication device 1420, data storage device 1430, one or more input devices 1440, one or more output devices 1450, and memory 1460. Communication device 1420 may facilitate communication with external devices, such as a data server and other data sources. Input device(s) 1440 may comprise, for example, a keyboard, a keypad, a mouse or other pointing device, a microphone, knob or a switch, an infra-red (IR) port, a docking station, and/or a touch screen. Input device(s) 1440 may be used, for example, to enter information into system 1400. Output device(s) 1450 may comprise, for example, a display (e.g., a display screen) a speaker, and/or a printer.
Data storage device 1430 may comprise any appropriate persistent storage device, including combinations of magnetic storage devices (e.g., magnetic tape, hard disk drives and flash memory), optical storage devices, Read Only Memory (ROM) devices, etc., while memory 1460 may comprise Random Access Memory (RAM), Storage Class Memory (SCM) or any other fast-access memory.
Inference engine 1432 may comprise program code executed by processor(s) 1410 (and within the execution engine) to cause system 1400 to perform any one or more of the processes described herein. Embodiments are not limited to execution by a single apparatus. Data storage device 1430 may also store data and other program code 1436 for providing additional functionality and/or which are necessary for operation of system 1400, such as device drivers, operating system files, etc.
An example of operation of a multivariable sensor that can be used in accordance with the present disclosure is depicted in
The several outputs of the single multivariable sensor included the magnitude (Zp) of a real part of the resonant impedance spectral response, a resonant frequency (F1), an antiresonant frequency (F2) of an imaginary part of the resonant impedance spectral response, and a zero-reactance frequency (Fz) at the imaginary part of the resonant impedance spectral response.
The sensor was exposed to two analyte vapors of interest such as dimethyl methylphosphonate (DMMP) and 1,5-dichloropentane (DCP). The DMMP is known as a nerve agent simulant and DCP is known as a blister agent simulant. These vapors were generated at three concentrations each. These concentrations were 0.04, 0.08, and 0.12 of the partial vapor pressure (P/Po) of these vapors, where P was the generated vapor pressure and Po was the saturated vapor pressure of these analyte vapors. These concentrations were further denoted in relative units as 1, 2, and 3 relative units of concentrations. During the exposures of the multivariable sensor to these vapors, water vapor was further mixed with these vapors to generate binary mixtures of these vapors with water vapor. The water vapor was at two concentrations such as 0.2 and 0.4 of its partial vapor pressure P/Po. These concentrations of water vapor were further denoted in relative units as 1 and 2 relative units of water concentrations. In addition, the sensor was exposed to DMMP and DCP vapors without mixing with water vapor.
Overall, this diversity in outputs of a single sensor permitted the discrimination and quantitation of DMMP and DCP vapors even in the presence of different concentrations of water vapor. In one embodiment, a processor analyzed outputs from the single sensor using a machine learning algorithm such as Support Vector Machine (SVM) algorithm.
Thus, this resulting sensor that had the sensing material and the transducer designed for operation in a multivariable mode, provided performance capabilities previously unavailable from conventional sensors. In particular, this individual sensor quantified two individual components such as DMMP and DCP in gas mixtures with water vapor such as an interference at different concentrations of water vapor.
In accordance with some embodiments, a computer program application stored in non-volatile memory or computer-readable medium (e.g., register memory, processor cache, RAM, ROM, hard drive, flash memory, CD ROM, magnetic media, etc.) may include code or executable instructions that when executed may instruct and/or cause a controller or processor to perform methods disclosed herein, such as a method of processing multivariable sensor data, sensor location, and AR device location to generate AR device visualization related to same (i.e., process 400,
The computer-readable medium may be a non-transitory computer-readable media including all forms and types of memory and all computer-readable media except for a transitory, propagating signal. In one implementation, the non-volatile memory or computer-readable medium may be external memory.
Although specific hardware and methods have been described herein, note that any number of other configurations may be provided in accordance with embodiments of the invention. Thus, while there have been shown, described, and pointed out fundamental novel features of the invention, it will be understood that various omissions, substitutions, and changes in the form and details of the illustrated embodiments, and in their operation, may be made by those skilled in the art without departing from the spirit and scope of the invention. Substitutions of elements from one embodiment to another are also fully intended and contemplated. The invention is defined solely with regard to the claims appended hereto, and equivalents of the recitations therein.