Patent Grant
7378963
This invention relates to analysis and display of signals representing location and angular orientation of a human's body.
In many environments, a central operator communicates with, and receives visual signals and/or auditory signals from, two or more mobile or non-mobile communicators who are responding to, or relaying information on, one or more events in the field through a signaling channel associated (only) with that communicator. The event(s) may be a medical emergency or hazardous substance release or may be associated with continuous monitoring of a non-emergency situation. The visual and/or auditory signals may be displayed through time sharing of the displays received by the operator. However, this approach treats all such signals substantially equally and does not permit fixing the operator's attention on a display that requires sustained attention for an unpredictable time interval. This approach also does not permit the operator to quickly (re)direct attention to, and assign temporary priority to, two or more communicators, out of the sequence set by the time sharing procedure. This approach, by itself, does not provide information on the present location, present angular orientation and present environment of the communicator.
What is needed is a signal analysis and communication system that (1) accepts communication signals from multiple signal sources simultaneously, (2) permits a signal recipient to assign priority to, or to focus on, a selected audio signal source. Preferably, the system should allow determination of location and angular orientation of a person associated with a signal source and should permit visual, audible and/or electronic monitoring of one or more parameters associated with the health or operational fitness of the person. The system should also allow easy prioritization of a selected individual's audio and visual communication, while allowing other communication channels to be monitored in the background.
These needs are met by the invention, which provides a method and system that allows auditory and visual monitoring of multiple, simultaneous communication channels at a centralized command post (“local control center”) with enhanced speech intelligibility and ease of monitoring visual channels; visual feedback as to which channel(s) has active audible communications; and orientation information for each of N monitored communicators (N≧1). Each monitored communicator wears a hard hat equipped with lighting according to O.S.H.A. regulations, headphone, throat microphone and visual image transmitter (e.g., a camera). The local control center, which may be embodied within a hardened laptop computer or equivalent device, includes software for modifying input audio signals via compression and binaural (three-dimensional audio) signal processing, combining these audio signals with visual video, location, angular orientation and situational awareness information, and presenting the audio signals from perceived locations that are spatially separated.
Each of N communicator channels is assigned an azimuthal angular sector associated with the apparent sound image perceived through the operator's headset, where N is normally between 2 and 8. Spatial audio filtering, using head-related transfer function filters, as described in “Multi-channel Spatialization System for Audio Signals” U.S. Pat. No. 5,483,623, issued to D. Begault and in D. Begault, “Three-dimensional Sound for Virtual Reality and Multimedia, Academic Press, 1994, esp. pp. 39-190 (content incorporated by reference herein), can be provided so that this signal appears to arrive from a specified location within sector number n at the operator's head, with the sector being non-overlapping so that the operator can distinguish signals “received” in angular sector n1 from signals “received” in angular sector n2 (≠n1), even where signals from two or more channels are present.
In U.S. Pat. No. 5,438,623, head related transfer functions (“HRTFs”) are measured for each of the left ear and the right ear for a given audio signal for selected azimuthal angles (e.g., ±60° and ±150°) relative to a reference line passing through an operators head, for each of a sequence of frequencies from 0 Hz to about 16,000 Hz, and a measured HRTF is formed for each ear. A synthetic HRTF is then configured, using a multi-tap, finite impulse response filter (e.g., 65 taps) and appropriate time delays, which compares as closely as possible to the measured HRTF over the frequency range of interest and which is used to “locate” the virtual source of the audio signal to be perceived by the operator. If the operator or an azimuthal angle is changed, the measured HRTF and synthetic HRTF must be changed accordingly.
Location and angular orientation of a communicator or helmet are estimated or otherwise determined using digital compass, global positioning system (GPS), general system mobile (GSM) or other location system, and are presented to the operator.
The invention creates a multi-model communications environment that increases the situational awareness for the operator (controller). Situational awareness is increased by a number of innovations such as spatially separating each voice communication channel, allowing a single voice channel to be prioritized while still allowing other channels to be monitored. This allows the controller to view real time video from each of the controlled communicators, allowing sensor data from these communicators to be electronically collected separately, rather than being collected over the voice channel. The approach also provides an interface for the operator to record and transmit event data. In addition, each communications channel is equipped with a video indicator that allows the operator to determine who is speaking and from which communication channel the signal is being received.
Examples of situations in which the invention will be uniquely useful include the following:
(1) A local control center in a search and rescue or monitoring operation often requires one operator with a portable communication device to focus attention simultaneously, both visually and audibly, on as many as four different personnel at once. The operator must be able to focus on a specific communicator without sacrificing active monitoring (e.g., in the background) of other communicators. By supplying a coordinated spatial display of visual and auditory information, greater ease of segregation of information (auditory, visual, state situation) may be conveyed.
(2) In high stress situations, such as search and rescue operations, a local controller must be provided with an optimal display of information, both visually and audibly, concerning both rescue personnel and the surrounding environment, such as a collapsed structure. A local controller must frequently act quickly on the basis of available (often incomplete) information because of the time-sensitive nature of rescue operations. An optimal display must provide as much information as the operator can accommodate, and as quickly and as unambiguously as possible, in a manner that allows selective prioritization of information, as required.
(3) Prior art for portable systems for rescue applications utilizes multiple audio communication channels mixed in and transmitted through a single channel, without video. The communication source (video and audio channels) are not prioritized to the operator. Supporting technology developed by one of the inventors (Begault., U.S. Pat. No. 5,438,623, 1995) allows spatialization of signals but does not contain a mechanism for prioritization.
A communicator helmet 21-n has an associated communicator headset 22-n and an associated communicator antenna 23-n for communicating, audibly and otherwise, with the operator. Optionally, the communicator helmet 21-n also has one or more (preferably, at least three) short- or medium range, spaced apart radio frequency identification devices (“RFIDs”) 24-n(k) (k=1, . . . , K;K≧3), positioned on the helmet and/or on the body of the communicator. Each RFID communicates (one way or two way) with three or more spaced apart locator modules 25-m (m=1, 2, 3, . . . ) that receive RFID signals from each RFID 24-n(k) and that estimate, by triangulation, the present location of the RFID, as discussed in Appendix 1. The RFID signals received from each RFID may be replaced by GPS signals or GSM signals received from three or more GPS signal receivers or GSM signal receivers, respectively, and the collection of locator modules 25-m can be replaced by a collection of GPS satellites or by a collection of GSM base stations (now shown in
Where the three dimensional location coordinates of the communicator or of the helmet are to be estimated and provided for the operator, use of a single RFID on the communicator's body or helmet may be sufficient. However, where the angular orientation of the communicator's body or helmet is also to be estimated and provided for the operator, preferably at least three spaced apart RFIDs should be provided on the communicator's body or helmet; and angular orientation can also be estimated as set forth in Appendix 1.
A “prioritization system” allows a selected channel to be brought “front and center” to an unused central angular sector in the display, allowing the operator to focus on an individual communicator while not sacrificing active monitoring of the other communicators. The spatializer output signals are received and converted to analog format by a digital-to-analogy converter (“DAC”) 36, with the converted signal being received by a headphone amplifier 37 to provide audibly perceptible signals for the operator 38.
Optionally, the visual and location/orientation (“L/O”) information received from each communicator channel can be presented in time sharing mode, where each of the N channels receives and uses a time slot or time interval of fixed or variable length Δt(n) in a larger time interval of length ΔT (>ΣnΔt(n)), where the remaining time, of length ΔT−ΣnΔt(n), is reserved for administrative signals and for special or emergency service and/or exception reporting, as required by a specified channel, using a prioritization procedure for the specified channel. Sensing of a non-normal environmental situation at a communicator's location optionally assigns this remainder time (of length ΔT−ΣnΔt(n)) to reporting and display on that channel. Preferably, the time interval lengths Δt(n) should not exceed a temporal length that would cause communication through the channels to appear non-continuous. The audio signals received from a communicator are preferably presented using the spatializer, as discussed in the preceding.
Still frame images from the still frame processor and corresponding event information from the event database 46 are received and combined in an internal display module 51 and associated processing and recording module 52. An optional external display module 53 receives and displays selected images and alphanumeric information from the internal display 51. Selected information from the processing and recording module 52 is received by a rescue sensor module 54, which checks each of a group of situation parameters against corresponding event threshold values to determine if a “rescue” or emergency situation is present. If a rescue or emergency situation is present, an audibly perceptible alarm signal and/or visually perceptible alarm signal is provided by an alarm module 55 to advise the operator (and, optionally, one or more of the communicators) concerning the situation. Optionally, the alarm signal may have two or more associated alarm modes, corresponding to two or more distinct classes of alarm events.
A first class of alarm event parameters specifies a maximum time interval Δt(max;m) during which an event (no. m) can persist and/or a minimum time interval during which an event (no. m) should persist; a range, Δt(min;m) ≦t≦ Δt(max;m), is thus specified, where Δt(min;m) may be 0 or Δt(max;m) may be ∞.
As a first example, the system may specify that, if the communicator is substantially motionless and (optionally) supine (estimated using knowledge of the communicator's angular orientation) for a time interval exceeding 30 sec, a communicator-down alarm will be issued. As a second example, if the system senses that the communicator has not drawn a breath within a preceding time interval of specified length (e.g., within the last 45 sec), a communicator-disabled alarm will be issued.
As a third example, an exposure-versus-time threshold curve can be provided for exposure (1) to a specified hazardous material (e.g., trichloroethylene or polychlorinated biphenols), (2) to specified energetic particles (e.g., alphas, betas, gammas, X-rays, ions or fission fragments) or (3) to noise or other sound at or above a specified decibel level (e.g., 90 dB and above); and a sensor carried on a communicator's body or helmet can periodically sense (e.g., at one-sec intervals) the present concentration or intensity of this substance and issue an exposure alarm signal when the time-integrated exposure exceeds the threshold value.
In addition to environmental parameters, physiological parameters, such as heart rate, breathing rate; temperature of a selected body component and/or pH of blood or of another body fluid, may be measured and compared to a permitted range for that parameter.
dL={(xS+0.5ΔxS)2+yS2}1/2, (1)
dR={(xS−0.5ΔxS)2+yS2}1/2, (2)
Δφ=(dL−dR)/λ, (3)
where λ is a representative audio wavelength of the perceived source signal and (x,y)=(±0.5ΔxS,0) are the location coordinates of the operator's right and left ears relative to an origin O within the operator's head.
Where a single channel (e.g., n=1) is prioritized, the channel icon is moved from its non-prioritized location to a “front and center” location at the center of the screen, as illustrated in
Development of Location Relations
Consider a location determination (LD) system having at least three spaced apart signal receivers 81-k (k=1, . . . , K(K≧4) in
{(x−xk)2+(y−yk)2+(z−zk)2}1/2=c·Δtk−b, (A1)
Δtk=tk−t0, (A2)
b=cτ, (A3)
where tk is the time the transmitted LD signal is received by the receiver no. k and τ is a time shift (unknown, but determinable) at the source that is to be compensated.
By squaring Eq. (A1) for index j and for index k and subtracting these two relations from each other, one obtains a sequence of K−1 independent relations
Equations (A4) may be expressed as K−1 linear independent relations in the unknown variable values x, y, z and b.
If K≧5, any four of these K−1 relations alone suffice to determine the variable values x, y, z and b. In this instance, the four relations in Eq. (A4) for determination of the location coordinates (x,y,z) and the equivalent time shift b=cτ can be set forth in matrix form as
If, as required here, any three of the receivers are noncolinear and the five receivers do not lie in a common plane, the 4×4 matrix in Eq. (A6) has a non-zero determinant and Eq. (A6) has a solution (x,y,z,b).
If K=4, the three relations in Eq. (A4) plus one additional relation can determine the unknown values. To develop this additional relation, express Eqs. (A4) in matrix form as
These last relations are inverted to express x, y and z in terms of b:
These expressions for x, y and z in terms of b in Eq. (A10) are inserted into the “square” in Eq. (A1),
{(x−x1)2+(y−y1)2+(z−z1)2}=(c·Δt1)2−2b.c·Δt1+b2, (A14)
to provide a quadratic equation for b,
A·b2−2B·b+C=0, (A15)
A={m′11Δt12+m′12Δt13+m′13Δt14}2+{m′21Δt12+m′22Δt13+m′213Δt14}2+{m′31Δt12+m′32Δt13+m′213Δt14}2, (A16-1)
B={m′11ΔD12+m′12ΔD13+m′13ΔD14−x1}{m′11Δt12+m′12Δt13+m′13Δt14}+{m′11ΔD12+m′12ΔD13+m′13ΔD14−y1}{m′11Δt12+m′12Δt13+m′13Δt14}+{m′11ΔD12+m′12ΔD13+m′13ΔD14−z1}{m′11Δt12+m′12Δt13+m′13Δt14}, (A16-2)
C={m′11ΔD12+m′12ΔD13+m′13ΔD14−x1}2+{m′21ΔD12+m′22ΔD13+m′23ΔD14−y1}2+{m′31ΔD12+m′32ΔD13+m′33ΔD14−z1}2, (A16-3)
The solution b having the smaller magnitude is preferably chosen as the solution to be used. Equations (A15) and (A13-j) (j=1, 2, 3) provide a solution quadruple (x,y,z,b) for K=4. This solution quadruple (x,y,x,b) is exact, does not require iterations or other approximations, and can be determined in one pass.
This approach can be used, for example, where a short range radio frequency identifier device (RFID) or other similar signal source provides a signal that is received by each of K signal receivers 81-k. The signal source may have its own power source (e.g., a battery), which must be replaced from time to time.
Alternatively, each of the K (K≧3) signal transceivers 91-k can serve as an initial signal source, as illustrated in
Δtk=tr,k−te,k={tf,k−te,k−Δtd,k}/2(k−1, . . . , K), (A17)
and the time interval Δtk set forth in Eq. (A14) can be used as discussed in connection with Eqs. (A1)-(A17). However, in this alternative, times at the initial signal sources 91-k are coordinated, and any constant time shift b at target receiver 93 is irrelevant, because only the time differences (of lengths Δtr,k) are measured or used to determine the time(s) at which the return signal(s) are emitted. Thus, b=0 in this alternative, and the relation corresponding to Eq. (A10) (with b=0) provides the solution coordinates (x,y,z).
The method set forth in connection with Eqs. (A1)-(A7-4) for K≧5 receivers, and the method set forth in connection with Eqs. (A1)-(A17) for K=4 receivers, will be referred to collectively as a “quadratic analysis process” to determine location coordinates (x,y,z) and equivalent time shift b for a mobile object or Carrier.
Determination of Spatial Orientation Relations
The preceding determines location of a single (target) receiver that may be carried on a person or other mobile object (hereafter referred to as a “Carrier”). Spatial orientation of the Carrier can be estimated by positioning three or more spaced apart, noncollinear target receivers on the Carrier and determining the three-dimensional location of each target receiver at a selected time, or within a time interval of small length (e.g., 0.5-5 sec). Where the Carrier is a person, the target receivers may, for example, be located on or adjacent to the Carrier's head or helmet and at two or more spaced apart, noncollinear locations on the Carrier's back, shoulders, arms, waist or legs.
Three spaced apart locations determine a plane Π in 3-space, and this plane Π can be determined by a solution (a,b,c) of the three relations
x·cos α+y·cos β+z·cos γ=p, (A18)
where α, β and γ are direction cosines of a vector V, drawn from the coordinate origin to the plane Π and perpendicular Π, and p is a (signed) length of V (W. A. Wilson and J. I. Tracey, Analytic Geometry, D. C. Heath publ. Boston, Third Ed. 1946, pp. 266-267). Where three noncollinear points, having Cartesian coordinates (xi,yi,zi) (I=1, 2, 3), lie in the plane Π, these coordinates must satisfy the relations
xi·cos α+yi·cos αβ+zi·cos αγ=p, (A19)
and the following difference equations must hold:
(x2−x1)·cos α+(y2−y1)i·cos β+(z2−z1)·cos γ=0, (A20-1)
(x3−x1)·cos α+(y3−y1)i·cos β+(z3−z1)·cos γ=0. (A20-2)
Multiplying Eq. (A20-1) by (z3−z1), multiplying Eq. (A20-2) by (z2−z1), and subtracting the resulting relations from each other, one obtains
{(z3−z1)(x2−x1)−(z2−z1)(x3−x1)}cos α, +{(z3−z1)(y2−y1)−(z2−z1)(y3−y1)}cos β=0, (A21)
The coefficient {(z3−z1)(y2−y1)−(z2−z1)(y3−y1)} of cos β is the (signed) area of a parallelogram, rotated to lie in a yz-plane and illustrated in
Equation (21) has a solution
cos β=−{(z3−z1)(x2−x1)−(z2−z1)(x3−x1)}cos α/{(z3−z1)(y2−y1) −(z2−z1)(y3−y1)} (A23)
Multiplying Eq. (A20-1) by (y3−y1), multiplying Eq. (A20-2) by (y2−y1), and subtracting the resulting relations, one obtains by analogy a solution
cos γ=−}(y3−y1)(x2−x1)−(y2−y1)(x3−x1)}cos α/{(z3−z1)(y2−y1) −(z2−z1)(y3−y1)}. (A24)
Utilizing the normalization relation for direction cosines,
cos2α+cos2β+cos2γ=1, (A25)
one obtains from Eqs. (A23), (A24) and (A25) a solution
cos α=(±1)/{1+{(z3−z1)(x2−x1)−(z2−z1)(x3−x1)}2/{(z3−z1)(y2−y1) −(z2−z1)(y3−y1)}2+{(y3−y1)(x2−x1)−(y2−y1)(x3−x1)}/{(z3−z1)(y2−y1) −(z2−z1)(y3−y1)}2}1/2. (A26)
Equations (A23), (A24) and (A26) provide a solution for the direction cosines, cos α, cos β, and cos γ, apart from the signum in Eq. (A26). The signum (±1) in Eq. (A26) is to be chosen to satisfy Eq. (A18) after the solution is otherwise completed. The (signed) length p can be determined form Eq. (A18) for, say, (x1,y1,z1).
A fourth point, having location coordinates (x,y,z)=(x4,y4,z4), lies on the same side of the plane Π as does the origin if
x4·cos α+y4·cos αβ+z4·cos αγ=p4<p, (A27-1)
lies on the opposite side of the plane Π from the origin if
x4·cos α+y4·cos αβ+z4·cos αγ=p4>p, (A27-2)
and lies on the plane Π if
x4·cos α+y4·cos αβ+z4·cos αγ=p4=p, (A27-3)
The fourth point may have location coordinates that initially place this point in the plane Π, for example, within a triangle Tr initially defined by the other three points (xi,yi,zi). As a result of movement of the Carrier associated with the RFIDs, the fourth point may no loner lie in the (displaced) plane Π and may lie to one side or to the other side of Π. From this movement of the fourth point relative to Π, one infers that the Carrier has shifted and/or distorted its position, relative to its initial position.
The analysis presented here in connection with Eqs. (A18)-(A27-3) will be referred to collectively as a “quadratic orientation analysis process.”
An initial set of spatial orientation parameters (α0,β0,γ0,p0) may be specified, and corresponding members of a subsequent set (α,β,γ,p) can be compared with (α0,β0,γ0,p0) to determine which of these parameters has changed substantially.
As an example, the Carrier may be an ESW, and the initial plane Π may be substantially horizontal, having direction cosines cos α≈0, cos β≈0 and cos γ≈1 (e.g., cos γ≧0.97). If, at a subsequent time, cos γ≦0.7 for a substantial time interval, corresponding to a Carrier “lean” angle of at least 45°, relative to a vertical direction, the system may conclude that the Carrier is no longer erect and may be experiencing physical or medical problems.
As another example, if (α0,β0,γ0) are substantially unchanged from their initial or reference values but the parameter p is changing substantially, this indicates that the Carrie is moving, without substantial change in the initial posture of the Carrier.
This invention was made, in part, by one or more employees of the U.S. government. The U.S. government has the right to make, use and/or sell the invention described herein without payment of compensation therefor, including but not limited to payment of royalties.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5448220 | Levy | Sep 1995 | A |
| 5689234 | Stumberg et al. | Nov 1997 | A |
| 5793882 | Piatek et al. | Aug 1998 | A |
| 5990793 | Bieback | Nov 1999 | A |
| 6268798 | Dymek et al. | Jul 2001 | B1 |
| 6778081 | Matheny | Aug 2004 | B2 |
| 7019652 | Richardson | Mar 2006 | B2 |
| 7064660 | Perkins et al. | Jun 2006 | B2 |
