Patent Application
20210134129
Public safety personnel (for example, first responders, investigators, and the like) responding to an incident scene may be equipped with video recording devices to document the response to an incident. The resulting video includes objects of interest located at the incident scene during the response. Objects of interest include people, vehicles, buildings, or portions of buildings, and other physical objects involved or potentially involved in the incident. In some instances, the effectiveness of the response or the safety of the responders to an incident may depend on whether the incident scene is under adequate visual control. The video of the incident may be used subsequent to the incident as evidence in criminal trials or other official investigations or proceedings, or for training purposes.
The accompanying figures, where like reference numerals refer to identical or functionally similar elements throughout the separate views, together with the detailed description below, are incorporated in and form part of the specification, and serve to further illustrate embodiments of concepts that include the claimed invention, and explain various principles and advantages of those embodiments.
Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of embodiments of the present invention.
The apparatus and method components have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments of the present invention so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.
Public safety personnel responding to an incident scene may be equipped with video recording devices to document the response to an incident. For example, public safety personnel may use body worn cameras, helmet cameras, or smart glasses to take video of their actions responding to the incident. In addition, other cameras (for example, traffic cameras, surveillance cameras, vehicle dash cameras, and the like) may capture video of the incident. The resulting video feeds include objects of interest located at the incident scene during the response. Objects of interest include people, vehicles, buildings, or portions of buildings, and other physical objects involved or potentially involved in the incident. In some instances, the effectiveness of the response or the safety of the responders to the incident may depend on whether the incident scene is under adequate visual control. In other cases, when and by whom an object of interest was observed during the incident may need to be determined subsequent to the incident. For example, video of the incident may be used as evidence in criminal trials, other official investigations or proceedings, or for training purposes.
Although video streams are recorded for some individual responders, the video stream itself may not be enough to indicate that the responder was observing a particular object at a particular time. For example, several objects of interest may be present in the responder's field of view. Additionally, some video sources (for example, a stationary camera) may not be associated with an individual responder. As a consequence, current video recording and playback systems are unable to reliably identify whether objects of interests are or were being watched during an incident. Accordingly, systems and methods are provided herein for, among other things, identifying watchers of an object of interest at an incident scene.
One example embodiment provides a system for identifying watchers of an object of interest at an incident scene. The system includes a communication interface and an electronic processor communicatively coupled to the communication interface. The electronic processor is configured to receive, via the communication interface, an object identifier corresponding to the object of interest. The electronic processor is configured to determine a watcher status for the object of interest. The electronic processor is configured to generate a notification based on the watcher status and the object identifier. The electronic processor is configured to transmit the notification to an available watcher at the incident scene.
Another example embodiment provides a method for identifying watchers of an object of interest at an incident scene. The method includes receiving, with an electronic processor, an object identifier corresponding to the object of interest. The method includes determining a watcher status for the object of interest. The method includes generating a notification based on the watcher status and the object identifier. The method includes transmitting, with a communication interface communicatively coupled to the electronic processor, the notification to an available watcher at the incident scene.
For ease of description, some or all of the example systems presented herein are illustrated with a single exemplar of each of its component parts. Some examples may not describe or illustrate all components of the systems. Other example embodiments may include more or fewer of each of the illustrated components, may combine some components, or may include additional or alternative components.
The server 102 is communicatively coupled to a first video device 106, a second video device 108, and a camera 110 via a communications network 112. The communications network 112 is a communications network including wireless and wired connections. The communications network 112 may be implemented using a wide area network, such as the Internet, a local area network, such as a Bluetooth™ network or Wi-Fi, a Long Term Evolution (LTE) network, a Global System for Mobile Communications (or Groupe Special Mobile (GSM)) network, a Code Division Multiple Access (CDMA) network, an Evolution-Data Optimized (EV-DO) network, an Enhanced Data Rates for GSM Evolution (EDGE) network, a 3G network, a 4G network, and combinations or derivatives thereof.
The first video device 106, the second video device 108, and the camera 110 include image capture devices for capturing images and video streams, including a portion of or the entire incident scene 114. The first video device 106 and the second video device 108, described more particularly below with respect to
The camera 110 may be a vehicle dash camera, a surveillance camera, a traffic camera, or another suitable image capture device that records video of the incident scene 114 from a fixed or movable position. As shown in
The first video device 106, the second video device 108, and the camera 110 transmit their respective captured images and video streams to the server 102, via the communications network 112. In some embodiments, the first video device 106 and the second video device 108 may receive video streams directly from one another or the camera 110, or indirectly from the server 102 via the communications network 112.
An incident may be a public safety operation, for example, responding to a crime, responding to a traffic accident, searching for a suspect, locating a missing person, responding to a fire, responding to a medical emergency, and the like. As noted above, the response to the incident by public safety personnel may be recorded on video by the first video device 106, the second video device 108, and the camera 110. The recorded video may capture video of objects of interest within the incident scene 114. For example, an incident may involve the first object of interest 116, the second object of interest 118, and the third object of interest 120.
An object of interest may be any object present at the incident scene, which object is related to the incident. An object is related to the incident if it is involved in or relevant to the response to or investigation of the incident. Objects of interest may include, for example, automobiles (for example, in the case of a traffic accident), weapons (for example, in the case of a crime scene), people (for example, victims or suspects), buildings, or portions of buildings (for example, doors and windows). Objects of interest may also be tangible things not commonly thought of as objects, but which are still relevant to the response to or subsequent investigation of an incident response (for example, fluids leaked from automobiles, debris from damaged property, and blood). The incident scene 114 may include fewer or more than three objects of interest.
In some embodiments, the server 102, the first video device 106, and the second video device 108 include software and hardware to electronically detect and classify objects within images and video streams captured by the devices (for example, video processors and object classifier algorithms). Object classification is known in the art, and will not be described in detail herein. The detected objects of interest are assigned object identifiers, and stored in the database 104.
All objects of interest may not be captured by all image capture devices monitoring the incident scene 114. For example, as shown in
As described in detail below, with respect to
The electronic processor 205 obtains and provides information (for example, from the memory 210 and/or the input/output interface 215), and processes the information by executing one or more software instructions or modules, capable of being stored, for example, in a random access memory (“RAM”) area of the memory 210 or a read only memory (“ROM”) of the memory 210 or another non-transitory computer readable medium (not shown). The software can include firmware, one or more applications, program data, filters, rules, one or more program modules, and other executable instructions. The electronic processor 205 is configured to retrieve from the memory 210 and execute, among other things, software related to the control processes and methods described herein.
The memory 210 can include one or more non-transitory computer-readable media, and includes a program storage area and a data storage area. The program storage area and the data storage area can include combinations of different types of memory, as described herein. In the embodiment illustrated, the memory 210 stores, among other things, the first watcher identifier 130 and metadata 250 (both described in detail below).
The input/output interface 215 is configured to receive input and to provide system output. The input/output interface 215 obtains information and signals from, and provides information and signals to, (for example, over one or more wired and/or wireless connections) devices both internal and external to the first video device 106.
The electronic processor 205 is configured to control the baseband processor 220 and the transceiver 225 to transmit and receive video and other data to and from the first video device 106. The baseband processor 220 encodes and decodes digital data sent and received by the transceiver 225. The transceiver 225 transmits and receives radio signals to and from, for example, the communications network 112 using the antenna 230. The electronic processor 205, the baseband processor 220, and the transceiver 225 may include various digital and analog components, which for brevity are not described herein and which may be implemented in hardware, software, or a combination of both. Some embodiments include separate transmitting and receiving components, for example, a transmitter and a receiver, instead of a combined transceiver 225.
Some embodiments include an object marker 235 for marking on an object of interest being observed by the watcher 107. In some embodiments, the object of interest being observed is determined based on a direction in which the head of the watcher 107 is pointing. For example, the first video device 106 may be integrated with or coupled to smart glasses or a helmet worn by the first watcher 107. In some embodiments, the object marker 235 is a laser marker projecting from the watcher in the direction of the watcher's gaze. For example, a laser may be integrated into or mounted on smart glasses worn by the first watcher 107. The laser would project a modulated beam outside of the human-visible spectrum in the direction of the gaze 126 (see
In some embodiments, the electronic processor 205 determines what object of interest is being observed by using known gaze detection techniques and hardware (not shown) integrated into or coupled to the first video device 106. Gaze detection may be used to aim a laser in the direction of the gaze, or may be used in conjunction with video processing to identify the object of interest being observed by the first watcher 107.
The camera 240 is an image capture device for capturing images and video streams, including a portion or the entire incident scene 114, by, for example, sensing light in at least the visible spectrum. The camera 240 communicates the captured images and video streams to the electronic processor 205 via the input/output interface 215. It should be noted that the terms “image” and “images,” as used herein, may refer to one or more digital images captured by the camera 240, or processed by the electronic processor 205, or displayed on the display 245. Further, the terms “image” and “images,” as used herein, may refer to still images or sequences of images (that is, a video stream). As illustrated, the camera 240 is integrated into the first video device 106. In alternative embodiments, the camera 240 is separate from the first video device 106, and communicates captured images to the first video device 106 via a wired or wireless connection. For example, the camera 240 may be integrated into a body-worn camera or smart glasses, which communicate with the first video device 106.
In some embodiments, the camera 240 may be a stereoscopic camera, or the first video device 106 may include a stereoscopic camera. In such embodiments, the first video device 106 can capture three-dimensional information about the incident scene 114 and the objects of interest. In some embodiments, three-dimensional information may be captured using radar sensors or infrared ranging sensors (not shown).
The display 245 is a suitable display such as, for example, a liquid crystal display (LCD) touch screen, or an organic light-emitting diode (OLED) touch screen. The first video device 106 implements a graphical user interface (GUI) (for example, generated by the electronic processor 205, from instructions and data stored in the memory 210, and presented on the display 245), that enables a user to interact with the first video device 106. In some embodiments, the first video device 106 operates or is integrated with a head-mounted display (HMD), an optical head-mounted display (OHMD), or the display of a pair of smart glasses.
In some embodiments, the first video device 106 operates using, among other things, augmented reality technology, where live images are captured by the camera 240 and displayed (for example, on the display 245) with text, graphics, or graphical user interface elements superimposed on or otherwise combined with the live images.
As described in detail below, the first video device 106 is capable of receiving and processing images captured by the camera 240, and displaying processed images in a graphical user interface on the display 245. Computerized image capturing and processing techniques are known, and will not be described in detail.
In some embodiments, the first video device 106 is a body worn camera. In other embodiments, the first video device 106 may be smart glasses, integrated into a helmet or other headgear, or may be another portable or mobile electronic device containing software and hardware enabling it to operate as described herein.
The electronic processor 302 may include a microprocessor, application-specific integrated circuit (ASIC), or another suitable electronic device. The storage device 304 includes a non-transitory, computer-readable storage medium. The communication interface 306 may include a transceiver (for example, a Wi-Fi or Ethernet transceiver) for communicating over the communications network 112 and, optionally, one or more additional wired or wireless communication networks or connections. The electronic processor 302 is configured to retrieve from the storage device 304 and execute, among other things, software related to the processes and methods described herein.
As noted above, an object of interest may be any object present at the incident scene, which object is related to the incident. In some instances, the effectiveness of the response or the safety of the responders to an incident may depend on whether the incident scene and, one or more of the objects of interest present at the scene, are under adequate visual control. For example, when police officers respond to an incident including a group of criminal suspects, it may be advisable to visually monitor each suspect. In another example, it may be advisable to visually monitor each of multiple windows and doors on a building to prevent the undetected escape of a suspect or the appearance of a threat to the responders. In another example, some objects of interest may have evidentiary value and ensuring that those objects are continuously visually monitored may improve confidence, during future proceedings, that the evidence has been preserved. In another example, it may be desirable that each victim on the scene of a medical emergency is being or has been observed by emergency personnel. As a consequence, there is a need for identifying which personnel (“watchers”) responding to an incident scene are watching which object of interest, and alerting available watchers that one or more objects of interest are unwatched or not being watched adequately.
Accordingly,
As illustrated in
At block 404, the electronic processor 302 determines a watcher identifier corresponding to the object identifier. In some embodiments, the electronic processor 302 determines the watcher identifier from data received in the video stream from the first video device 106 or the second video device 108. The watcher identifier corresponds to a watcher of the object of interest, as identified by the object identifier received at block 402. In one example, shown in
At block 406, the electronic processor 302 determines a watcher status for the object of interest. A watcher status is an indication of which and how many watchers are watching the object of interest, and may also include some information relating to the watchers of the object of interest (for example, from the metadata 250). As noted above, the first video device 106 stores metadata 250 in the memory 210. The metadata 250 relates to the first watcher 107, who operates the first video device 106. In some embodiments, the metadata 250 may include the rank of the first watcher 107, a role assigned to the first watcher 107, and identifying information for the first watcher 107. The second video device 108 similarly incudes metadata for the second watcher 109.
In some embodiments, the electronic processor 302 receives an available watcher identifier corresponding to an available watcher at the incident scene. An available watcher is a responder to the incident, who is using a video device to record the incident response. For example, for the incident scene 114, the available watchers are the first watcher 107 and the second watcher 109. In such embodiments, the electronic processor 302 determines the watcher status based on the watcher identifier and the available watcher identifier. For example, in some instances, the electronic processor 302 is unable to determine a watcher identifier for the object of interest. This indicates that there are no watchers of that object of interest. For example, where the object identifier corresponds to the second object of interest 118 or the third object of interest 120, neither of the available watchers (the first watcher identifier 130 and the second watcher identifier 132) corresponds to those objects of interest.
In some embodiments, the watcher status is based on a quantity of watchers for the object of interest. For example, when responding to apprehend a criminal suspect, it may be advisable to have two officers watching the suspect at all times. In such embodiments, the electronic processor 302 determines more than one watcher identifier corresponding to the object identifier. This indicates that more than one watcher is watching the object of interest. The electronic processor 302 determines a quantity of watchers for the object of interest based on the watcher identifiers. The electronic processor 302 also determines a watcher threshold for the object of interest. In this example, the threshold is two watchers. In such embodiments, the electronic processor 302 determines the watcher status based on the quantity of watchers and the watcher threshold. For example, the watcher status indicates that more watchers are needed if the threshold is not met, and indicates that the object of interest is adequately watched when the threshold is met.
In some embodiments, the watcher status is based on a watcher priority. For example, it may be desirable that a watcher of sufficient priority is watching an object of interest. In such embodiments, the electronic processor 302 determines a watcher priority threshold for the object of interest. The watcher priority threshold may be, for example, a minimum rank or a particular role assignment. For example, it may be desirable that an object of interest at an incident be monitored by at least one supervisor. In another example, it may be desirable that an object of interest considered to be evidence be watched at all times by an evidence technician. The electronic processor 302 determines a watcher priority for the object of interest based on the watcher identifier. For example, the watcher identifier may be used to retrieve information on the rank or role assigned to the watcher. In such embodiments, the electronic processor 302 determines the watcher status based on the watcher priority and the watcher priority threshold. For example, the watcher status may indicate that the watchers of the object of interest meet the threshold, or it may indicate that the watcher priority threshold is not met.
In some embodiments, the watcher status is based on whether a particular watcher, deemed to be critical, is watching the object of interest. In such embodiments, the electronic processor 302 determines a critical watcher identifier for the object of interest. For example, the electronic processor 302 may receive the critical watcher identifier from a computer aided dispatch system via the communication interface 306. In such embodiments, the electronic processor 302 determines the watcher status based on the critical watcher identifier and the watcher identifier. For example, the watcher status will indicate whether or not the critical watcher is watching the object of interest.
Regardless of how the watcher status is determined, at block 408, the electronic processor 302 generates a notification based on the watcher status and the object identifier. The notification message includes the watcher status for the object of interest corresponding to the object identifier. The notification message may include a request for an available watcher, or a particular watcher, to watch the object of interest.
At block 410, the electronic processor 302 transmits the notification to an available watcher at the incident scene. For example, as illustrated in
In some embodiments, the server 102 receives and processes video streams to identify objects of interest and watchers at an incident scene. In some embodiments, this may be performed while the incident is occurring, to provide near real time feedback to responders. In other embodiments, the video processing may be performed subsequent to the incident, and may be used to assist investigators reviewing the incident response.
In one embodiment, the electronic processor 302 receives a first video stream including the object of interest. In one example, the electronic processor 302 receives a video stream produced by the first video device 106. In such embodiments, the electronic processor 302 detects the object of interest (for example, the first object of interest 116) in the first video stream, for example, using an object classifier. When the object is detected, the electronic processor 302 receives the object identifier by, for example, assigning an object identifier or retrieving an object identifier from the database 104. The electronic processor 302 determines a watcher identifier by detecting the watcher identifier on the object of interest. In some embodiments, the electronic processor 302 detects at least one optical laser marker on the object of interest, for example, produced by the object marker 235 of the first video device 106. The electronic processor 302 detects the watcher identifier based on the at least one optical laser marker. In some embodiments, the electronic processor 302 decodes a modulation of the optical laser marker to determine the watcher identifier. For example, as illustrated in
In some embodiments, the electronic processor 302 generates a first modified video stream by superimposing the watcher status on the first video stream, and presents the first modified video stream on a display. For example, in as illustrated in
In some embodiments, the display 503 is part of a video device (for example, the first video device 106) worn by a watcher during the incident. For example, the display 503 may be part of a pair of smart glasses worn by Officer X. In such embodiments, the modified video stream 502 provides the wearer of the display with a near real time view of the incident and intelligence on who is watching what during the incident response. For example, a modified video stream based on the incident scene 114 of
In some embodiments, the first modified video stream is generated by superimposing the watcher status and the notification on the first video stream. For example, when the notification indicates that an object of interest is not being watched, that object of interest may be highlighted. In another example, an indicator such as “No Watchers” or “Priority Not Met” may be superimposed on objects of interest as appropriate.
In some embodiments, the electronic processor 302 receives a second video stream including the object of interest (which is also present in the first video stream). As described above, the electronic processor 302 detects the object of interest in the second video stream. The electronic processor 302 generates a second modified video stream by superimposing the watcher status on the second video stream. For example,
In some embodiments, when both the modified video stream 502 and the second modified video stream 504 are generated by the server 102, the server 102 is configured to receive a user input selecting which video stream to present on the display 503. In some embodiments, the user input corresponds to one of the watcher identifiers present in the modified video streams, allowing a user to view the incident response from the different perspectives. For example, an investigator reviewing video for an incident response involving objects of interest 506, 508, and 510 may wish to focus on object of interest 508. When the watcher status 509 (
In the foregoing specification, specific embodiments have been described. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the invention as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of present teachings.
The benefits, advantages, solutions to problems, and any element(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential features or elements of any or all the claims. The invention is defined solely by the appended claims including any amendments made during the pendency of this application and all equivalents of those claims as issued.
Moreover in this document, relational terms such as first and second, top and bottom, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms “comprises,” “comprising,” “has,” “having,” “includes,” “including,” “contains,” “containing” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, has, includes, contains a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by “comprises . . . a,” “has . . . a,” “includes . . . a,” or “contains . . . a” does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises, has, includes, contains the element. The terms “a” and “an” are defined as one or more unless explicitly stated otherwise herein. The terms “substantially,” “essentially,” “approximately,” “about” or any other version thereof, are defined as being close to as understood by one of ordinary skill in the art, and in one non-limiting embodiment the term is defined to be within 10%, in another embodiment within 5%, in another embodiment within 1% and in another embodiment within 0.5%. The term “coupled” as used herein is defined as connected, although not necessarily directly and not necessarily mechanically. A device or structure that is “configured” in a certain way is configured in at least that way, but may also be configured in ways that are not listed.
It will be appreciated that some embodiments may be comprised of one or more generic or specialized processors (or “processing devices”) such as microprocessors, digital signal processors, customized processors and field programmable gate arrays (FPGAs) and unique stored program instructions (including both software and firmware) that control the one or more processors to implement, in conjunction with certain non-processor circuits, some, most, or all of the functions of the method and/or apparatus described herein. Alternatively, some or all functions could be implemented by a state machine that has no stored program instructions, or in one or more application specific integrated circuits (ASICs), in which each function or some combinations of certain of the functions are implemented as custom logic. Of course, a combination of the two approaches could be used.
Moreover, an embodiment can be implemented as a computer-readable storage medium having computer readable code stored thereon for programming a computer (e.g., comprising a processor) to perform a method as described and claimed herein. Examples of such computer-readable storage mediums include, but are not limited to, a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, a ROM (Read Only Memory), a PROM (Programmable Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read Only Memory) and a Flash memory. Further, it is expected that one of ordinary skill, notwithstanding possibly significant effort and many design choices motivated by, for example, available time, current technology, and economic considerations, when guided by the concepts and principles disclosed herein will be readily capable of generating such software instructions and programs and ICs with minimal experimentation.
The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/PL2017/050031 | 5/26/2017 | WO | 00 |
