Patent Application
20110243532
The present disclosure relates generally to video systems and in particular to a system and method of video stabilization during movement.
With ever increasing technological advances in camera and video technology, cameras and video recorders as incorporated within an array of portable electronic devices are becoming a ubiquitous aspect for enhancing our everyday lives. In one illustration, personal video recorders are integrated within remote speaker microphones as used by law enforcement officers and first responders in an effort to ensure their individual well-being, act as a deterrent so as to improve overall public safety, and enhance the quality and process of collecting legal evidence. In other cases, personal video recorders are implemented as stand-alone devices.
Ideally, a captured video stream would also simplify the process of indisputably establishing the facts in a particular incident and would provide other valuable legal evidence while law enforcement officials or other first responders are engaged in that particular incident. Optionally, a wearable video recorder provides real-time awareness to remote viewers to enhance responsiveness to a dynamically changing situation.
In practice, a wearable video recorder is worn by a user at any of a variety of locations, such as their shoulder, chest, waist, head or elsewhere. As the video recorder moves in tandem with the wearer during activity such as standing/sitting, walking, running or other movements, the resulting captured video stream often features image frames that vary suddenly and widely such that it is difficult to view a desired reference area. Currently, an electronic image stabilization (EIS) system technology is used to remove jitter or other small, irregular movements. However, an EIS system cannot correct for the larger movements encountered during increased or non-stable activity as referenced above.
Accordingly, there is a need for a system and method of video stabilization during movement.
The accompanying figures, where like reference numerals refer to identical or functionally similar elements throughout the separate views, together with the detailed description below, are incorporated in and form part of the specification and serve to further illustrate various embodiments of concepts that include the claimed invention, and to explain various principles and advantages of those embodiments.
Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help improve understanding of various embodiments. In addition, the description and drawings do not necessarily require the order illustrated. It will be further appreciated that certain actions and/or steps may be described or depicted in a particular order of occurrence while those skilled in the art will understand that such specificity with respect to sequence is not actually required.
Apparatus and method components have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the various embodiments so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein. Thus, it will be appreciated that for simplicity and clarity of illustration, common and well-understood elements that are useful or necessary in a commercially feasible embodiment may not be depicted in order to facilitate a less obstructed view of these various embodiments.
Generally speaking, pursuant to the various embodiments, the present disclosure provides a system and method of video stabilization during movement, such as a video stabilization system featuring an imaging system and a motion sensor. The imaging system moves in tandem with a wearer during activity while the motion sensor detects motion of the imaging system. Specifically, illustrative embodiments of the present disclosure are described below with reference to wearable video recorders. However, at least some of the illustrative embodiments and their associated advantages can be implemented and appreciated in a variety of other video systems capable of supporting the systems and methods as described herein. Further, embodiments of the present disclosure are described herein with reference to applications in law enforcement as well as associated with other first responders. However, at least some of the illustrative embodiments as described below can be implemented in a variety of other applications and settings including business, security or other commercial use, as well as personal and recreational use.
As described herein, methods for video stabilization during movement include a method for editing a video stream by a video stabilization system that features an imaging system and a motion sensor. The video stream includes a sequence of image frames captured by the imaging system. As the video stream is captured, the motion sensor from the video stabilization system detects a pattern of recurring motion. The video stabilization system generates motion sensor data for the sequence of image frames that characterizes, corresponds to, and represents the pattern of recurring motion. In some embodiments, at least one image frame of the sequence of image frames is edited based on a comparison of the motion sensor data for the sequence of image frames to motion sensor data for a region of interest for the imaging system.
Optionally, the video stream and the motion sensor data are synchronized, for example, by a time stamp in order to facilitate the editing of the at least one image frame. In some embodiments, the unedited video stream is recovered from the edited video stream. It may be desirable, for example, to view the original, unedited video stream, or to select a different region of interest or reference frame, as discussed below, for editing of the video stream.
A video stabilization system includes a processor. Methods for stabilizing a video stream by the video stabilization system include receiving a video stream from an imaging system. The video stream includes a plurality of image frames. The processor receives motion sensor data for a set of image frames of the plurality of image frames. The motion sensor data characterizes a pattern of recurring motion for the imaging system. The video stabilization system determines an image orientation. Based on a comparison of the motion sensor data to the image orientation, at least one image frame from the set of image frames is edited at the processor.
A system for video stabilization as disclosed herein includes an imaging system, a motion sensor coupled to the imaging system, and a processor coupled to the motion sensor. The imaging system captures a video stream including a plurality of image frames. The motion sensor detects a pattern of recurring motion of the imaging system and generates corresponding motion sensor data. The processor edits at least one of the plurality of image frames based on the motion sensor data corresponding to the pattern of recurring motion of the imaging system.
The embodiments as disclosed herein can generally be applied to any mobile imaging system experiencing a pattern of recurring motion. Further, the benefits and advantages as described herein are also applicable to systems employing electronic image stabilization (EIS) technology as well as other hardware and software devices that are capable of supporting the embodiments disclosed below. Those skilled in the art will realize that the above recognized advantages and other advantages described herein are merely illustrative and are not meant to be a complete rendering of all of the advantages of the various embodiments.
Referring now to the figures,
Consequently, in various embodiments, the FOV is directly dependent on a lens or set of optics that is used in the imaging system. In some embodiments, the ROI includes the entire FOV. Alternatively, the ROI includes a subset of the entire FOV. In some instances, the ROI is completely outside the FOV.
Illustratively, in some embodiments, a region of interest, ROI, is determined according to a desired image orientation, where the motion sensor determines an orientation of the imaging system by measuring a direction of acceleration due to gravity. For example, an orientation of the imaging system, and thus the region of interest, is determined such that the selected region of interest aligns with forward motion of the officer while in pursuit of the suspect 102. Alternatively, the region of interest is determined by selecting a fixed reference object within the field of view of the imaging system, such as a tree 108. In other embodiments, the region of interest is automatically determined, for example, by using motion sensor data for the sequence of image frames and/or by detection of a horizon, as indicated by a dashed line H corresponding to FOV(N) and a dashed line H′ corresponding to FOV(N+1). Optionally, the region of interest is configured using external input such as by being selected manually by a user based on a specific situation or other preference.
In some embodiments, as the video recorder moves in tandem with the officer while in pursuit of the suspect 102, the motion sensor detects a pattern of recurring motion. Illustratively, the video stabilization system 100 edits the video stream by selecting, removing, rotating, shifting or otherwise editing a set or subset of image frames of the video stream. The edited image frames are either partially or completely outside of the desired region of interest, for example, as determined by the motion sensor. Alternatively, the edited image frames are shifted and/or rotated and are still completely within the region of interest. As used herein, an unedited video stream is equivalently referred to as a “raw video stream”. Likewise, as used herein, an edited video stream is equivalently referred to as a “corrected video stream”, a “smoothed video stream” or a “stabilized video stream”.
In some embodiments, the raw video stream is edited either locally or remotely in real-time or during video stream capture, wherein real-time means with negligible time delay. The resulting corrected video stream of the suspect's 102 pursuit is monitored remotely, for example, by transmitting the video stream to a command center, individual supervising personnel, other officers in the field searching for the suspect 102, or other support personnel. Thus, as the officer pursues the suspect 102, the command center or other support personnel are able to enhance their responsiveness to a dynamically changing situation and readily deduce critical information, for example, what additional support is needed, a location of the incident in progress, and an identification of the suspect 102, among others. In addition, in some embodiments, the video stabilization system 100 includes a display 106 for local viewing of the video stream as captured by the imaging system. Various embodiments of the present disclosure for acquiring and editing a video stream based on motion sensor data are discussed in further detail below with reference to
As indicated by an arrow 207, the video stabilization system 202 is coupled to a general-purpose computer system 200 by way of, for example, any well-known combination of a wired and/or wireless network connection. The computer system 200 is suitable for implementing one or more embodiments as herein described, including editing of a captured video stream based on motion sensor data. Further, the computer system 200 includes a local or remote computer system optimally configured for real-time and/or post-processing of the video stream, wherein post-processing means at the termination or end of or after the video stream capture. In particular, the computer system 200 comprises a processor 220, which is often referred to as a central processor unit or CPU, that is in communication with various memory devices including a secondary storage 250, a read only memory (ROM) 240, a random access memory (RAM) 230, a plurality of input/output (I/O) devices 210, and a plurality of network connectivity devices 260. Furthermore, the processor 220 can send and/or receive data from the memory 208 of the video stabilization system 202 by way of the network connection as indicated by the arrow 207. The processor can be implemented as one or more CPU chips.
The secondary storage 250 comprises one or more disk drives or tape drives. The secondary storage 250 is used for non-volatile storage of data and as an overflow data storage device if the RAM 230 is not large enough to hold all working data. The secondary storage 250 can be used to store programs that are loaded into the RAM 230 when selected for execution. The ROM 240 is used to store instructions and perhaps data, which are read during program execution. The ROM 240 is a non-volatile memory device that often has a small memory capacity relative to the larger memory capacity of secondary storage. The RAM 230 is used to store volatile data and optionally to store instructions. Access to both the ROM 240 and the RAM 230 is often faster than accessing the secondary storage 250.
Illustratively, the plurality of I/O devices 210 include printers, video monitors, liquid crystal displays (LCDs), touch screen displays, keyboards, keypads, switches, dials, mice, track balls, voice recognizers, card readers, paper tape readers, or other well-known input devices. The plurality of network connectivity devices 260 can comprise, among others, modems, modem banks, Ethernet cards, universal serial bus (USB) interface cards, serial interfaces, token ring cards, fiber distributed data interface (FDDI) cards, wireless local area network (WLAN) cards, radio transceiver cards such as code division multiple access (CDMA) and/or global system for mobile communications (GSM) radio transceiver cards, and other well-known network devices.
These plurality of network connectivity devices 260, for instance, enable the processor 220 to communicate with a global information system or one or more intranets. With such a network connection, the processor 220 receives information from a connected network or outputs information to the network in the course of implementing one or more of the embodiments as described herein. The information is received by the processor 220 and provided to the network in a variety of formats such as, among others, a computer data baseband signal and a signal embodied in a carrier wave. In one illustrative embodiment, the information comprises data and instructions for execution by the processor 220.
In one illustrative implementation, the baseband signal or signal embodied in the carrier wave generated by the plurality of network connectivity 260 devices propagates in or on the surface of electrical conductors, in coaxial cables, in waveguides, in optical media such as optical fiber, or in the air or free space. In another illustrative implementation, the information contained in the baseband signal or signal embedded in the carrier wave is arranged in accordance with different sequences that are desired for processing, generating the information, transmitting, and receiving the information. The baseband signal or signal embedded in the carrier wave, or other types of signals currently used or hereafter developed, referred to herein as the transmission medium, are readily generated according to several methods well-known to one skilled in the art. The processor 220 executes instructions, codes, computer programs, scripts which it accesses from the ROM 240, the RAM 230, the plurality of network connectivity devices 260 or from a hard disk, a floppy disk, and/or an optical disk, where various disk-based systems are considered as the secondary storage 250.
Referring now to
As used herein, a “pattern of recurring motion” includes a computer-recognizable pattern or trend in the motion sensor data. In some embodiments, the pattern of recurring motion includes periodic motion. However, embodiments of the present disclosure are not meant to be limited to periodic motion, but instead are meant to include any computer-recognizable pattern of recurring motion. Thus, the various embodiments as disclosed herein take into account variations introduced by differences in individual gait, body height and weight, differences in walking and running speeds, and variations occurring when alternating between running and walking motions, among others. For instance, in one illustrative implementation, the pattern of recurring motion is detecting by comparing the motion sensor data for the image frames to stored threshold motion sensor data or stored patterns or using any suitable pattern recognition algorithm.
Still referring to
The motion sensor data 304 comprises a computer-recognizable pattern that corresponds to the movement of the FOV of the imaging system through the sequence of positions N, N+1, N+2, N+3, N+4, N+5, as shown. In particular, the motion sensor data 304 varies in accordance with the orientation of the FOV. Thus, the pattern present in the motion sensor data 304 is representative of the orientation of the FOV of the imaging system, where similar patterns in the motion sensor data 304 correspond to similar orientations of the FOV. For example, the similar patterns in the motion sensor data 304 at positions as indicated by an arrow 308 and an arrow 310 correspond to similar orientations of the FOV(N) and the FOV(N+4), as represented by the image frames M and M+4.
Accordingly, in some embodiments, it is desirable to edit the raw video stream 306 such that an edited video stream comprises only those image frames having a desired image orientation as determined by the motion sensor data 304. In this example, it is desirable to construct an edited video stream that comprises only those image frames where the FOV includes the entire ROI such as, for example, the image frames M and M+4. Thus, an edited video stream 312, comprising the image frames M and M+4, is constructed from the raw video stream 306 by comparison of the motion sensor data 304 for the sequence of image frames to motion sensor data for the ROI as defined for the imaging system, for example corresponding to the FOV(N). As shown, the raw video stream 306 has a greater frame rate than the edited video stream 312. However, in some embodiments as discussed below, the raw video stream 306 and the edited video stream 312 have equal frame rates.
Illustratively, selection and editing of specific image frames, for example, that match motion sensor data corresponding to the ROI for correction of the captured video stream includes one or more of a plurality of editing processes such as, for instance: an image frame delete; an image frame crop; an image frame shift; or an image frame rotate. For example, as the FOV(N+1) is shifted and rotated with respect to the FOV(N) and includes a portion of the ROI, the image frame represented by the FOV(N+1) is corrected by shifting, rotating, and cropping the image frame. In like manner, as the FOV(N+2) and the FOV(N+3) do not include any of the ROI, the image frames represented by the FOV(N+2) and the FOV(N+3) are deleted or dropped. Thus, the unedited video stream has a greater frame rate than the corrected video stream. In other embodiments, the captured video stream is corrected without deletion of any of the original image frames. Thus, the unedited video stream and the corrected video stream have equal frame rates.
On receipt of the motion sensor data 608 and the corresponding raw video stream, the processor 604 edits the raw video stream to generate a corrected video stream as indicated by an arrow 607. Accordingly, the raw video stream is edited based on a comparison of the motion sensor data for the sequence of image frames for the raw video stream to motion sensor data corresponding to a defined region of interest (ROI) for an imaging system of a video stabilization system.
As shown, the corrected video stream includes a plurality of image frames M, M+1, M+2, M+3. Thus, in the example of
With further reference to
On receipt of the motion sensor data and the corresponding raw video stream, the processor 710 edits, in real-time, the raw video stream to generate an edited or corrected video stream as indicated by an arrow 714, where the corrected video stream includes a plurality of image frames M, M+1, M+2, M+3, M+4, M+5, M+6. The editing of the raw video stream is based on a comparison of the motion sensor data for the sequence of image frames for the raw video stream to motion sensor data corresponding to a defined region of interest (ROI) for the imaging system 704 of the video stabilization system 702. As shown, the raw video stream and the corrected video stream have equal frame rates. However, in other embodiments of real-time processing, the raw video stream has a greater frame rate than the corrected video stream.
In some embodiments, such as in the example of a law enforcement officer pursuing a suspect as discussed above, the remote computer system 708 is located at a command center or at a location linked with other personnel remotely supporting the pursuit, where the corrected video stream is monitored in real-time. Thus, as the officer pursues the suspect, the command center or other support personnel are able to improve incident responsiveness by deducing, for example, whether additional support is needed, a general location of the suspect and of the incident in progress, and an identification of the suspect, among others. Optionally, the remote computer system 708 is located on the law enforcement officer, for example attached to the officer's uniform, where the captured video stream is corrected locally at the officer's location and the corrected video stream is transmitted to the command center, other support personnel, or other remote location.
In some embodiments, the motion sensor detects a pattern of recurring motion (808). If a pattern of recurring motion is detected at 808, the video stabilization system determines if image frames of the captured video stream are outside the region of interest (810). That is, the video stabilization system determines if the field of view, as represented by each of the image frames of the video stream, does not include any portion of the region of interest.
At 812, the video stabilization system determines if periodic frame selection is required. For example, if the video stabilization system determines that each of the image frames of the video stream include at least a portion of the region of interest, then periodic frame selection is not required. However, if the video stabilization system determines that one or more image frames of the captured video stream are outside the region of interest, then periodic frame selection is required, and the video stabilization system performs periodic image frame selection (814).
Still referring to
In the foregoing specification, specific embodiments have been described. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the invention as set forth in the claims below. For example, in some embodiments, editing of the raw video stream is manually triggered. In other embodiments, editing of the raw video stream is automatically triggered based on detection of a pattern of recurring motion or based on detection of a threshold amount of movement as determined by a motion sensor. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of present teachings.
The benefits, advantages, solutions to problems, and any element(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential features or elements of any or all the claims. The invention is defined solely by the appended claims including any amendments made during the pendency of this application and all equivalents of those claims as issued.
Moreover in this document, relational terms such as first and second, top and bottom, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms “comprises,” “comprising,” “has”, “having,” “includes”, “including,” “contains”, “containing” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, has, includes, contains a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by “comprises . . . a”, “has . . . a”, “includes . . . a”, “contains . . . a” does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises, has, includes, contains the element. The terms “a” and “an” are defined as one or more unless explicitly stated otherwise herein. The terms “substantially”, “essentially”, “approximately”, “about” or any other version thereof, are defined as being close to as understood by one of ordinary skill in the art, and in one non-limiting embodiment the term is defined to be within 10%, in another embodiment within 5%, in another embodiment within 1% and in another embodiment within 0.5%. The term “coupled” as used herein is defined as connected, although not necessarily directly and not necessarily mechanically. A device or structure that is “configured” in a certain way is configured in at least that way, but may also be configured in ways that are not listed. Also, the sequence of steps in a flow diagram or elements in the claims, even when preceded by a letter does not imply or require that sequence.
It will be appreciated that some embodiments may be comprised of one or more generic or specialized processors (or “processing devices”) such as microprocessors, digital signal processors, customized processors and field programmable gate arrays (FPGAs) and unique stored program instructions (including both software and firmware) that control the one or more processors to implement, in conjunction with certain non-processor circuits, some, most, or all of the functions of the method and/or apparatus described herein. Alternatively, some or all functions could be implemented by a state machine that has no stored program instructions, or in one or more application specific integrated circuits (ASICs), in which each function or some combinations of certain of the functions are implemented as custom logic. Of course, a combination of the two approaches could be used.
Moreover, an embodiment can be implemented as a computer-readable storage medium having computer readable code stored thereon for programming a computer (e.g., comprising a processor) to perform a method as described and claimed herein. Examples of such computer-readable storage mediums include, but are not limited to, a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, a ROM (Read Only Memory), a PROM (Programmable Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read Only Memory) and a Flash memory. Further, it is expected that one of ordinary skill, notwithstanding possibly significant effort and many design choices motivated by, for example, available time, current technology, and economic considerations, when guided by the concepts and principles disclosed herein will be readily capable of generating such software instructions and programs and ICs with minimal experimentation.
The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.
