The present invention generally relates to video imagery and more specifically to digital video processing.
Digitally processing streaming video data, such as television data, currently requires very powerful computing equipment to perform various calculations for each frame of video. Typically, reducing the processing time for video data requires corresponding increases in the processing power of the computing equipment. However, in applications such a space-based systems, there are financial, system resource, and other practical constraints that limit the complexity, memory, and computing power of the available computing equipment. For the reasons stated above and for other reasons stated below which will become apparent to those skilled in the art upon reading and understanding the specification, there is a need in the art for improved systems and methods that enable faster digital processing of video data without the need for increasing the processing power of the computing equipment.
The Embodiments of the present invention provide methods and systems for processing digital video data and will be understood by reading and studying the following specification.
In one embodiment, a digital video processing system is provided. The system comprises one or more optical sensor modules adapted to generate serial digital video signals of captured images and a reconfigurable computer coupled to the one or more optical sensor modules configured to process the serial digital video signal. The reconfigurable computer is further configured to implement a first digital filter adapted to remove from the serial digital video signal data pertaining to one or more of color, sound, and control symbols, and the reconfigurable computer is further configured to implement one or more memories adapted to store the serial digital video signal filtered by the first digital filter, as a memory-mapped image.
In another embodiment, a method for processing of images captured by a satellite in orbit is provided. The method comprises capturing one or more video images with an orbiting optical sensor; generating serial digital video signals of the one or more video images in orbit; filtering one or more of color, sound, and control symbols from the serial digital video signals in orbit; storing the filtered serial digital video signals as a memory mapped image in one or more memories in orbit; and transmitting the memory mapped images to a ground station.
In yet another embodiment, a method for processing video images is provided. The method comprises capturing one or more video images; generating serial digital video signals of the one or more video images; filtering one or more of color, sound, and control symbols from the serial digital video signals; storing the serial digital video signals as one or more memory mapped images in one or more memories; performing one or more image processing operations on the memory mapped images, including one or more of digital filtering, edge detection, image cropping and image magnification; and generating second serial digital video signals of the transmitted memory mapped images by one or more of, restoring control symbols, inserting blank color data samples, inserting blank sound data samples.
The present invention can be more easily understood and further advantages and uses thereof more readily apparent, when considered in view of the description of the preferred embodiments and the following figures in which:
In accordance with common practice, the various described features are not drawn to scale but are drawn to emphasize features relevant to the present invention. Reference characters denote like elements throughout figures and text.
In the following detailed description, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific illustrative embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that logical, mechanical and electrical changes may be made without departing from the scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense.
Embodiments of the present invention address the problem of providing faster digital processing of video data without the need for increasing available computing resources by filtering out unnecessary data samples from the video data stream and storing video image frames as memory mapped images prior to performing digital image processing operations on the data. Not only does the reduced data set reduce memory requirements for storing video data, and bandwidth requirements for transmitting video data, efficiency is gained because the video data retrieved from memory and processed by the computing resources contains only the information needed to perform the desired image processing operation.
Sensor data output by the sensor modules 102 is processed by one or more reconfigurable computers (RC) 104, included in system 100, configured for digital video processing. As shown in
In order increase the rate at which DVP 124 can processes video frames, RC 104 is further configured to filter out unnecessary data from the data signal output from sensor module 102. As illustrated in
While DFA 120 reduces the volume of data samples which must be processed by DVP 124, the removal of control symbols from the data also eliminates structure information required to re-assemble the numerous luminance data samples into a meaningfully accurate representation of the captured image. Accordingly, RC 104 is further configured to store the data in memory in such a way as to preserve the image structure. In one embodiment, memories 126 are memory mapped so that each memory address correlates to a specific pixel position in the image. For example, in one embodiment, video image data for a frame of video comprises lines 1 through L with each line comprising N pixels. In one embodiment, the luminance samples for the N pixels of the first line of the video image are stored in a first row of memory addresses in memories 126. Likewise, luminance samples for the N pixels of the line 2 through L of the video image are stored in rows 2 through L of memory addresses in memories 126. In this way, each pixel in the video image frame is mapped to a specific memory address in memories 126, thus preserving the video image without the need for preserving control symbols from the serial digital video signal. The advantage of memory mapped image frames is that only the data of interest need be preserved in applications where memory resources are limited. When DVP 124 performs the one or more image processing operations on the image frame, efficiency is gained because the data retrieved by DVP 124 from memories 126 contains only the information needed by DVP 124 to perform the operation. Efficiency is also gained when saving the processed image back to memories 126 as a memory mapped image because the data saved by DVP 124 also contains only the data of interest.
In one embodiment, in order to communicate the processed image in memories 126 to one or more external devices, RC 104 is further configured to implement a second digital filter (DFB) 128. DFB 128 employs a filter algorithm that inputs the digital images stored in memories 126 and outputs the images as a serial digital video signal by restoring control symbols and inserting blank (e.g. zero value) data samples for one or more of, but not limited to, color and sound. In one embodiment, DFB 128 converts the images stored in memory mapped memories 126 and outputs an SDI standard signal.
In one embodiment, spacecraft 210 is a satellite, such as a weather satellite used to observe weather patterns occurring on earth or other body. Spacecraft 210 includes an optical sensor module 202 having a optical sensor 203, adapted to capture video imagery of the earth. In one embodiment, optical sensor 203 outputs captured video imagery as an analog signal, such as, but not limited to an NTSC standard analog signal or a PAL standard analog signal. In one embodiment, optical sensor module 202 converts the analog signal into a serial digital video signal, such as, but not limited to an SDI standard signal. In another embodiment, optical sensor 203 includes an array of optical sensors such as, but not limited to, an array of charge coupled device (CCD) sensors, or an array of complimentary metal oxide system (CMOS) sensors, and outputs captured video imagery as a serial digital video signal, such as, but not limited to an SDI standard signal.
In one embodiment, spacecraft 210 further comprises RC 220 configured to perform digital video processing. In one embodiment, RC 220 comprises a reconfigurable computer as described in the '3944 Application herein incorporated by reference. In one embodiment, RC 220 is configured to input the serial digital video signal from optical sensor module 202 through input/output (I/O) interface 214 and store the signal as digital data samples in one or more memories 216. In one embodiment, RC 220 is configured to implement a digital filter, DFA 225. DFA 225 employees a “lossy” compression filter algorithm that removes from the serial digital video signal data pertaining to one or more of, but not limited to, color, sound and control symbols, before the signal data is stored in memories 216. Further, memories 216 are memory mapped so that as DFA 225 stores the filtered signal data into memories 216, the structure of the video image is maintained. For example, in one embodiment, where the serial digital video signal is an SDI standard signal, DFA 225 removes Timing Reference Signal (TRS) data, chrominance (Cb and Cr) data samples, Audio Engineering Society/European Broadcasting Union (AES/EBU) audio channel data samples, and any other information stored in the blanking interval of the SDI video stream. The remaining data samples, comprising only a stream of luminance samples (Y) for each pixel of the captured video image, are stored in Memories 216.
In one embodiment, RC 220 is further configured to implement digital video processor (DVP) 204 to perform digital enhancements to video image frames stored in memories 216. DVP 204 performs one or more image enhancement operations such as, but not limited to, digital filtering, edge detection, image cropping, image magnification, or other image enhancement, and stores the processed images in memories 216. By processing the image in space, individuals on earth can download only the features of the image they are interested in.
In one embodiment spacecraft 210 further comprises a transmitter 230 that is adapted to wirelessly transmit a data stream containing the images captured by optical sensor module 202 and processed by RC 220. After enhancing the images, DVP 204 outputs the images to transmitter 230 via I/O port 215. In one embodiment, the data stream is wirelessly received by receiving station 250, illustrated in
As illustrated in
In one embodiment, monitoring station 450 comprises a NIC 455 and RC 460. NIC 455 receives the filtered video data signal from network 408 and outputs the data to RC 460 via I/O port 466. RC 460 stores the filtered video data in one or more memories 456. As described pertaining to memories 216, memories 456 are memory mapped so that each memory address contains a luminance sample that correlates to a specific pixel of an image.
In one embodiment, RC 460 is configured to implement a digital filter, DFB 465. DFB 465 employees a filter algorithm that inputs digital images stored in memory mapped memories 456 and outputs the images via I/O 467 as a serial digital video signal by restoring control symbols and blank data samples for one or more of, but not limited to, color and sound. In one embodiment, DFB 465 inputs the images stored in memory mapped memories 456 and outputs an SDI standard signal by reconstructing TRS data, and inserting zero value data samples for the Cb and Cr data samples and in the blanking interval of the SDI video stream. In one embodiment, RC 460 is further configured to implement digital video processor (DVP) 464 to perform digital image enhancement on the video image frame stored in memories 456. DVP 464 performs one or more image enhancement operations such as, but not limited to, edge detection, image cropping, image magnification, or other image enhancement, and stores the processed image in memories 456.
In one embodiment, surveillance system 400 includes serial digital video signal converter (SDVS Conv.) 475 adapted to convert serial video data stream into one or more standard analog video signals, such as, but not limited to NTSC signals and PAL signals for display on an output device 490, such as a video monitor. In one embodiment, surveillance system 400 includes video storage device 470 (such as a magnetic tape or magnetic or optical disk) adapted to store the video images carried for later viewing or processing. In one embodiment, surveillance system 400 includes a facial feature recognition device 476, adapted to correlate facial images of individuals captured by one of the plurality of surveillance stations 410 with a database of facial images of persons of interest such as know, but not limited to, known and suspected criminals, fugitives, or terrorists.
Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that any arrangement, which is calculated to achieve the same purpose, may be substituted for the specific embodiment shown. This application is intended to cover any adaptations or variations of the present invention. Therefore, it is manifestly intended that this invention be limited only by the claims and the equivalents thereof.
This application is also related to the following co-pending United States patent application filed on Jul. 23, 2004, which is hereby incorporated herein by reference: U.S. patent application Ser. No. 10/897,888 (attorney docket number H0003944-1628 entitled “3944”) and which is referred to here as the '3944 Application. This application is also related to the following co-pending United States patent application filed on even date herewith, which is hereby incorporated herein by reference: U.S. patent application Ser. No. ______ (attorney docket number H0008827-1628 entitled “Systems and Methods for Semi-Permanent, Non-Precision In-space Assembly of Space Structures, Modules and Spacecraft”) and which is referred to here as the '8827 Application.