Patent Application
20130216135
Embodiments relate generally to visual search systems and, more specifically to visual search system architectures that operate within the bandwidth restrictions of various components contained in the system and in a variety of different system applications, such as object detection and navigation in automotive systems and object detection and recognition in surveillance systems.
The widespread use of mobile devices equipped with high resolution cameras is increasingly pushing computer vision applications within mobile scenarios. A common paradigm is represented by a user taking a picture of the surroundings with a mobile device to obtain informative feedback on its content. This is the case, e.g., in mobile shopping, where user can shop just by taking pictures of products, or landmark recognition for ease of visiting places of interest. As in the aforementioned scenarios visual search needs to be typically performed over a large image database, applications communicate wirelessly with a remote server to send visual information and receive the informative feedback. As a result, a constraint is set forth by the bandwidth of the communication channel, whose use ought to be carefully optimized to bound communication costs and network latency. For this reason, compact but informative image representation typically in the form of a set of local feature descriptors (e.g. SIFT, SURF, etc.) extracted from the image are generated and must be communicated and processed. Improved architectures for implementing visual search systems are needed for various applications, such as object detection and navigation in automotive systems and object detection and recognition in surveillance systems.
An embodiment of a visual search system includes at least one imaging device, each imaging device operable to capture a corresponding image, a feature extraction device coupled to each imaging device, each feature extraction device operable to generate feature descriptors from the image received from the corresponding imaging device. A descriptor encoding device is coupled to each feature extraction device and operable to generate compressed feature descriptors from the received feature descriptors. An application processor is coupled to each descriptor encoding device. The application processor is operable to process the received compressed feature descriptors to generate output information as a function of the processed compressed feature descriptors.
Body design of modern and future vehicles will challenge the driver who has to deal with reduced direct visibility. For example, when he/she is driving the car at low speed to park it, he/she should be able to visually detect what objects or people are around the vehicle, evaluate the relative distance since by maneuvering the car might collide with against them, resulting in unacceptable injuries and/or costly damages.
One efficient solution to this issue is to install multiple “satellite” cameras (usually 4 attached to a powerful application processor, e.g. see [2]), on the car periphery to cover 360 degree space around it. The cameras send their video signals to a central Electronic Control Unit (ECU) that generates an aggregated 360° view displayed on the dashboard screen, thus contributing to the driver's situational awareness. Such applications have been on the market since 2009.
On top of the 360° surround image on the screen, machine vision assisted detection of obstacles, associated with warnings to the driver of potential dangers during low-speed maneuvers can further increase the driver's situational awareness and reduce the risk of collisions. Automotive manufactures target this enhancement on new models of vehicles to be put on the market in the coming years (starting since 2013) by incrementally lowering the cost of the initial solutions and increasing the features available to cover an increasing number of use cases.
In order to minimize the system cost and thus the number of cameras to cover the whole 360° space and, due to the constraints of their physical placement, very wide lenses have to be used on each of the four cameras, typically above 180° of effective horizontal Angle of View (AOV). For practical reasons, such lenses use near equidistant projection mapping and are called, due to their appearance, “fisheye lenses”. To the opposite of rectilinear lenses, the images projected by fisheye lenses on the image plane appear “stretched” in the center and compressed toward the border of the projected circle. A solution could be to un distort the fisheye images to fall back into the rectilinear case but this solution will affects and degrades the image gradient distribution as captured by the sensor and it would require too costly HW and frame buffering in the camera.
Presently, landmarks and buildings [3] taken with rectilinear lenses only are representing the largest number of images into compact descriptors for visual search (CDVS) datasets which are the closest in term of “objects captured on the road” to automotive related use cases. Unfortunately no fish eye images have been used during the development of CDVS studies.
Consequently, in order to consider a potential application of the CDVS standard into the automotive context, the fact of using the fish eye optics has to be considered and taken into account for upcoming core experiments.
The satellite camera connection with the central ECU can be wired or wireless. With OEM car makers, wired connection is clearly preferred, while wireless remains interesting for potential after-market solutions. First wired systems were using analog video transmission (CVBS) over coaxial or Shielded Twisted Pair (STP) cabling, associated with CAN cable for camera control. In order to improve the overall quality of the partial video signals received by the central ECU, as well as for the reason of cabling simplification, proprietary digital serial link semiconductor products (a pair of serializer and deserializer chips required per camera position) have enabled first digital camera solutions, while removing the resolution limitation previously imposed by NTSC or PAL analog video standards. They provide enough bandwidth to transmit uncompressed video as well as camera control in the same STP cable. However, there is no standard on the market.
The race for cost efficiency as well as car architecture simplification bringing reliability benefits motivates the automotive industry to look for cheaper and simpler solutions. One of important contributors to the system cost is the cabling, another one is the multiplication of dedicated systems. In the past few years, a new trend of larger adoption of Ethernet technologies in vehicles is observed. Physical layer automotive grade-qualified semiconductor products allowing the use of cheap Unshielded Twisted Pair (UTP) cabling have become available. The use of UTP cabling brings further cost benefits as it can be integrated into the cable harnesses and take advantage of cheap multi-pole connectors, as opposite to expensive separate connectors for shielded or coaxial cables. However, to benefit from the UTP cabling, the electrical content transported must intrinsically ensure the electro-magnetic compatibility through low emissions and low susceptibility to being disturbed by inducted parasitic signals. This is strongly related to the frequency that determines data bandwidth of the link. The current solution of physical layer compatible with Ethernet protocols and complying with EMC standards while using UTP cabling is limited to 100 Mbps. To further reduce the overall system cost, the Ethernet network installed in the car is intended to be shared by several subsystems, such as 360° cameras, infotainment, diagnostics, etc. The limited data throughput that at later point may even need to be shared in between several sub-systems will further increase the requirement for the large data sources such as cameras, to compress their output data.
Video data compression such as MJPEG or H.264 is lossy, hence reducing the video quality received at the ECU end. While that quality decrease may not be noticeable to human perception in the displayed images, it may substantially affect the operation of feature extraction algorithms that may require high quality video not affected by encode/decode process causing loss of information, artifacts and alteration of gradient distributions. For this reason, it is recommended that, those stages or processes of video analytics requiring high quality video, such as interest point detection and compact descriptor generation, be implemented on the video being acquired by the image sensor prior to any H264 or HEVC encoding The data stream produced by these stages should be standardized in term of structure and transmission such that the upper stages of the video analytics residing in the central ECU can receive and correctly interpret it. Also, to be future-proof, the amount of the compact data should be kept low enough to allow the system operation over 100 Mbps link shared by several sources, which in practice may allow as little as about 10 Mbps per camera. Any means adopted to reach this goal further need to be flexible such that lower compressions hence higher performances can be reached if more data throughput is available. Ideally, the camera intrinsic properties such as optics used should create as few dependencies to central ECU as possible. It would be reasonable to require that, the stages of video analytics eventually implemented at camera level, do not create any such dependencies. E.g., if interest point detection, its descriptor generation and compression shall reside in the camera then the correct operation of these processes should be ensured by the camera itself whatever optics, sensor or other components of the camera may be, and without the need for the ECU to intervene at this level.
This new scenario suggests that existing requirements defined by CDVS have to be complemented with new ones suitable for embedded implementation in order to enable its future application to the automotive context.
In surveillance embodiments a standalone fixed position camera watches its surrounding environment. A battery powered camera due to power consumption constraints cannot send the video so that it has to integrate basic capabilities to extract interest points, their descriptions and their compaction.
Camera connectivity can be, for example, either Ethernet, WiFi or Zigbee (IEEE 802.15.4). In most cases the bandwidth limitation is paired with possible discontinuity in service (variable Quality of Service—QoS) and/or different imposed levels of quality of the sent information to reduce latencies (variable Quality of Experience—QoE).
The security market is split into several fragments, most of them using industry standard technologies for camera video transmission and control.
Consumer or “Prosumer” IP cameras complete the traditional expensive Video Surveillance camera market that itself undergoes a transition from analog cameras (CCTV) via coaxial cabling to digital (IP) cameras via STP cabling, accompanied with a very strong cost and price decrease. As for the new fast-growing market of small and cheap consumer/prosumer IP cameras, they use Ethernet, power line or wireless transmission following consumer electronics and IT industry standards such as 100/1000Base-Tx, BPL (Broadband over Power Line), WiFi a/b/g/n.
Some market sub-segments such as Intruder Alarm Systems (IAS) adopt non-standard or less popular standard wireless technologies as there are strong battery life constraints.
A large portion of the consumer/prosumer wireless cameras are also battery-powered for the sake of ease of installation or/and mobility, but their autonomy does not need to last for years as opposed to IAS detectors.
There are two common denominators to the large variety of security-related camera systems—ever-increasing use of Internet and need of storage for the video footage. These set specific constraints in term of data rate of the transmitted video that needs to pass through the bottleneck of the transport chain, usually situated somewhere in the Internet path, as well as the amount of data to store which determines the amount and cost of storage to install.
In general practice, the video needs to be compressed and further intelligence is required to generate alerts, trigger alarms or/and select relevant video to store, which reduces the amount of the required storage, the amount of video to transfer and provides means for speeding up the video search based on computer generated metadata.
Due to the request of strong video compression, the quality of the video received at central equipment level, whether local or distant, is reduced, which potentially creates the need of integration of video quality sensitive stages of video analytics in the camera end. For wired cameras that can stream continuously, the remaining part of the video analytics can reside in central equipment. Definition of a standard way of interfacing between the analytics stages in the camera and the central equipment is of high interest here.
In other cases, such as battery-powered surveillance cameras or camera-fitted IAS detectors, a higher degree of video analytics is required to reside directly in the camera in order to detect relevant events and only stream relevant video, generate relevant alerts or/and trigger relevant alarms. These devices may need different standards but will not benefit from a standard targeting the split of analytics stages between camera and central equipment based on the criterion of video quality.
In this case description stage is still needed at camera side but not the compression one, similarly as shown in
Consequently there is the need to decouple the description stage from the compression one, suggesting the need for a separability requirement.
As previously explained the strong video compression request implies that the wired camera sends Compact descriptors to the central equipment and then the remote system does some visual search to e.g. recognize objects and generate alerts, as shown on
In an example a building is captured with both fish eye lens (
In relation to descriptor extraction, among the many descriptors that have been proposed, the most common is SIFT. This approach, described by Lowe et al. [1] provides two distinct main phases: keypoints detection: significant image points that are invariant to scale and rotation; descriptor extraction: in the neighborhood of each keypoint information about the gradient is recorded.
Regarding scale-space extrema detection, in order to search all potential keypoints, first you have to look for points that are invariant under changes of scale. This is done by searching extrema in the Difference-of-Gaussian function convolved with the image.
For keypoint localization, each point found in the previous phase, they determine the interpolated location of each extrema with a method based on Taylor expansion. All extrema with a value of |D(x)| less than a threshold is discarded, because they are extrema with low contrast. The difference-of-Gaussian will have a strong response along edges, even if the location along the edge is poorly determined. So the keypoints with a ratio between principal curvature less than a threshold are discarded. This is done in a very efficient way using the Hessian matrix.
Regarding orientation assignment, in order to achieve invariance to image rotation, they assign a consistent orientation to each keypoint, so it can be represented relative to this orientation. The orientation is assigned selecting the highest peak in the histogram of gradient, calculated within a region around the keypoint. For locations with multiple peaks of similar magnitude, there will be multiple keypoints with different orientation.
In relation to keypoint descriptors, after keypoint detection and orientation assignment, descriptors can be extracted. These are obtained by sampling magnitude and orientation of the gradient in a neighborhood of the keypoint. In order to achieve orientation invariance coordinates and gradient orientation are rotated relative to the keypoint orientation. To avoid sudden changes in the descriptor with small changes in the position of the window, a Gaussian weighting function is used to assign a weight to the magnitude of each sample point, depending on the distance from the keypoint. The descriptor consists of a set of histograms of orientation calculated over regions of 4×4 with 8 orientation bins. Finally, the vector representing the descriptor is normalized to be invariant to changes in lighting.
For descriptor matching, the best candidate match for each keypoint is its nearest neighbor in the database. The nearest neighbor is defined as the keypoint with minimum Euclidean distance. To discard feature that do not have any good match to the database there is a threshold on the ratio of distance from the first and second closest neighbor.
For fixed length encoding, thresholding: in this phase we set to 0 each value that are less than a threshold (e.g. 0.01). This has a very little impact on descriptor performance
For run length encoding, the analysis of the SIFT descriptors to find out the descriptors' values distribution, and in particular the number of values less than a threshold. The table below shows the results.
As we can see the percentage of zero values inside the SIFT descriptor is such that make applicable a run length encoding. Run-length encoding (RLE) is a very simple form of data compression in which runs of 0 data (that is, sequences in which the 0 data value occurs in many consecutive data elements) are stored as a single data value and count, rather than as the original run.
For example, consider a vector containing many zero values and number different than 0. There will be many long runs of 0, and many short runs values different than zero. Let us take a hypothetical single vector, with B representing the value different than 0 and many 0:
If we apply the run-length encoding (RLE) data compression algorithm to the above vector, we get the following:
This is to be interpreted as twelve 0s, one B, twelve 0s, three Bs, etc. The run-length code represents the original 67 numbers in only 18.
For variable length encoding, N-bit Quantization: Let P={p1, p2, . . . pn} be the descriptor after the run length, and Q={q1, q2, . . . qn} the quantized descriptor. For each value pi we compute the corresponding qi as follow:
q
i=└(26−1)·pi+½┘ 1.
In this example N=6.
For Arithmetic Coding, now there is a descriptor that is a vector of n-bit integer, but more than 70% of these values are equal to 0. We can use this fact to make a lossless compression using an adaptive Arithmetic Coding algorithm.
Regarding descriptor decoding, the inverse arithmetic decoding is applied. For matching in the compressed domain when we use quantization for compressing SIFT descriptor, it is possible to do descriptor matching in compact domain. If we also apply Arithmetic Coding we need to decode the compact descriptor to obtain the quantized version.
The only difference between each values of the quantized descriptor and the decoded descriptor is a scale factor. In fact
p′
i=½b·qi
where p′i is the decoded value, qi is the quantized value and b is the number of quantization bit.
For this reason if the distance is computed between two descriptors in the quantized domain we get a result that is 2b times the distance between the same descriptors in the decoded domain.
When, during the descriptor matching, the ratio between the distances of the first and second nearest neighbor is computed we get the same result both in quantized and decoded domain.
One skilled in the art will understand that even though various embodiments and advantages of the present invention have been set forth in the foregoing description, the above disclosure is illustrative only, and changes may be made in detail, and yet remain within the broad principles of the invention. For example, many of the components described above may be implemented using either digital or analog circuitry, or a combination of both, and also, where appropriate, may be realized through software executing on suitable processing circuitry. It should also be noted that the functions performed can be combined to be performed by fewer elements or process steps depending upon the actual embodiment being used in the system 100 of
This application claims priority to U.S. Provisional Patent Application Nos. 61/596,149, 61/596,111, and 61/596,142 all filed on Feb. 7, 2012 and each of which is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 61596142 | Feb 2012 | US | |
| 61596111 | Feb 2012 | US | |
| 61596149 | Feb 2012 | US |
