Image enhancement including contrast enhancement and detail sharpening techniques are well known in the television and electronic still photography industries. Generally prior art image enhancement techniques electronically process electronic forms of still and moving images, for example such as analog and digital video signals carrying images, digital data representing images and other electronic forms of images. The electronic processing operates to alter characteristics of images such as by increasing contrast of low contrast images, decreasing contrast of high contrast images, sharpening lines and edges and increasing high frequency detail amplitude. Overall these prior art electronic processing of images are intended to improve the visual quality of images to facilitate enjoyment of the image by the viewer.
It is noted that the prior art includes many examples of selectively enhancing images by intentionally applying an unnatural coloring or contrast to the image or its position. One example is given in U.S. Pat. Nos. 6,154,250 and 5,564,698 to Honey, et al. in which the televised image of objects such as a hockey puck is caused to be enhanced when presented, for example by leaving a colored trail of its motion across the ice. Note in one of Honey's embodiment the puck itself is not enhanced, rather a colored line representing its path is electronically added to the image.
The Honey et al. '250 abstract states “A system for enhancing the television presentation of an object at a sporting event includes one or more sensors which are used to determine the location of the object. Based on the location of the object and the field of view of a broadcast camera, a processor determines the position of the object in a video frame of the broadcast camera. Once knowing where the object is positioned within the video frame, the television signal can be edited or augmented to enhance the presentation of the object.” While the Honey et al. system works, it requires a considerable amount of specialized equipment, such as the modified hockey puck and specialized position sensors. Also, the Honey et al. system does not locate the hockey puck within the program video image itself, but rather locates the puck utilizing the specialized puck and specialized position sensors and then using that information to locate the puck in the program video frame, enhances the presentation of the puck.
Another example of unnatural prior art image enhancement is in ubiquitous highlighting of important text images by electronically underlining, changing color of the text or coloring the background with a yellow (or other color) highlighting. Yet another example of unnatural prior art enhancement is the drawing of arrows, circles, lines or the like on television scenes such as by a weather reporter who electronically marks important weather images to call the viewer's attention thereto. These prior art techniques share a common attribute of highlighting—an unmoving image or relatively slowly moving images.
Thus, there exists a need in the art for a system and method for improved image enhancement. As will be seen, the invention provides such a system and method in an elegant manner.
The invention is directed to a system and method of providing enhancement of an object in a video frame by first recognizing and locating an object within an image frame and performing an enhancement process on the object. The enhancement process may include performing image or object enhancement on the object to make the object easier for an observer to detect. The video recognition techniques include recognizing an object by identifying characteristics of the object using video recognition techniques. The video recognition techniques may include identifying size, color, surface texture, brightness, histograms, shape or other characteristics.
Various embodiments of a system and method are described below that are provided for enhancement of an object in a video frame. The method includes recognizing and locating an object within an image frame and performing an automatic enhancement process on the object. The automatic enhancement process may include performing image and/or object enhancement on the object to make the object easier for an observer to detect. Such object enhancement may include enhancement to part of the object, such as the perimeter of the object, or to the entire object.
Object enhancement will be understood to include operations which help make an object more pleasing to view by hiding, covering, disguising or otherwise reducing the visibility of blemishes and flaws on objects in electronic photographs or videos. Examples of such enhancement include whitening teeth, adding a tan to skin, softening wrinkles, removing blemishes or hiding beard shadows on faces. Object enhancement may also include varying enhancement, such as causing enhancing features to blink, flash or otherwise change to catch a viewer's attention, making the object easier to detect.
In one method, providing enhancement of an object in a video frame may include first receiving video data, then searching for an object within an image frame. Then, an object is recognized and located within a frame, and an automatic enhancement process is performed on the object.
In another embodiment, after receiving video data, the process then determines whether object enhancement is activated. If object enhancement is activated, the process performs searching for an object within an image frame. Then, an object is recognized and located within a frame, and an automatic enhancement process is performed on the object.
In yet another embodiment, after receiving video data, an image frame is searched for an object where enhancement is desired. If an object is found, it is determined whether object enhancement is activated. It is also determined whether temporal enhancement is activated. These past two steps may be performed separately, together or in different order. If object enhancement is activated and temporal enhancement is not activated, the process performs an automatic enhancement process on the object. If object enhancement is activated and temporal enhancement is activated, then the process performs an automatic enhancement process on the object for a set duration.
In yet another embodiment after receiving video data and searching for an object within an image frame, it is determined whether object enhancement is activated. If object enhancement is activated, it is then determined whether temporal enhancement is activated. If object enhancement is activated and temporal enhancement is not activated, then an automatic enhancement process is performed on the object. If object enhancement is activated and temporal enhancement is activated, the process performs an automatic enhancement process on the object according to a timing algorithm.
In either of these methods, a timing algorithm may be configured to cause the enhancement of the object to blink, flash or to be enhanced during a predetermined time periods to cause the enhancement of the object to appear non-continuous to the user. The received video data may be one or more frames of video data, or multiple, a plurality, of frames.
As used herein, image enhancement is meant to have a narrower meaning than that generally used in the art. Image enhancement as used hereinafter in the specification and claims means electronic processing of images, carried or stored in electronic form, to improve the natural appearance and visibility thereof to facilitate visibility and enjoyment of the image when it is ultimately viewed by a viewer.
As used herein, object enhancement is meant to have a narrower meaning than that generally used in the art. Object enhancement as used hereinafter in the specification and claims means electronic processing of images, carried or stored in electronic form, to improve the visibility of one or more object within an image. This may be done by giving the object a natural, yet improved appearance so that it is draws a viewer's attention more or so that it is more easily detectable to a user that may wish to follow the object more easily as it travels across or about a video presentation. This may also be done by giving an object an unnatural appearance, where an enhanced object is more obvious to a user, more clear or more overtly highlighted.
These definitions of image enhancement and object enhancement will find particular applicability to particular objects in televised images as will be described in more detail below. It may be noted that image enhancement and object enhancement as defined herein are not mutually exclusive, and can be applied to a television image simultaneously. For example a ball may be image enhanced to make it appear sharper and also object enhanced by circling it with a red border. Either enhancement may include a temporal aspect, for example by flashing the enhancement or by providing enhancement for only a portion of the time a ball is visible.
Unfortunately many conventional image enhancement techniques known in the prior art must be manually or automatically adjusted in order not to over enhance images. Such over enhancement can lead to objectionable visual artifacts, for example such as ringing on edges, and contrast or detail taking on an artificial or unnatural look. The level of desired enhancement varies depending on the particular scene and may change during changes in the scene such as camera movement, lighting changes or movement of objects in the scene. According to the invention, objects can be specifically recognized and targeted, allowing for consistent and predictable enhancement that is more enjoyable to a viewer.
Of particular interest in respect to the invention is the use of image enhancement for televised events where a ball, puck, birdie, fowl, fish, animal or other object that moves quickly in its physical environment. Such an object moves quickly within the television image frame (which may be a field in interlaced display systems). Such an image may appear relatively small in the television image frame. Thus, in either case, it is difficult for a viewer to see the object, which can be frustrating to watch. It will be recognized that the invention is useful in many types of still and motion imaged sporting events and the like. However, for purposes of illustrating and describing one embodiment of the invention, television coverage of the game of golf will be utilized as an example.
In televising golf, it is desired to have continuous camera images of the golf shots starting with the ball on the tee, through the swing and hit, as it is flying down the fairway, as it lands through when it rolls to a stop. Unfortunately for a significant period of this travel, the ball moves very quickly in its physical environment and despite the skills of the cameraman, the ball moves within the television frame and appears very small in the television image frame for much of the shot. In such instances, image enhancement (to enhance the natural visibility of the ball) is desirable in order provide better visibility to the viewer. Additionally object enhancement is also desirable, for example such as by changing the color of the ball to further improve the visibility to the viewer. Unfortunately, Honey's system is unsuitable in this application due to the difficulty in installing emitters in the golf ball and the large number of location sensors which would be required to cover an entire golf course.
In order to provide either image enhancement or object enhancement of the golf ball (or both) the invention utilizes object recognition to locate a desired object within the image frame (or frames) and, once the object is located, image enhancement and/or object enhancement is performed on the object. In the example of a televised golf game, characteristics of the golf ball for example such as its size, color, surface texture, brightness, histograms or shape are utilized to recognize and locate the golf ball in each image frame of the televised images. When located, the golf ball is then image enhanced or object enhanced or both.
In
In the preferred embodiment it is desired that the image and object enhancement circuit 110 include adjustable parameters 114 to allow the type and amount of image and/or object enhancement to be utilized to the identified objects (golf balls). The parameters may be automatically established by 110 in response to the image, 104 or may be established by a user or a combination of user and automatically established parameters may be utilized.
In one embodiment, the video data input 104 can be coupled to adjustable parameters circuit 114, where the input video data is received by circuit 114 via enhancement circuit 110, by two way connection 111 as shown. In this embodiment, the special parameters are established by adjustable parameters circuit 114 instead of enhancement circuit 110 in response to the input video data 104. Alternatively, connection 111 may be a one-way connection from the adjustable parameters circuit 114 to enhancement circuit 110. In such an embodiment, the special parameters may be established in enhancement circuit 110 using the parameters of adjustable parameters circuit 114 that would send the adjustable parameters to enhancement circuit 110.
It is also desired that the object recognition circuit 106 may include parameters 112 to allow the characteristics of the object which are to be utilized in its location. The parameters may be automatically established by 112 in response to the image 104 via the two way connection 113, or may be established by a user or a combination of user and automatically established parameters may be utilized. Automatic updating or modification of the parameters based on successful location of objects in the image frame is also desirable.
Object recognition circuit and adjustable parameters 112 may be implemented by any well know object recognition circuit or software. For example, numerous inventions found in class 382/170 will find application for the object recognition circuit. U.S. Pat. No. 6,952,496 entitled “Object Recognition System and process for Identifying People and Objects in an Image of a Scene” is one example of object recognition with automatic updating of image parameters which may be utilized.
Image and object enhancement circuit 110 may be implemented by any well known video image processing circuitry or software. In the preferred embodiment image and object enhancement circuit of
Enhance circuit 118 with parameters 114 is preferred to be a video graphics device with capability to sharpen detail and highlight objects by changing background colors according to user inputs of parameters 114. Manufacturers including Thomson Grass Valley of Grass Valley, Calif., Snell and Wilcox of Santa Clara, Calif. and Sony Broadcast of San Jose, Calif. provide many versions of such products. By incorporating a down stream keyer in the video graphics device the capability of video selector 120 may be incorporated.
Referring to
Referring to
Referring to
Referring to
As will be understood by those skilled in the art, many different algorithms are possible to cause the enhancement of the object to flash or blink. The algorithm illustrated is one embodiment, where the enhancement is activated over different time periods. According to the illustrated embodiment, the time periods are from a to b, then c to d, then e to f, and so on. During these time periods, the enhancement can be activated, and the resulting view to a user is a blinking or flashing enhancement of the object. For example, a golf ball traveling across a video screen is very difficult to see in real time after a golfer hits the ball. According to the invention, the ball can be enhanced to improve the view.
However, even an enhanced object may not be visible enough to a user to detect it on a video screen, such as a golf ball traveling at a high speed in real time and from a distance. According to the embodiment illustrated in
In either of these embodiments, the various components or process steps may be rearranged or interchanged depending on a particular application, and those skilled in the art will understand that such changes can be made without departing from the spirit and scope of the invention, which is defined by the appended claims and their equivalents.
The invention may also involve a number of functions to be performed by a computer processor, such as a microprocessor. The microprocessor may be a specialized or dedicated microprocessor that is configured to perform particular tasks by executing machine-readable software code that defines the particular tasks. The microprocessor may also be configured to operate and communicate with other devices such as direct memory access modules, memory storage devices, Internet related hardware, and other devices that relate to the transmission of data in accordance with the invention.
The software code may be configured using software formats such as Java, C++, XML (Extensible Mark-up Language) and other languages that may be used to define functions that relate to operations of devices required to carry out the functional operations related to the invention. The code may be written in different forms and styles, many of which are known to those skilled in the art. Different code formats, code configurations, styles and forms of software programs and other means of configuring code to define the operations of a microprocessor in accordance with the invention will not depart from the spirit and scope of the invention.
Within the different types of computers, such as computer servers, that utilize the invention, there exist different types of memory devices for storing and retrieving information while performing functions according to the invention. Cache memory devices are often included in such computers for use by the central processing unit as a convenient storage location for information that is frequently stored and retrieved. Similarly, a persistent memory is also frequently used with such computers for maintaining information that is frequently retrieved by a central processing unit, but that is not often altered within the persistent memory, unlike the cache memory.
Main memory is also usually included for storing and retrieving larger amounts of information such as data and software applications configured to perform functions according to the invention when executed by the central processing unit. These memory devices may be configured as random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, and other memory storage devices that may be accessed by a central processing unit to store and retrieve information. The invention is not limited to any particular type of memory device, or any commonly used protocol for storing and retrieving information to and from these memory devices respectively.
The apparatus and method include a system and method for enhancing an object within a video image. Although this embodiment is described and illustrated in the context of devices, systems and related methods of enhancing images within a video presentation of a golf game, the scope of the invention extends to other applications where such functions described herein are useful. Furthermore, while the foregoing description has been with reference to particular embodiments of the invention, it will be appreciated that these are only illustrative of the invention and that changes may be made to those embodiments without departing from the principles of the invention.
This application is a continuation of application Ser. No. 11/514,075 filed Aug. 30, 2006 and issued as U.S. Patent ______ on ______, the continents of which are incorporated herein in their entirety by reference.
Number | Date | Country | |
---|---|---|---|
Parent | 11514075 | Aug 2006 | US |
Child | 15072882 | US |