Patent Grant
11174665
The present disclosure relates to a camera for use in automotive applications including but not limited to rear-view cameras, and more particularly to methods and apparatus for opening and closing a liftgate based on camera based gesture recognition.
A method and system for a camera based hands free power liftgate is described herein. The system may power on following a key approach and/or the doors unlocking. The system may then initialize. The approach detection feature of the system may then determine whether an approach was detected. If there is an approach detected, the lights may be activated and the gesture detection feature of the system may then determine whether a gesture was detected. If a gesture was detected, the liftgate may then be activated. If no gesture was detected, the lights may be activated.
Under an approach, the system may power and initialize. The system may then determine whether an approach is detected. When an approach is detected, lights of the vehicle may activate and a determination whether a gesture is detected is made. On a condition that a gesture is detected, a liftgate of the vehicle may be activated.
There has been a proliferation of camera technology in consumer devices such as cellular phones and computing devices. Automobile manufacturers have increasingly included cameras as a standard or optional feature for benefits including but not limited to enhancing vehicle safety, improving vehicle maneuverability, and providing convenience to the vehicle driver. Due to a rear-view camera's ability to prevent back over accidents and thereby reduce fatalities and serious injuries caused by those accidents, the National Highway Traffic Safety Administration (NHTSA) will require all new vehicles under 10,000 pounds to include rear visibility technology by May 2018. Accordingly, there is a need for cost effective designs for rear visibility technology in order to comply with NHTSA regulations.
The embodiments described herein are generally directed to a system and method of activating a vehicle liftgate. Specifically, the embodiments are directed to a gesture method, which may include a control gesture that is a non-contact “press” of the rear view camera using a hand, carried object, or other body part. The embodiments described herein include features including but not limited to approach monitoring, cross traffic detection, flare/AEC detection, gesture recognition, gesture timeout, initialization control, and a user feedback system. The embodiments described herein include tunable parameters including but not limited to approach sensitivity, approach time, approach region of interest, gesture sensitivity, gesture time, gesture region of interest, initialization sensitivity, initialization time, process sensitivity, process learning rate, and process unlearning rate. Example tunable parameter values are shown in Table 1 below:
The system provides improvements to other gesture based systems including but not limited to avoiding flaring/AEC changes in specific lighting conditions that may cause lockout and potential false triggers; gesture control methods which are flare/AEC independent; improved camera performance in dynamic lighting situations (including both high- and low-light); providing access to camera settings to enable image stability; and reducing both patterned and random noise within the camera, especially in low-light conditions in order to improve image noise limit sensitivity.
In the embodiment set forth in the diagrams and described hereinafter, the vehicle owner key fob 102 may activate the vehicle bus 106 of the vehicle system 104 when the key fob 102 is in proximity to the vehicle, but that is just by way of example. The camera system 108 may activate based on a message signal from the vehicle bus 106 and may go into an “awaiting intent” mode. A vehicle user 114 may enter an activation zone 116 for 0.5 seconds to signal intent. The camera system 108 detects the approach of the user 114 in the activation zone 116 and confirms intent of the user 114. With intent confirmed, the user is provided a signal to initiate a gesture. In the present embodiment, the lights 110 of the vehicle system 104 may then flash, signaling that the vehicle system 104 is ready for gesture, but that is just by way of example.
The user 114 may then initiate a gesture such as, for example, a kick gesture. The power liftgate warning chime may then indicate a successful gesture, and the liftgate 112 may then open. The system 100 may then enter a standby and ready-to-close state. The user 114 may then deposit and/or remove items from a cargo area of the vehicle. The user 114 may then re-enter the activation zone 116 for 0.5 seconds to signal intent. The lights 110 of the vehicle system 104 may then flash, signaling the vehicle system 104 is ready for gesture. The user 114 may then initiate a gesture such as for example a kick gesture. The power liftgate warning chime may then indicate a successful gesture. The liftgate 112 may then close, and the camera system 108 may then go back to sleep mode. In one embodiment, the liftgate 112 may close if the key fob 102 is no longer in proximity to the vehicle for a specified period of time while the liftgate 112 is open.
If a valid gesture is not confirmed within a specified timeout period, a signal is sent to confirm timeout. In one embodiment, detection of the user 114 within the activation zone after a cool-down period may reinitializes intent by the vehicle system 104. In another embodiment, the vehicle system 104 will not reinitialize without the key fob 102 re-entering the activation zone 116. The key fob 102 leaving the activation zone 116 may end determination of intent by the vehicle system 104.
The approach detection feature 202 therefore determines whether an approach by the vehicle user is detected (S203). If an approach is not detected, the approach detection feature 202 continues to detect for an approach. Once an approach is detected, the gesture detection feature 204 of the system may subsequently determine whether a gesture is detected (S204), using various methods of gesture detection. If a gesture is detected, the vehicle system may then open the liftgate (S205). If a gesture is not detected, the system proceeds back to initialization (S206), and the process is repeated until a gesture is detected.
The system may then initialize (S302). The approach detection feature 302 then begins to detect approaches by a user (S303). If an approach is not detected, the approach detection feature 302 continues to detect for an approach. Once an approach is detected, the vehicle system may activate the lights (S304). The gesture detection feature 304 of the system may then determine whether a gesture by the user is detected (S305). If a gesture is detected, the vehicle system may activate the liftgate (S306). If no gesture was detected, the vehicle system may activate the lights (S307) and the system may be initialized (S302) and the process (steps S302-S305) may be repeated until a gesture is detected and the vehicle system activates the liftgate (S306).
The vehicle system may then initialize (S402). In the present embodiment as described hereinafter, the initialization may include obtaining parameters from an initialization file 402, but that is just by way of example. The system may then read an image by passing the image from a video buffer for processing (S403), which may include activating, controlling, and communicating with an image sensor 403. The system may then check a latch status based on the read image (S404). The latch status is configured to determine whether the liftgate is open when the latch is open or closed when the latch is closed.
If the latch is open, the system may reset auto exposure of the image, which may include resetting camera parameters including memory and enabling auto exposure to optimize the detected scene (S405), and subsequently reading the image again (S403). If the latch is closed, the system may then turn off auto exposure (S406), which may include disabling dynamic camera settings in order to minimize variability in the detected scene and fix the lighting environment around the camera. The system may then smooth the image to minimize noise/artifacts in the image (S407), and may subsequently subtract background and threshold to determine the moving features within the image (S408). The system may then determine feature sizes within a center region of the image (S409), determine the feature size within a left region of the image (S410), and then determine the feature size within a right region of the image (S411).
The system may then determine whether an approach has been detected (S412). In one embodiment, the system may determine an approach by subtracting the background of the read image and sampling and applying the feature sizes within the center, left, and right regions, but that is just by way of example. The system may use additional methods known in the art to determine whether an approach is detected. If an approach is detected, the system may then update gesture features (S413).
If an approach is not detected, the system may update detected features (S414), including updating background scores of the image by applying scene learning methods such as confidence in the learned scene (S415), updating the background by refreshing components out of confidence (S416); and subsequently limiting background scores by applying bounds to scene confidence (S417). The system may then determine whether initialization has occurred and whether the system successfully learned the scene and has checked the background (S418). If initialization has not occurred, the system may begin the process again at step S403 by reading an image. If initialization has occurred, then the system may determine whether an approach by an individual occurred by checking to see if the features match an approaching target (S419). If the approach is not detected (i.e. a false reading), the system may begin the process again at step S403 by reading an image. If the approach is detected, the system may update the gesture features in order to execute an additional response check (S413).
Once the system updates the gesture features, the system may continue checking the response to determine whether an individual is still interacting with the vehicle or whether the detected approach has timed out (S420). If the approach has timed out, the system may enter a safe mode and reset the auto exposure (S405). If the approach has not timed out, the system may then detect a gesture by checking for features matching a desired gesture (S421). If the features do not match a desired gesture, then the system may begin the process again at step S403 by reading a new image. If the features do match a desired gesture, then the system may activate and trigger the liftgate/portal (S422) and continue to read images.
As described herein, the methods described herein are not limited to any particular element(s) that perform(s) any particular function(s) and some steps of the methods presented need not necessarily occur in the order shown. For example, in some cases two or more method steps may occur in a different order or simultaneously. In addition, some steps of the described methods may be optional (even if not explicitly stated to be optional) and, therefore, may be omitted. These and other variations of the methods disclosed herein will be readily apparent, especially in view of the description of the systems described herein, and are considered to be within the full scope of the invention.
Having thus described the presently preferred embodiments in detail, it is to be appreciated and will be apparent to those skilled in the art that many physical changes, only a few of which are exemplified in the detailed description of the invention, could be made without altering the inventive concepts and principles embodied therein. It is also to be appreciated that numerous embodiments incorporating only part of the preferred embodiment are possible which do not alter, with respect to those parts, the inventive concepts and principles embodied therein. The present embodiments and optional configurations are therefore to be considered in all respects as exemplary and/or illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all alternate embodiments and changes to this embodiment which come within the meaning and range of equivalency of said claims are therefore to be embraced therein.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6142551 | Ciavaglia | Nov 2000 | A |
| 9470033 | Dudar | Oct 2016 | B1 |
| 10329829 | Matsui et al. | Jun 2019 | B2 |
| 20040046409 | Hirai | Mar 2004 | A1 |
| 20150127193 | Tofilescu | May 2015 | A1 |
| 20160083995 | Dezorzi | Mar 2016 | A1 |
| 20160281410 | Schanz et al. | Sep 2016 | A1 |
| 20160323484 | Moenig | Nov 2016 | A1 |
| 20170009509 | Ryu et al. | Jan 2017 | A1 |
| 20170190314 | Ette | Jul 2017 | A1 |
| 20180236929 | Gocke | Aug 2018 | A1 |
| 20190031146 | Etonye | Jan 2019 | A1 |
| 20190122056 | Tran | Apr 2019 | A1 |
| 20190217778 | Swain | Jul 2019 | A1 |
| 20190256049 | Jawany | Aug 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 2014089696 | Jun 2014 | WO |
| 2016088359 | Jun 2016 | WO |
| Entry |
|---|
| Office of Vehicle Safety Compliance, U.S. Department of Transportation National Highway Traffic Safety Administration Laboratory Test Procedure for FMVSS 111 Rear Visibility, Feb. 8, 2018, Washington, DC, USA. |
| Number | Date | Country | |
|---|---|---|---|
| 20190330908 A1 | Oct 2019 | US |
