Patent Application
20240166153
This application claims priority to EP App. No. 22 208 272 filed Nov. 18, 2022, the entire disclosure of which is incorporated by reference.
The present disclosure relates to a system for interior sensing in a vehicle and associated method, and more particularly to monitoring of an occupant by a camera within a moving vehicle.
Interior sensing devices have been introduced into vehicles, e.g. cameras for monitoring driver movements/characteristics for the purpose of safety. Particularly, such sensing capabilities are now implemented into the cabin of larger commercial vehicles such as trucks.
It has been recognized that common interior sensors such as DMS (driver monitoring system) cameras, which are widely used in modern passenger vehicles, are not suited for direct implementation in commercial vehicles for several reasons, e.g. related not only to environmental or vibration profile requirements, compelling suppliers to produce adequately more robust hardware, but also to signal processing demands. For example, the driver's seat of a commercial vehicle typically has its own suspension that introduces significant head movement relative to the interior camera, since the camera is usually fixed/mounted on top of a central display in the independently suspended cabin.
Accordingly, where interior monitoring has become a desirable safety feature for a large vehicle, there is now a requirement to provide a DMS camera which will address image instability caused by the above factors. Particularly, some functions that interior sensing devices are required to provide are based on human head movements, which can be easily distorted by the vibration based “noise” when the measuring device and measured object are not fixed to the same coordinate system.
The background description provided here is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.
In light of the above identified issues, the present invention seeks to address shortcomings associated with known interior sensing systems, e.g. as suitable for commercial vehicles, and propose an improved solution that may enhance aspects of a user/operator experience.
A first broad aspect is outlined according to claim 1 of the appended claims. For example, the invention may be expressed as:
A system for interior sensing in a vehicle, e.g. serving as an instability mitigation system for a vehicle interior sensing device such as a camera, comprising: a sensing device for capturing image data relating to an occupant within a vehicle, the sensing device being subject to a first coordinate system of the vehicle; and at least one processor configured to perform the steps of: identifying feature points of the occupant from captured image data and generating raw feature points data; compensating the raw feature points data with displacement data correlated with a second coordinate system, and generating processed result data; and determining, based on the processed result data, one or more occupant characteristics indicative of the occupant's behavior.
The camera is a driver monitoring camera that may be configured for capturing an image (e.g. as a channel) in the infrared spectrum. The camera may be a stereoscopic camera.
In one form, the displacement data is a signal input obtained from at least one sensor, such as an accelerometer. The accelerometer may be located with the sensing device/camera. Alternatively, or in addition the accelerometer may be located with an extant device associated with the vehicle or occupant (e.g. a smart phone/device).
In one form, the displacement data is obtained from the captured image data or from captured image data collected from a second sensing device (e.g. camera). The sensing device or second sensing device may have a wide-angle lens to capture further details of cabin structure such as a seat (e.g. headrest). A data stream obtained from the captured image itself (or a second image) may monitor movement of the seat to calculate how it moves independently from or relative to other fixed points in a cabin.
In forms of the invention, the system is adapted for incorporation into a vehicle comprising a seat independently suspended (e.g. the second coordinate system) from the structure of the vehicle (e.g. the first coordinate system). In one form the seat (e.g. headrest) may comprise a seat feature point and/or sensor for deriving the displacement data.
In forms of the invention, the displacement data is time correlated with the captured image data.
In one form, the generated action includes one or more of: an alert (e.g. audible, visual and/or tactile) that an occupant has strayed outside predetermined safety parameters; modifying an augmented reality display/projection to correct for driver's perspective; activating a cabin illumination based on driver's line of sight.
In a second aspect, a method is outlined according to claim 11.
In a third aspect, a non-transitory computer readable medium is provided, comprising instructions which, when executed by a vehicle control system, implement the method disclosed herein.
In a particular form of the invention, there may be provided an instability mitigation system, comprising: a camera for positioning within the cabin of a vehicle having a driver's seat, e.g. with independent suspension, a displacement sensor (e.g. an accelerometer) associated with the camera, cabin, occupant and/or vehicle more generally; and a processor configured for: receiving image data from the sensing device, identifying feature points there within; and further receiving movement data from the sensor to perform a subtraction function to determine an accurate spatial location of the feature points corresponding to movement of a driver's head. The processor may be configured to compile a model from movement data sensed by the accelerometer and correlate this for subtraction from observed movement (image) data taken from identified feature points of a user. Results may be refined by a machine learning algorithm for greater accuracy over time.
In this way, the invention described herein reveals the possibility of making use of a common sensor, e.g. an accelerometer. An accelerometer is well known as an instrument that measures acceleration, the change in velocity of an object over time (SI unit: m·s−2). Acceleration is directly proportional to the force acting on the object to move it (as is the mass of the object).
An accelerometer may already be installed in relation to other vehicle/navigation functions in combination with an interior sensing device (e.g. an inward facing camera), to mitigate inaccuracies in reading feature point movements within the sensing device's field-of-view (FOV) caused by road-based vibrations and/or non-uniformity. The accelerometer may be substituted or supplemented by an alternative means of collecting displacement/movement data, such as by another movement sensor configuration or image processing to track the movement of part of a vehicle seat.
By finding a correlation between the readings from the accelerometer or other data source (e.g. frequency and amplitude of acceleration) and the fluctuation of the feature points (e.g. frequency and amplitude) positions captured by, e.g. a DMS camera, the processing unit can be configured to filter-out (subtract) any unwanted offset signals caused by the vibrations and improve the quality in identifying/reading the driver's natural head movement which are used in calculations related to some functions of the vehicle safety system. Data from the accelerometer is used to more accurately interpret image data, e.g. to determine a state closer to actual driver behavior than that apparently observed by the camera. Ultimately, the system is configured to generate actions such as alerts/warnings that may be visual, tactile or audible cues for a driver or other operator. The action may be to modify projection of an augmented reality, HUD, so that it is more accurate from a driver's perspective.
The accelerometer may be integrated with the camera assembly and/or the processing unit may utilize a signal generated by an accelerometer located in other vehicle devices/functional units. Signals throughout the vehicle may be communicated by a CAN interface or the like. As mentioned earlier, the signal relating to displacement may be supplemented or substituted by a data stream obtained from another source such as from the captured image itself where movement of the seat is monitored to calculate how it moves independently from other fixed points in a cabin.
It is noteworthy that the invention is not an image stabilization system as is known in the art; it is rather a workaround that helps to deal with observed problems that vibrations have on the accurate reading of feature points position during an interpretation step of the processor. Images from the internal camera are not ordinarily displayed for a driver and so the system does not need to serve to correct a displayed image.
It will be apparent that a system according to the invention has particular utility in land vehicles and analogous environments where an operator is subject to movement (e.g. independent movement, gyroscopic mechanisms) relative to the cabin and requires monitoring for safety, such as maritime and aerospace applications.
The invention was devised to address image processing inaccuracies, e.g. due to a bumpy road. As such it has application in the field of augmented reality functions for vehicle. For example, it is necessary for tracking of eyes to be accurate so that a projected image appears at a correct position from a driver's perspective, i.e. a frame that appears around a pedestrian needs to be accurately placed.
The invention may make use of different signal inputs, other than or in addition to an accelerometer. For example, DMS cameras have a narrow field of view but if a fish eye lens were utilized then it will be able to capture a greater area of the cabin and include the geometry of a seat as a whole. The headrest itself may be a reference point and its movement can be subtracted from the occupant reference points (e.g. eyes) to determine a more accurate behavior of the occupant.
In the case above, the time correlated/comparative data is obtained from the image itself. In further forms, the headrest or other vehicle part moving with the occupant may be configured to be tracked by easily identifiable feature points (e.g. bright and/or reflective spots). Indeed, a seat may incorporate some other form of sensor for triangulation/spatial tracking relative to the cabin/DMS camera.
As will be apparent, the invention is concerned with recognizing separate coordinate systems in order to interpret reference points, i.e. translating the relationships between two or more coordinate systems separating a driver from the road, e.g. an air suspended seat.
The invention is not image stabilization in a conventional sense because, as it may never be seen by a person, the image itself is not corrected. Indeed, DMS cameras often operate in the infrared part of spectrum and there is no user VDU as such. There may be cases where images are reviewed by a human, e.g. following an incident, but the normal use of the system is to aid interpretation within a machine environment for implementing functions after the machine has interpreted user behavior.
Further areas of applicability of the present disclosure will become apparent from the detailed description, the claims, and the drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.
The present disclosure will become more fully understood from the detailed description and the accompanying drawings.
In the drawings, reference numbers may be reused to identify similar and/or identical elements.
The following description presents embodiments and, together with the drawings, serves to explain principles of the invention. However, the scope of the invention is not intended to be limited to the precise details of the embodiments or exact adherence with all features and/or system/method steps, since variations will be apparent to a skilled person and are deemed also to be covered by the description. Terms for components used herein should be given a broad interpretation that also encompasses equivalent functions and features. In some cases, several alternative terms (synonyms) for structural/functional features have been provided but such terms are not intended to be exhaustive. Descriptive terms should also be given the broadest possible interpretation; e.g. the term “comprising” as used in this specification means “consisting at least in part of” such that interpreting each statement in this specification that includes the term “comprising”, features other than that or those prefaced by the term may also be present. Related terms such as “comprise” and “comprises” are to be interpreted in the same manner. Directional terms such as “vertical”, “horizontal”, “up”, “down”, “sideways”, “upper” and “lower” may be used for convenience of explanation, usually with reference to the orientation shown in illustrations and are not intended to be ultimately limiting if an equivalent function can be achieved with an alternative dimension and/or direction. Indeed, in the present case a term such as “horizontal” axis or “vertical” axis (y, z) can be affected by the orientation imposed on the sensor mounting structure. Therefore, all directional terms are relative to each other.
The description herein refers to embodiments with particular combinations of steps or features, however, it is envisaged that further combinations and cross-combinations of compatible steps or features between embodiments will be possible. Indeed, isolated features may function independently as an invention from other features and not necessarily require implementation as a complete combination.
It will be understood that the illustrated embodiments show applications only for the purposes of explanation. In practice, the invention may be applied to many different configurations, where the embodiment is straightforward for those skilled in the art to implement.
Referring to
In accordance with the invention, it has been recognised that the degree of movement, due to road vibration and non-uniformity, of the head shown in
Since driver seat suspension/shock absorption is typically in the vertical, up-down, z-direction, the example outlined herein relates to removal of z-axis “noise”.
The relationship between the z-displacement and acceleration can be specified experimentally during a development phase of the vehicle systems, depending on the accelerometer location.
The “dash-dot” line (-.-.-) 16, having a second highest z-displacement over time shown in
With time-correlated displacement data collected by both the accelerometer and feature points in the FOV observations, the solid line (-) 17 of
The concept of refining the signal can be generally described by the equation:
z
h
=z
c
−z
a
The foregoing simplified example relates to monitoring the z direction, but full three-dimensional space (x, y, z) directions are to be implemented by the system for the most accurate interpretation of user feature points.
Block diagrams presented in
As mentioned above, the vibration corrected signal 23 is output to a DMS 24 for prompting an action.
It will be apparent to a skilled person that the invention takes a data input related to a coordinate system and injects it into the processing system, downstream of feature point detection image processing, to then reach more accurate conclusions when interpreting driver behaviour.
One example of data input is use of an accelerometer or other sensor that may be resident in a driver's own phone. Sensors do not have to be fixed on the vehicle so long as they are in or around the moving vehicle.
Alternative forms of data input may include derivations of a second coordinate system from the image itself, e.g. tracking a structure, such as a seat, that is subject to the same origin of movement as the feature points.
The invention has implications not only for safety but in general for comfort/improved driver experience and entertainment. Implementation of action includes not only alarm but helpful actions to improve comfort and user experience, such as detecting that a driver is glancing over at a passenger area (e.g. glovebox) in a dimmed cabin and automatically raising the level of illumination in that area.
It is noteworthy that, in general, 2D cameras may be utilised with the invention, e.g. in IR spectrum, and the wider cameras also mentioned herein are usually RGB/IR cameras (i.e. operating in both the IR spectrum, as well as in the visual range, such as for entertainment purposes).
There are also cameras, including 3D/stereoscopic cameras, that can be utilised with the concepts herein. For example, a TOF (time of flight) camera. Such devices have been used to track drivers' gestures in vehicles, and also to precisely track the 3D eyes position, e.g. for compensating augmented reality parallax as mentioned herein. A TOF camera utilizes light to bounce off an observed object and measures the time between the light emission and reception (i.e. its travel to the object and reflecting back to the sensor).
By way of summary, the invention is generally embodied by an instability mitigation system for a vehicle interior sensing device. A camera captures image data relating to an occupant within a vehicle, where after feature points of the occupant are identified from the captured image data. The raw feature points data is then compensated by displacement data correlated with a second coordinate system. Resultant data, where the feature points are more accurately represented (e.g. road vibration and its effect upon an independently suspended driver's seat is cancelled out), is interpreted according to occupant behaviour and an action is generated. The action may be an alert or modification of an augmented reality projection visible to a driver.
The term non-transitory computer-readable medium does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave). Non-limiting examples of a non-transitory computer-readable medium are nonvolatile memory circuits (such as a flash memory circuit, an erasable programmable read-only memory circuit, or a mask read-only memory circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).
The term “set” means a grouping of one or more elements. The elements of a set do not necessarily need to have any characteristics in common or otherwise belong together. The phrase “at least one of A, B, and C” should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.” The phrase “at least one of A, B, or C” should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR.
| Number | Date | Country | Kind |
|---|---|---|---|
| 22208272 | Nov 2022 | EP | regional |
