This application claims priority to Japanese Patent Application No. 2021-189197 filed on Nov. 22, 2021, which is incorporated herein by reference in its entirety including the specification, claims, drawings, and abstract.
The present specification discloses an image display apparatus that displays a target image in a superimposed manner in the field of view of a user who is a person on board a vehicle.
Conventionally, there are known techniques of displaying a predetermined image in a superimposed manner in the field of view of a user, to thereby cause the user to perceive that a virtual object represented by the image is present in reality. For example, Patent Literature 1 discloses a technique in which smart glasses, which are an eyeglass type display device, are worn by a driver, and an image representing a leading vehicle, that guides the vehicle the driver is in, is displayed on the smart glasses. In Patent Literature 1, the leading vehicle represented by the image moves so as to guide the vehicle the driver is in to a destination. Accordingly, the driver can travel to the destination by performing driving manipulations to follow the leading vehicle.
Patent Literature 2 discloses a contact lens type display device, instead of an eyeglass type display device.
Here, in order to cause the user to perceive that a virtual object is present in reality, it is necessary to determine the display position of the image to be displayed on the display device (hereinafter referred to as the “target image”) based on the position in real space of the virtual object represented by the target image and the position in real space of the display device.
In Patent Literature 1, for the purpose of identifying the position of the display device in real space, a camera is mounted to the display device, and a marker is provided by, for example, mounting a dedicated marker for that purpose on the dashboard, or allowing the windshield to serve as the marker. An image of a scene including the marker is captured using the camera, and based on the captured image, the position of the display device in real space is identified.
However, a dedicated marker as noted above must be specially provided. Further, since a marker implemented by the windshield varies depending on the surrounding lighting environment conditions and the like, there may be difficulties in recognizing that marker, and its detection may require time or may incur a large processing load. Furthermore, when the position of the marker cannot be detected, the position of the display device in real space cannot be detected, resulting in that the display position of the target image cannot be determined.
In view of the above situation, the present specification discloses an image display apparatus that can determine the display position of the target image in a more appropriate manner.
An image display apparatus as disclosed in the present specification includes: a display device to be worn on the head of a user who is a person on board a vehicle; and configured to display a target image in a superimposed manner in a field of view of the user; a SLAM-purpose camera fixed to the display device and configured to obtain a SLAM-purpose image capturing surroundings of the display device; a memory configured to store marker information indicating features of interior parts for each vehicle; and a device controller configured to detect, using the marker information, a marker from the SLAM-purpose image in which interior parts inside the vehicle are captured, and determine a display position of the target image based on the detected marker.
The marker information may be downloaded from outside and stored in the memory.
The marker may be a shape provided in an instrument panel inside the vehicle, or a shape of a black ceramic part on a windshield.
According to the technique disclosed in the present specification, the display position of a target image can be determined in a more appropriate manner.
Embodiment(s) of the present disclosure will be described based on the following figures, wherein:
A configuration of an image display apparatus will now be described by reference to the drawings. Although the following description refers to specific aspects in order to facilitate understanding, those aspects are examples only and may be changed as appropriate.
The wearable device 12 is a device to be worn on the head of a person (e.g., a driver) on board a vehicle, and is, for example, an eyeglass type or goggle type device. In order to function as the image display apparatus 10, the wearable device 12 comprises a display device 14, a SLAM-purpose camera 16, a pupil position sensor 18, and a device controller 20. As the wearable device 12, a contact lens type device may alternatively be used. In that case, the SLAM-purpose camera 16 and the device controller 20 are mounted to the contact lens. Since the contact lens basically moves following the movement of the pupil, the pupil position sensor 18 is not necessary. Although the device appearance differs greatly from an eyeglass type device, the contact lens type device is substantially identical thereto in function, and the configuration of an eyeglass type device can be employed for the contact lens type device without change.
The wearable device 12 will be described by reference to
The display device 14 displays images in the field of view of the user 100 wearing the wearable device 12. In the present embodiment, the display device 14 is an organic EL display or liquid crystal display having a display area 22 located on the inside of the rim 24, and displays images in a part or the entirety of this display area 22. The display area 22 has high transparency. Accordingly, when no image is displayed in the display area 22, the user 100 (i.e., the person on board) can view the scene in front over the display area 22. Further, when an image is displayed only in a part of the display area 22, the user 100 can view the scene in front and the displayed image at the same time. At that time, the image may be opaque or semi-transparent. In the following description, an image displayed on the display device 14 will be referred to as a “target image” in order to distinguish from other images.
The SLAM-purpose camera 16 is a camera which is fixed to the display device 14 and which captures images of the surroundings of the display device 14. The SLAM-purpose camera 16 is, for example, fixed facing forward in the vicinity of a front end of a temple 26, and captures images of a region similar to the field of view of the user 100. In the following, an image captured using this SLAM-purpose camera 16 will be referred to as a “SLAM-purpose image”. As will be described further below, the device controller 20 identifies the position and orientation of the display device 14 in real space based on AR markers captured in a SLAM-purpose image.
The pupil position sensor 18 is a sensor that detects the position of the pupils of the right and left eyes of the user 100, and is, for example, fixed near the center of the rim 24. This pupil position sensor 18 may be formed using, for example, a camera and the like.
The device controller 20 controls the operation of the wearable device 12. The device controller 20 obtains images and position information obtained using the SLAM-purpose camera 16 and the pupil position sensor 18, processes such information, and causes the display device 14 to display a target image.
In physical terms, the device controller 20 is a computer comprising a processor 20a, a memory 20b, and a communication I/F 20c. The term “computer” as used herein covers a microcontroller incorporating a computer system in a single integrated circuit. Further, the processor 20a denotes a processor in a broad sense, and includes a general-purpose processor (e.g., a CPU (central processing unit), etc.), a dedicated processor (e.g., a GPU (graphics processing unit), an ASIC (application-specific integrated circuit), a FPGA (field-programmable gate array), a programmable logic device, etc.), and the like.
The memory 20b stores digital data necessary for the computer to perform processing. This memory 20h includes at least one of a main memory connected to the processor 20a via a memory bus, or a secondary storage device accessed by the processor 20a via an input/output channel. The memory 20b can be constituted of a semiconductor memory (e.g., a RAM, a ROM, a solid-state drive, etc.).
The communication I/F 20c is wirelessly connected to another electronic device, specifically an in-vehicle system 28, and can access various websites via the Internet. In particular, the communication I/F 20c can communicate with an information center 30 that provides vehicle information. Further, the communication I/F 20c may perform data transmission and reception with the in-vehicle system 28 via near-field communication such as CAN (controller area network) communication, Bluetooth (registered trademark), Wi-Fi (registered trademark), and infrared communication.
The above-described functions of the device controller 20 may alternatively be implemented by an external system such as a computer of the in-vehicle system 28, a computer of the information center 30, or a separate portable computer (e.g., a smartphone, etc.). In that case, the device controller 20 transmits the information from the SLAM-purpose camera 16 and the pupil position sensor 18 to the external system such as the in-vehicle system 28, receives back image data which are the results of processing, and displays the image data on the display device 14. It is also possible to execute a part of these processes in an external system.
The in-vehicle system 28 is a system installed in the vehicle, and controls various in-vehicle devices. Here, as shown in
The multi-function display 40b is a display that displays information related to in-vehicle electronic devices (such as a navigation device and an audio device). As shown in
The electronic inner mirror 40c is a display that displays images of the vehicle rear scene captured by a rear camera (not shown in drawing). This electronic inner mirror 40c is used in place of a rearview mirror that shows the vehicle rear scene by optical reflection. The electronic inner mirror 40c may be one that is switchable between a digital mode for displaying images and a mirror mode for showing the vehicle rear scene by optical reflection. As shown in
As noted above, the device controller 20 generates data of a target image to be displayed on the display device 14. Here, although it is possible to use a “device-fixed display mode” and a “space-fixed display mode” as the display modes for displaying a target image on the display device 14, in the present embodiment, the “space-fixed display mode” is used. This space-fixed display mode is a display mode in which a target image representing a predetermined object is displayed so as to appear to be present in real space.
As an example, reference will be made to a situation as shown in
When in the device-fixed display mode, the display position of the target object 72 (in the example of
In contrast, in the space-fixed display mode, it is identified where the target object 72 (in the example of
By displaying the target image 50 as such in the space-fixed display mode, the user 100 perceives an illusion that the target object 72 shown by the target image 50 is present in reality. In other words, by displaying the target image 50 in the space-fixed display mode, information can be added, deleted, emphasized, and attenuated in a real environment, and the real world as viewed by a human can be augmented. Such a technology is generally referred to as “augmented reality” or “AR”.
Next, an example display of target images 50 according to the present embodiment will be described.
As such, in the space-fixed display mode, since a target image 50 can be displayed in consideration of arrangements of actual objects, it is possible to reliably prevent the target image 50 from obstructing drive manipulations. Further, in the space-fixed display mode, a target image 50 can be displayed at a position having correlation with an actual object (such as a pedestrian), and it is thereby possible to effectively direct the attention of the user 100 to that object.
In order to perform display in the space-fixed display mode, it is necessary to accurately detect the position of the pupils relative to the display device 14, as well as the position and orientation of the display device 14 in real space. The device controller 20 determines the position and the like of a target image 50 within the display area 22 based on the position and orientation of the target object in real space, the position and orientation of the display device 14 in real space, and the position of the pupils relative to the display device 14. Among these, the position of the pupils relative to the display device 14 is detected using the pupil position sensor 18, as noted above.
The position and orientation of the display device 14 in real space are calculated by the device controller 20 by performing visual SLAM (simultaneous localization and mapping) based on a SLAM-purpose image obtained using the SLAM-purpose camera 16. Visual SLAM is a technology for simultaneously estimating, based on an image captured using a camera, three-dimensional environment information and the position and orientation of the camera. In order to perform visual SLAM, characteristic shapes of a plurality of interior parts inside the vehicle are recognized as AR markers 60 (see
In the present embodiment, the memory 20b comprises a marker information storage unit 20b-1, and marker information regarding the position, size, and shape of interior parts that serve as AR markers 60 are stored therein in advance. For example, at the time of manufacture of the vehicle, the marker information regarding interior parts that are candidates for AR markers 60 for that vehicle may be stored in a memory in the in-vehicle system 28, and at the time of an initial setting process of the image display apparatus 10 (or the wearable device 12), the image display apparatus 10 may communicate with the in-vehicle system 28 to obtain data regarding the interior parts that serve as AR markers 60, and store the data in the marker information storage unit 20b-1 of the memory 20b. The image display apparatus 10 may also obtain vehicle type information, which may be received from the in-vehicle system 28, via input of the vehicle type information by the user, or via communication with the information center 30, and may acquire data regarding the interior parts that serve as AR markers 60 from the vehicle type information.
As such, the marker information storage unit 20b-1 has stored therein information regarding the interior parts that serve as candidates for AR markers 60. Based on the marker information stored in the marker information storage unit 20b-1, the image display apparatus 10 performs image recognition processing with respect to a SLAM-purpose image captured using the SLAM-purpose camera 16, and achieves image recognition of the AR markers 60 in the SLAM-purpose image. At that time, since the marker information is used, the AR markers 60 can be reliably extracted by relatively simply processing similar to that in a case where AR markers 60 having fixed shapes are employed.
After that, using the recognition results concerning the recognized one or plurality of AR markers 60, the display position, size, and shape of the target image 50 are determined, and the target image 50 is displayed on the display device 14.
It is possible to adopt an arrangement in which: information regarding a plurality of AR marker candidates is included as the marker information; for each AR marker candidate, a score obtained in performing its image recognition from a SLAM-purpose image (i.e., a score of similarity in recognition) is detected as appropriate; and a higher priority level is assigned to a candidate having a higher score. Then, in detecting AR markers 60 during travel, by performing recognition of only a small number of (e.g., two) AR markers having the highest priority levels, the processing load can be reduced.
Further, from the information center 30 or the like, the data regarding interior parts that serve as AR markers 60 can be obtained corresponding to the vehicle type. Accordingly, marker information corresponding to the vehicle being used can be registered in the memory 20b, and the AR markers 60 can be detected based on appropriate information regarding the AR markers 60.
When the wearable device 12 is brought into the vehicle and the power is turned ON, a determination is made regarding whether to acquire marker information (S11). When a new wearable device 12 is brought into the vehicle, the image display device 10 may be automatically set to a marker information acquisition mode. The image display device 10 may communicate with the in-vehicle system 28 and thereby determines whether the wearable device 12 has been used in the past. The image display device 10 may periodically transmit an inquiry to the information center 30 so as to determine whether update information is available, and when the update information is available, YES may be determined in S11.
When the result of the determination in S11 is YES, marker information is acquired from the in-vehicle system 28 or the external information center 30, and the marker information is registered in the marker information storage unit 20b-1 (S12).
Next, a SLAM-purpose image is obtained (S13), and marker information regarding a single registered AR marker 60 is retrieved (S14). Using the retrieved marker information, the AR marker 60 is detected by performing image recognition (S15). Then, a score for the image recognition processing is recorded (S16) The score may be stored as one marker information item in the marker information storage unit 20b-1.
Subsequently, a determination is made regarding whether the processing is completed for all AR markers 60 stored in the marker information storage unit 20b-1 (S17), When the result of this determination is NO, the process returns to S14.
When the result of the determination in S17 is YES, priority levels are registered for all processed AR markers (S18). Here, even when the result of the determination in S11 is NO, S18 is performed to register priority levels. Information such as a score for image recognition processing obtained when a process of displaying a target image during driving is performed and the number of times an AR marker is used that are described later may be used for the priority registration of S18.
First, an image from the SLAM-purpose camera 16 is retrieved (S21). Using the information of the registered AR markers, the AR markers 60 are detected form the image (S22). In performing this AR marker 60 detection, processing may be executed simultaneously regarding the plurality of AR markers stored in the marker information storage unit 20b-1 based on the marker information thereof, or the processing may be performed sequentially, for one AR marker at a time.
Next, using the position information of the recognized AR markers 60, a display position of a target image 50 is determined (S23), and the target image 50 is displayed at the determined position (S24). Then, a score of the AR marker recognition and the like obtained during the display processing performed at this time are recorded (S25).
Although an eyeglass type device was used as the wearable device 12 in the above-described embodiment, a contact lens type device can alternatively be used. Further, although a display that displays an image in the display area 22 was described as an example display device 14, the display device 14 may alternatively be a projector that projects an image on a retina of the user 100. Furthermore, in the above description, the user 100 views the real space over the transparent display area 22. However, the display area 22 may alternatively be configured opaque such that the user 100 cannot view the real space over the display area 22. In that case, the device controller 20 displays, in the display area 22, a synthesized image formed by synthesizing a captured image of the real space and a target image representing a virtual object.
Number | Date | Country | Kind |
---|---|---|---|
2021-189197 | Nov 2021 | JP | national |