Patent Application
20220044450
The present invention relates to a portable video display device and a video display method.
In recent years, portable video display devices represented by smartphones have become widespread in the world. Among them, a head-mounted display (HMD) that is mounted on the head of the user displays an image of a real space and an image of augmented reality (AR) created by, for example, a computer on a glasses-type display screen so as to be superimposed on each other. In addition, a sensor can be attached to the HMD, and information acquired by the sensor can be displayed as an AR image on the display screen. For example, Patent Document 1 discloses, as a mobile terminal that describes an action history of a user, a configuration including: a terminal information acquisition means that acquires position information of a host terminal; a camera means that generates a camera image obtained by capturing the surroundings; a depiction history calculation means that calculates action history depiction information (avatar) to be displayed on the basis of the action history acquired in advance and an imaging range of the camera means; an image combination means that generates a composite image in which the action history depiction information has been depicted on the camera image; and a display means that displays the composite image.
Patent Document 1: WO 2011/093031 A
There are many cases in which old users forget the place where they stopped off or a return route while traveling on a complicated route. In this case, there is a method that guides a typical route back to a starting point when the user sets the starting point as in a navigation device. However, the method does not always guide the same route as that through which the user actually passed, and it is not expected that the method will accurately reproduce the route on the way.
In Patent Document 1, when the user inputs an instruction to record the action history, the action history is registered in an action history server. Then, the action history of a desired user is acquired from the action history server, and an avatar is displayed so as to be superimposed on the camera image according to the position information of the user. However, in Patent Document 1, it is assumed that the action history of any other user is acquired and displayed, and the display of the action history of the user of the terminal is not particularly considered. In addition, when the action history is displayed, the avatar is displayed in the order in which the user moved to the points where the user moved. Therefore, the disclosed configuration is not suitable as a tool to guide the user, who has forgotten the return route, from the current position to the return route.
An object of the invention is to provide a video display device that guides a return route along a route through which a user moved actually.
In order to solve the problem, a video display device according to the invention includes: a sensor that detects a position and a direction of a user carrying the video display device; a trajectory information acquisition unit that acquires information of a migration trajectory of the user from a detection result of the sensor; a storage unit that stores trajectory information of the user acquired by the trajectory information acquisition unit and information of an avatar which is a virtual image indicating the user; a display unit that displays the migration trajectory of the user with the avatar; and a control unit that controls the trajectory information acquisition unit and the display unit. The control unit generates the avatar from the information of the avatar stored in the storage unit, calculates a current visual field area of the user using the sensor, and displays the avatar on the display unit so as to be disposed in the migration trajectory of the user according to the current visual field area of the user on the basis of the trajectory information of the user stored in the storage unit.
According to the invention, it is possible to guide a user, who has forgotten the place where the user stopped off when moving or a return route, so as to easily understand the return route.
Hereinafter, embodiments of the invention will be described with a focus on an example of a head-mounted display (HMD) which is a head-mounted video display device.
The main control unit 10 is a microprocessor unit that controls the entire HMD 1 according to a predetermined operation program. The system bus 30 is a data communication path for transmitting and receiving various kind of commands and data between the main control unit 10 and each constituent block in the HMD 1.
The storage unit 40 stores various programs 41 for controlling the operation of the HMD 1 and various kinds of data 42 including operation setting values, detection values from a sensor unit which will be described below, and objects including contents, and has a work area 43 that is used for various program operations. At that time, the storage unit 40 can store an operation program downloaded from a network, various kinds of data created by the operation program, and contents such as downloaded moving images, still images, and voices. In addition, the storage unit 40 can store data such as moving images and still images captured by the imaging unit 71. The storage unit 40 needs to hold the stored information even in a state in which the HMD 1 is not supplied with power from the outside. Therefore, devices, such as semiconductor element memories including flash ROMs and solid state drives (SSDs) and magnetic disk drives including hard disc drives (HDDs), are used as the storage unit. Further, the update and function expansion of each of the operation programs stored in the storage unit 40 can be performed by a download process from each server device on the network.
The sensor unit 50 includes, for example, a global positioning system (GPS) sensor 51, a geomagnetic sensor 52, a distance sensor 53, an acceleration sensor 54, a gyro sensor 55, and an altitude sensor 56 in order to detect various states of the HMD 1. For example, the position, inclination, direction, movement, and altitude of the HMD 1 are detected by these sensor groups. In addition, the sensor unit 50 may include other sensors such as an illuminance sensor and a proximity sensor.
The communication processing unit 60 includes a local area network (LAN) communication unit 61 and a telephone network communication unit 62. The LAN communication unit 61 is connected to the network, such as the Internet, through an access point or the like and transmits and receives data to and from each server device on the network. The connection to the access point or the like may be a wireless connection such as Wi-Fi (registered trademark).
The telephone network communication unit 62 performs telephone communication (call) and data transmission and reception using wireless communication with, for example, a base station of a mobile telephone communication network. The communication with, for example, the base station may be performed by a Wideband Code Division Multiple Access (W-CDMA) (registered trademark) system, a Global System for Mobile communications (GSM) (registered trademark) system, a Long Term Evolution (LTE) system, or other communication systems. Each of the LAN communication unit 61 and the telephone network communication unit 62 includes, for example, an encoding circuit, a decoding circuit, and an antenna. In addition, the communication processing unit 60 may further include other communication units such as a Bluetooth (registered trademark) communication unit and an infrared communication unit.
The image processing unit 70 includes the imaging unit 71 and the display unit 72. The imaging unit 71 is a camera unit that converts light input from a lens into an electric signal using an electronic device, such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor, and inputs image data of an object in an outdoor scene. The display unit 72 is a display device of a transmission-type display using, for example, a laser projector and a half mirror and provides image data to the user of the HMD 1.
The voice processing unit 80 includes a voice input/output unit 81, a voice recognition unit 82, and a voice decoding unit 83. A voice input of the voice input/output unit is a microphone, and the microphone converts the user's voice or the like into voice data and inputs the voice data. In addition, a voice output of the voice input/output unit 81 is a speaker, and the speaker outputs, for example, voice information necessary for the user. The voice recognition unit analyzes the input voice information and extracts, for example, instruction commands. The voice decoding unit 83 performs an encoded voice signal decoding process and a voice synthesis process.
The operation input unit 90 is an instruction input unit that inputs an operation instruction to the HMD 1. The operation input unit 90 is composed of, for example, operation keys obtained by arranging button switches and may further include other operation devices. For example, the HMD 1 may be operated by a separate mobile terminal device that is connected by wired communication or wireless communication using the communication processing unit 60. Further, the HMD 1 may be operated by a voice command of an operation instruction using the voice recognition unit 82 of the voice processing unit 80.
The configuration example of the HMD 1 illustrated in
The main control unit 10 mainly performs a process of controlling the entire HMD 1 using the various programs 41 and the various kinds of data 42 in the storage unit 40. The processes of the HMD 1 include a trajectory collection process (S100) that collects and stores information related to the migration trajectory of the user and an avatar display process (S200) that displays an avatar which is a virtual image indicating the user on the basis of the stored trajectory information (which are represented by a dashed line and a chain line, respectively).
In the trajectory collection process (S100), information from various sensors of the sensor unit 50 is acquired by a various sensor information acquisition function 11, and the information acquired from the various sensors is converted into trajectory information that can be easily processed in the HMD by a trajectory information processing function 12. The converted trajectory information is stored by a trajectory information storage function 13. In addition, when a user inputs an imaging instruction during the acquisition of the trajectory information, the imaging unit 71 of the image processing unit 70 is used to perform imaging. An imaging process in this case is performed by an imaging data acquisition function 14, and the acquired imaging data is stored by an imaging data storage function 15 so as to be associated with the trajectory information.
In the avatar display process (S200), avatar information of the user stored in advance in an avatar information storage function 16 is read out, and an avatar display function 18 displays an avatar generated by an avatar generation function 17. A display position of the avatar is determined on the basis of the trajectory information stored in the trajectory information storage function 13. However, a visual field calculation function 19 determines whether or not the display position of the avatar is present in the visual field of the HMD 1. When the display position is absent, the avatar is not displayed. Further, an imaging data reproduction function 20 reproduces the imaging data stored in the imaging data storage function 15 according to a path trajectory.
The storage functions, such as the trajectory information storage function 13, the imaging data storage function 15, and the avatar information storage function 16, may be stored in the external server 6 through the LAN communication unit 61 of the communication processing unit 60 as illustrated in
S110: The trajectory collection process (S100) is started in response to an instruction from the user 2. In this embodiment, a trajectory collection start instruction is input by the voice of the user 2. For example, when the user utters a voice command “Trajectory Start”, the voice recognition unit determines that an instruction to start the trajectory collection process is input.
S111: The point numbers of the trajectory points are initialized. Here, it is assumed that 0 is set as an initial value of a point number p, and the point number p at the start time of the trajectory collection process is 0.
S112: A timer is started. The timer measures time using a clock provided in the main control unit 10. The timer is used to collect the data of the trajectory points of the user at regular time (unit time) intervals.
S113: The position of the HMD 1 is detected by the information from the sensor unit 50 (various sensor information acquisition function 11), is converted into a storage data format (trajectory information processing function 12), and is stored in the various kinds of data 42 of the storage unit 40 (trajectory information storage function 13). The position of the HMD 1 in the plane is detected by the GPS sensor 51 of the sensor unit 50. Radio waves can be received from a plurality of GPS satellites to detect the global position coordinates (longitude and latitude) of the HMD 1. In addition, the altitude can be detected by the height sensor 56 of the sensor unit 50. For example, the atmospheric pressure is measured to calculate the altitude.
For the position information of the HMD 1, global position information may be acquired using, for example, information transmitted from Wi-Fi (registered trademark), Bluetooth (registered trademark), and a mobile base station, in addition to the GPS information from the GPS satellites. Therefore, when it is difficult to acquire the GPS information, the position information of the HMD 1 is acquired from these information items. Of course, other information may be combined with the GPS information. Hereinafter, the information for acquiring the global position information is referred to as “GPS information or the like”.
In addition, in an environment in which the global position information can always be acquired with high accuracy, the trajectory information of the user can be disposed in a spatial coordinate system using the global position information. On the other hand, a unique spatial coordinate system (local coordinate system) having a trajectory acquisition start position as the origin may be generated, and a change in position may be calculated from the movement distance or movement direction of the user detected by, for example, the acceleration sensor 54 or the gyro sensor 55 of the sensor unit 50 and may be disposed in the local coordinate system. Then, the global position information may be acquired at a point where the global position information can be acquired with high accuracy and may be associated with the position information of the local coordinate system. With this configuration, the embodiment can also be applied to, for example, a case in which it is difficult to continuously acquire the trajectory or a case in which the user sees the trajectory information of another person.
S114: It is determined whether or not the user inputs an imaging instruction. When there is an imaging instruction, the process proceeds to S115. When there is no imaging instruction, the process proceeds to S116. In addition, two imaging modes, that is, a still image capture mode and a moving image capture mode are prepared in the imaging function of the imaging unit 71. In order to distinguish between the two imaging modes, for example, a voice command “Imaging” is used as a still image capture instruction, and a voice command “Imaging Start” is used as a moving image capture instruction. In addition, for moving images, a voice command “Imaging End” is used as an imaging end instruction.
S115: The imaging unit 71 of the image processing unit 5 captures an image of an object in front of the HMD 1 (imaging data acquisition function 14), and imaging data is stores as the various kinds of data 42 of the storage unit 40 (imaging data storage function 15).
S116: It is determined whether or not the user inputs an instruction to end the trajectory information acquisition process. For example, it is determined by a voice command “Trajectory End” that the instruction to end the trajectory collection process is input. When the end instruction is input, the process proceeds to S117 to end the trajectory information acquisition process. When the instruction to end the trajectory information acquisition process is not input, the process proceeds to S118.
S118: It is determined whether a unit time has elapsed. Specifically, it is determined whether the timer has exceeded the unit time. When the unit time has elapsed, the process proceeds to S119. When the unit time has not elapsed, the process returns to S114.
S119: 1 is added to the point number p, and the process returns to S112. Then, the timer is reset and restarted.
The trajectory point (position information) of the user can be collected at regular time intervals by the above-mentioned trajectory collection process (S100). For example, the unit time of the timer can be set to one second to store the trajectory of the HMD 1 every one second. Further, in parallel with this, the image of the object can be captured and stored during the trajectory collection process in response to an instruction from the user.
Here, two methods will be described for the trajectory information data (data table) stored in the storage unit 40 by the trajectory collection process (S100).
In addition, the position coordinates 712 are data with a fixed-length size in the data table. Therefore, when the position coordinates 712 are recorded in time series, the data of the point number 711 can be omitted. That is, the data of the position coordinates of a target point number p can be searched by a multiple value p of the fixed length size.
Here, when it is difficult to acquire the position information from the GPS information or the like, relative value information (difference value information) of, for example, the acceleration sensor 54 or the gyro sensor 55 of the sensor unit 50 is temporarily stored. Then, in a stage in which the position information (absolute value information) can be acquired from the GPS information or the like at any trajectory point, the temporarily stored relative value is corrected to an absolute value.
The difference trajectory information storage table 720 includes, as items, a point number 721 and difference position coordinates 722 corresponding to the trajectory point. In the difference position coordinates 722, difference values between the position coordinates of the previous trajectory point and the position coordinates of the current trajectory point are described. That is, difference values (ΔX, ΔY) between the planar position coordinates are described by the distances between two trajectory points in a longitude direction and a latitude direction. A difference value (ΔZ) in the height direction is also described by the distance.
Point number 1 indicates difference position coordinates (ΔX1, ΔY1, ΔZ1) one second after the start of the trajectory collection process, point number k indicates difference position coordinates (ΔXk, ΔYk, ΔZk) k seconds after the start, and point number n indicates difference position coordinates (ΔXn, ΔYn, ΔZn) at the time when the trajectory collection process is ended (n seconds after the start). The relationship with the values of the position coordinates 712 in
ΔX
p
=X
p
−X
p−1
, ΔY
p
=Y
p
−Y
p−1
, ΔZ
p
=Z
p
−Z
p−1.
The starting point/end point position coordinate table 730 includes, as items, starting point/end point distinguishment 731 that distinguishes between a starting point and an end point of trajectory collection and position coordinates 732 in which each position information item is represented by an absolute value. Of course, the position coordinates of the starting point are the same as the position coordinates (X0, Y0, Z0) of point number 0 and the position coordinates of the end point are the same as the position coordinates (Xn, Yn, Zn) of point number n in
As described above, when the position information (absolute value) 732 of the starting point and the end point is stored, it is possible to convert the difference trajectory information storage table 720 illustrated in
The avatar is displayed in reverse order of time from the end point (point number n) of the trajectory, which will be described below. At that time, when the position coordinates of the end point at which the display is started are not known, all of the difference position coordinates 722 illustrated in
These trajectory information items (the trajectory information storage table 710, the difference trajectory information storage table 720, and the starting point/end point position coordinate table 730) may be not only stored in the various kinds of data 42 of the storage unit 40, but also stored in the external server 6 through the network 5 for each trajectory point or when the trajectory information table is completed, as described with reference to
Next, the storage of the imaging data will be described. The imaging data corresponds to both still images and moving images.
The trajectory information storage table 710 and the still image data storage table 810 according to this example are stored so as to be associated with each other, which makes it possible to reproduce the still image captured in synchronization with the movement position of the HMD 1 every unit time (here, one second).
In this example, at the trajectory point k (point number k), the HMD 1 starts imaging from a start time Tks in the direction of Θk and stores imaging data Dk having a data size Mk captured until an end time Tke in the various kinds of data 42 of the storage unit 40 (S115). The moving image data may be reproduced on the entire display screen of the HMD. In this example, the moving image data is displayed on a reduced sub-screen and is reproduced from the moving image capture start time Tks to the moving image capture end time Ike.
The information of these imaging data items (the still image data storage table 810 and the moving image data storage table 820) is also stored in the external server 6 through the network 5 as described with reference to
Next, the avatar display process (S200) of the HMD 1 will be described. This processing procedure is stored in the various programs 41 of the storage unit 40. For the display of the avatar, the avatar information is extracted from the avatar information storage function 16 (avatar generation function 17), and the avatar is displayed on the display unit 72 of the HMD 1 (avatar display function 18). For the shape of the avatar, the avatar has the same life size as the user, and the height of the avatar is set by the user. For convenience, the avatar may have the height obtained by adding 10 cm to the distance from the ground (the height of the HMD 1 from the ground) measured by the distance sensor 53 of the sensor unit 50 in a standing state. For the display position of the avatar, the avatar is displayed on the display unit 72 of the HMD so as to be superimposed on a background image according to the position information stored in, for example, the trajectory information storage table 710.
When the trajectory information or the imaging data to be used are stored in the various kinds of data 42 in the HMD, the trajectory information or the imaging data is used. However, when the trajectory information or the imaging data is stored in the external server 6 through the LAN communication unit 61 of the communication processing unit 60, it is acquired from the server 6 and is then used.
In this embodiment, the avatar is displayed in reverse order of time from new trajectory information to old trajectory information on the basis of the trajectory point of the user. Since the avatar is displayed in reverse order of time, the user can return from the end point (current point) to the starting point (starting place) through the same route as that through which the user actually passed.
S210: The avatar display process (S200) is started. In this example, when the user utters a voice command “Avatar Start” which means the start of the display of the avatar, it is determined that an avatar display start instruction is input.
S211: The current position and direction of the HMD 1 worn by the user are detected. A detection method is performed by, for example, the GPS sensor 51, the geomagnetic sensor 52, and the height sensor 56 of the sensor unit 50 as in S113 of the trajectory collection process (S100).
S212: Point number s at the position coordinates closest to the current position of the HMD 1 is searched with reference to the trajectory information storage table 710 (or the difference trajectory information storage table 720). That is, when the current position is the end point in trajectory collection, s=n is established. When the user subsequently moves from the end point, s≠n is established.
S213: Point number s closest to the current position is set as the initial value of point number m to be displayed. Therefore, it is possible to perform display from the trajectory point close to the current position of the user.
S214: The timer is started. The timer uses the clock provided in the main control unit 10. Since the trajectory of the user is collected at regular time intervals (unit time), the timer is used in order to display the avatar in accordance with that. However, the unit time used to display the avatar may be different from the unit time in the trajectory collection process and may be the time multiplied by an arbitrary coefficient. For example, when the unit time used to display the avatar is set to ½ of the time (0.5 seconds in this example), the displayed avatar can be moved at double speed. On the contrary, when the unit time used to display the avatar is set to twice the time (two seconds in this example), the displayed avatar can be moved at ½ speed (slow). Further, the avatar may be sequentially interpolated and displayed (animated) between the trajectory points (for example, every 1/24 seconds) so as to be displayed as a moving image.
S215: The display position and direction of the avatar are calculated. When the trajectory information is stored in the trajectory information storage table 710, the values (Xm, Ym, Zm) of the position coordinates 712 corresponding to point number m are read out, and the avatar is disposed at a planar position (Xm, Ym) and an altitude (Zm).
When the trajectory information is stored in the difference trajectory information storage table 720, the values (difference values) of the difference position coordinates 722 of point number (m+1) are read out and subtracted from the previous display position (point number=m+1). That is, Xm=Xm+1−ΔXm+1, Ym=Ym+1−ΔYm+1, and Zm=Zm+1−ΔZm+1 are established. However, the initial position (m=s) is calculated using the values of the position coordinates (Xn, Yn, Zn) of the end point described in the starting point/end point position coordinate table 730 and the values of the difference position coordinates 722 up to point number s.
For convenience, the direction in which the avatar is facing is a direction connecting the previous trajectory point (m+1) and the current trajectory point (m).
S216: The current position and direction of the HMD 1 are detected. This is the same process as S211. The initial position (m=s) can be omitted because it has already been detected in S211.
S217: The visual field calculation function 19 calculates a display area of the HMD 1, that is, a visual field area of the user from the current position and direction of the HMD 1 and determines whether or not the position of the avatar calculated in S215 is within the display area. When it is determined that the avatar can be disposed in the display area of the HMD 1, the process proceeds to S218. When it is determined that the avatar is not capable of being disposed in the display area, a relief process in S300 is performed. In the relief process in S300, a guidance process that allows the user to change the viewing direction or to move to the position where the avatar can be seen is performed, which will be described in detail below with reference to
S218: When it is determined that the avatar can be displayed in the display area of the HMD 1, the avatar information is read from the avatar information storage function 16, and the avatar is generated. Then, the avatar is displayed on the display unit 72 of the HMD 1 at the display position and in the display direction of the avatar calculated in S215.
S219: It is determined whether or not the displayed current point number m is the imaging point k with reference to the still image data storage table 810 or the moving image data storage table 820. When it is determined that the trajectory point is the imaging point, the process proceeds to S220. When it is determined that the trajectory point is not the imaging point, the process proceeds to S222.
S220: It is notified that the current time is the imaging point. As a method for notifying the imaging point, the color of the displayed avatar is changed or the avatar is blinked to notify the user of the imaging point.
S221: The corresponding imaging data is read from the still image data storage table 810 or the moving image data storage table 820 and is then reproduced in response to an instruction from the user. The reproduction of the imaging data is performed by a voice command “Reproduction”. The imaging data may be reproduced on the entire display screen of the HMD. However, in this example, the imaging data is displayed on the reduced sub-screen. The imaging data is continuously displayed for a certain period of time and then the reproduction of the imaging data ends.
S222: It is determined whether or not the user inputs an instruct to end the avatar display process. For example, it is determined by a voice command “Avatar End” that the instruction to end the avatar display process is input. When the instruction to end the avatar display process is input, the process proceeds to S226 to end the avatar display process.
S223: It is determined whether the unit time has elapsed. When it is determined that the unit time has elapsed, the process proceeds to S224. When it is determined that the unit time has not elapsed, the process returns to S222.
S224: 1 is subtracted from point number m to obtain a new point number. Then, the trajectory point returns to the previous trajectory point in time.
S225: It is determined whether the value of new point number m is less than 0 (that is, whether the value exceeds the starting point m (=0) of the trajectory). When the value of point number m is not less than 0, the process returns to S214, and the display of the avatar is continued for the next trajectory point. When the value of point number m is less than 0, the display of the avatar is completed at all of the trajectory points. Therefore, the process proceeds to S226, and the avatar display process ends.
S301: The visual field calculation function 20 determines whether or not the avatar can be disposed by changing the direction of the HMD 1. When the avatar can be disposed, the process proceeds to S302. When the avatar is not capable of being disposed, the process proceeds to S303.
S302: Since this corresponds to a case in which the HMD is not oriented in the direction in which the avatar is present, the user is guided by voice or display to change the direction of the HMD. For example, the voice decoding unit 83 of the voice processing unit 80 generates voice guides such as “Right” and “Left”. At that time, the avatar may be displayed on the screen and may be controlled to utter the words. After that, the process returns to S211.
Here,
S303: When the avatar is not capable of being disposed by changing the direction of the HMD 1 in S301, it is considered that the current position is far from any point where the trajectory is collected. Therefore, it is determined whether or not map information can be acquired in order to guide the user to a proximity point. For example, when a navigation application can be used, the process proceeds to S304. When the map information is not capable of being acquired, the process proceeds to S306.
S304: Route analysis to the nearest trajectory point s searched in S212 is performed using the navigation application.
S305: Route information (new trajectory point) up to the trajectory point s is added to the trajectory information storage table 710. After that, the process returns to S211, and the trajectory points including the added trajectory point become the positions where the avatar is to be displayed. As a result, the avatar is disposed in the display area of the HMD in S217 and can be displayed in S218.
S306: When the map information is not capable of being acquired in S303, a movement direction to the nearest trajectory point s is presented to the user. A presentation method is performed by voice or AR display. After that, the process returns to S211. The user can move in the presented direction to approach the nearest trajectory point s. As a result, the avatar is disposed in the display area in S217 and can be displayed in S218.
Here,
As described above, when it is determined that the avatar is not capable of being disposed in the display area of the HMD, the relief process in S300 can be performed to guide the user to the position and direction where the avatar is seen. Therefore, the user can move (track) without losing sight of the avatar.
The avatar 3 is displayed on the display screen 400 illustrated in
The display screen 400 illustrated in
In this example, the trajectory points (● marks) and the traveling directions (arrows) are displayed on the display screen 400 in order to inform the user of the return route. However, the trajectory points and the traveling directions may not be displayed.
Further, the trajectory points of the avatar are displayed every unit time (every one second). However, the invention is not limited thereto. The trajectory points may be displayed at other time intervals. For example, when the trajectory points are interpolated and displayed every 1/24 seconds, the avatar is displayed as a smooth moving image like a movie.
Next, an operation method of the user for the HMD 1 will be described. As described above, in this embodiment, a voice command is used as the operation of the user.
The voice command classification 601 is divided into an item related to trajectory collection and an item related to avatar display. The voice commands 602 related to trajectory collection include “Start of Trajectory” and “Trajectory Start” for starting the trajectory collection process, “End of Trajectory” and “Trajectory End” for ending the trajectory collection process, “Imaging” and “Recording” for capturing still images, “Imaging Start” and “Recording Start” for starting the capture of moving images, and “Imaging End” and “Recording End” for ending the capture of moving images.
The voice commands 602 related to avatar display include “Start of Avatar” and “Avatar Start” for starting the avatar display process, “End of Avatar” and “Avatar End” for ending the avatar display process, and “Reproduction” and “Imaging Data” for reproducing still images or moving images. It goes without saying that the voice commands listed here are examples and can be appropriately set according to the user's preference.
On the other hand, the HMD 1 can make a voice response corresponding to the voice command issued by the user using the voice synthesis function of the voice decoding unit 83 of the voice processing unit 80. As a result, for example, it is possible to respond with the same voice (echolalia) as the voice command of the user or to make a voice response for confirming the voice command of the user. For example, when the user's response to an inquiry “XX, isn't it?” from the HMD is “Yes”, the HMD 1 responds with a confirmation voice “Execute XX”. When the user's response is “No”, the HMD 1 responds with a confirmation voice “Cancel XX”. In addition, the voice commands “Yes” and “No” used by the user in the response are not illustrated in
Further, the voice synthesis function of the voice decoding unit 83 can be used to generate a synthetic voice such as a voice “Right” for prompting rotation to the right and a voice “Left” for prompting rotation to the left. This can be used in the process of S306 in
An operation instruction for the HMD 1 may be achieved by a method other than the voice commands. For example, an operation menu is displayed on the display screen of the HMD 1, and the user selects an item by a gesture (the movement of a hand or a finger). In this case, the imaging unit 71 of the image processing unit 70 can capture an image of the user's finger, recognize the movement of the finger from the captured image, and use it for the operation.
When the user brings a finger 415 closer to the display screen 410 from the outside of the HMD 1 in order to select, for example, the “video” menu 412, the imaging unit 71 detects a finger image 415 that gradually increases toward the display screen 410. Further, the imaging unit 71 recognizes that the finger image 415 is heading toward the “video” menu 412 and determines that “video” is selected and designated. As other operations, only a “Start” menu is simply displayed when the trajectory collection or the avatar display starts, and an “End” menu is simply displayed when the trajectory collection or the avatar display ends. Then, the imaging unit 71 detects the approach of the finger image 415 of the user and determines that a start instruction or an end instruction is input.
In addition to this, a method may be used in which a touch sensor is provided in the operation input unit 90 of the HMD 1 to detect that the user touches the HMD 1 with a finger or hand.
According to Embodiment 1, it is possible to guide the user, who has forgotten the place where the user stopped off when moving or the return route, so as to easily understand the return route.
In Embodiment 1, the position coordinates of the HMD are as three-dimensional coordinates (X, Y, Z). However, in Embodiment 2, a case in which the position coordinates are stored as two-dimension coordinates (X, Y) will be described. The basic configuration of an HMD according to Embodiment 2 is the same as that in Embodiment 1, and the differences from Embodiment 1 will be described.
Since a change in the height direction is small in a normal moving state on a flat ground, the avatar can be displayed on the basis of the planar position coordinates even though data in the height direction is omitted. A trajectory collection process is performed according to the same flowchart (S100) as that in Embodiment 1 (
In this example, since the unit time is set to one second, point number 1 indicates difference position coordinates (ΔX1, ΔY1) one second after the start of the trajectory collection process, point number k indicates difference position coordinates (ΔXk, ΔYk) k seconds after the start, and point number n indicate difference position coordinates (ΔXn, ΔYn) at the time when the trajectory collection process ends (n seconds after the start). A method for calculating the difference position coordinates is the same as that in Embodiment 1 (
The two-dimensional starting point/end point position coordinate table 750 includes, as items, starting point/end point distinguishment 751 for distinguishing between the starting point and the end point of the trajectory collection and position coordinates 752 for indicating each position information item as an absolute value, and is the same as that in
In Embodiment 2, only the two-dimensional position coordinates are dealt with. However, when the altitude changes, the following process is performed.
When the user goes up (down) a hill, a change (ΔX, ΔY) in the planar position coordinates of the HMD is linear, and a change (ΔZ) in the altitude of the HMD is also linear. When the user goes up (down) stairs, a change (ΔX, ΔY) in the planar position coordinates of the HMD is the width of the stairs, and a change (ΔZ) in the altitude of the HMD is also the difference between the stairs.
In the case of an escalator, a change (ΔX, ΔY) in the planar position coordinates of the HMD is linear, and a change (ΔZ) in the altitude of the HMD (ΔZ) is also linear. The escalator is distinguished from the hill because the amount of change in acceleration in the escalator is smaller than that in the hill. In the case of an elevator, the planar position coordinates of the HMD hardly change (ΔX=0, ΔY=0), and only the altitude of the HMD changes (ΔZ≠0).
As described above, when the user moves on, for example, the hill, the stairs, the escalator, or the elevator, characteristic changes (ΔX, ΔY) and (ΔZ) occur in the detection values of the planar position coordinates and the altitude. Therefore, when the avatar is displayed, the change is captured, and route information (for example, the hill, the stairs, the escalator, or the elevator) is displayed. In addition, the avatar is displayed so as to be superimposed on the image of, for example, the hill, the stairs, the escalators, or the elevators, which makes it possible to display the avatar with a sense of realism. Furthermore, a great effect can be obtained by linking with map information such as an underground shopping mall map.
When the amount of change in the altitude is greater than a predetermined value (threshold value), the trajectory collection process using the two-dimensional trajectory information storage tables 740 and 750 ends to acquire a trajectory A. Then, the process newly proceeds to the trajectory collection process using the three-dimensional trajectory information storage tables 710 to 730 according to Embodiment 1 to acquire a trajectory B. After that, when the amount of change in the altitude falls within the threshold value, the trajectory collection process using the two-dimensional trajectory information storage tables 740 and 750 is restarted to acquire a trajectory C.
The use of this method makes it possible to reduce the amount of data stored as much as possible. Of course, the two-dimensional trajectory information storage tables 740 and 750 according to Embodiment 2 may be converted into the three-dimensional trajectory information storage tables 710 to 730 according to Embodiment 1.
In the above-described Embodiments 1 and 2, the outdoor space in which the GPS sensor 51 of the sensor unit 50 relatively easily acquires GPS information (position information) is assumed. However, there is a case in which the GPS information is not capable of being acquired as in underground malls and indoors. When the GPS information is not capable of being acquired, the GPS information is supplemented with information from, for example, the acceleration sensor 54 or the gyro sensor 55 of the sensor unit 50.
When the position information is not capable of being acquired from, for example, the GPS information from the starting point to the end point even once, the position coordinates of the starting point are stored as X0=0 and Y0=0. In this case, the position coordinates of the height are also stored as Z0=0. Further, the position coordinates of the end point are calculated by integrating all of the difference position coordinates from the starting point to the end point.
In Embodiment 1, when the avatar is displayed, the avatar is oriented in a direction opposite to the traveling direction and moves backward. In contrast, in Embodiment 3, the avatar is displayed so as to be oriented in the traveling direction.
Whether the avatar is oriented forward (Embodiment 3) or backward (Embodiment 1) depends on the user's preference. Therefore, it is more preferable that the user can select and set the display form of the orientation of the avatar.
In Embodiment 4, when there is a building or the like in the visual field of the HMD and the avatar is located behind the building, the avatar is hidden.
Therefore, when the avatar is displayed so as to be superimposed on a real space that can be visually recognized through the HMD, it is possible to perform realistic display without a sense of discomfort. When the entire avatar is hidden and is not displayed, there is a concern that it will be difficult to track the avatar after that. Therefore, the hidden portion of the avatar may be translucently displayed such that the avatar can be tracked.
The images of the building 521 and the building 522 may be displayed by using, for example, Street View (Google Inc.) which is an Internet service that provides a scene along the road as a panoramic photograph. In addition, the name of a building or the like attached to the building or the like may be used depending on the position and direction of the HMD 1.
In Embodiment 5, imaging is not performed in response to an instruction from the user, but is automatically performed.
The imaging unit 71 of the HMD starts imaging from the starting point, always captures images for a predetermined period of time (for example, a period of time that is twice the unit time between the trajectory points used in the trajectory collection process) from the present time using loop imaging, and temporarily stores the images. The images automatically captured by this operation are a moving image or still images for a predetermined period of time (for example, two seconds) before the present time. In the case of the still image, a sufficient function can be exhibited only by temporarily storing one or two still images between a trajectory point and the next trajectory point. Then, the image captured at a peculiar place, for example, the point where the movement direction is changed on the movement route remains.
A specific operation of the automatic imaging will be described with reference to the schematic diagram illustrating trajectory collection in
According to the imaging method of Embodiment 5, when the user changes the traveling direction, the images captured in the traveling direction immediately before the traveling direction is changed are stored. Therefore, it is possible to display the images most suitable to evoke the memory of the user.
In Embodiment 6, information related to, for example, the posture of the user is also collected in the trajectory collection process and is reflected when the avatar is displayed.
For example, the distance sensor 53 of the sensor unit 50 detects the distance from the ground or floor to the HMD 1 (that is, the height of the head of the user), and the distance is collected at each trajectory point. The detection value of the distance is compared with the height (known) of the user to estimate the posture (for example, a standing posture, a sitting posture, a crouching posture, or a sleeping posture) of the user who wears the HMD 1. The information of the posture is reflected in the avatar to be displayed, and a standing avatar, a sitting avatar, or the like is displayed.
Further, the geomagnetic sensor 52 of the sensor unit 50 detects the direction of the face of the user who wears the HMD 1. Information of the direction of the face is reflected in the avatar to be displayed such that the direction of the face of the avatar is changed at each trajectory point. In addition, the direction of the body of the avatar may be estimated on the basis of the direction of the face and then changed, or may be changed according to vector information (traveling direction) between the trajectory points. The display of the information makes it possible to display a more realistic avatar.
In Embodiment 7, the time when the user passed through each trajectory point is stored in the trajectory collection process, and the passage time is displayed on the display screen when the avatar is displayed.
As described above, according to Embodiment 7, since the time when the user passed through the currently displayed trajectory point is displayed, the time is effective information for evoking the memory of the user.
In Embodiment 8, a case in which a smartphone is used as a portable video display device instead of the HMD will be described. The smartphone has almost the same hardware configuration and software configuration as the HMD and can implement the same functions as the HMD.
As illustrated in (b), a camera 901 is provided on the back side to capture an outdoor scene. The smartphone 9 has a sensor for detecting a three-dimensional position or the direction of the camera 901 and can collect trajectory information similarly to the HMD.
As illustrated in (a), a display screen 902 into which a touch panel is incorporated is provided on the front side, and an image of an outdoor scene captured by the camera 901 is displayed on the display screen 902. Then, as in Embodiment 1 (
As compared to the HMD, in the case of the smartphone 9, it is necessary to aim the camera 901 at an object each time the imaging process is performed in the collection of the trajectory information, and an imaging operation becomes complicated as the number of imaging points increases. Further, for automatic imaging, there is a disadvantage that it is not realistic for the user to keep the imaging direction of the camera 901 constant. However, since the trajectory information can be collected by the existing smartphone, convenience is improved. However, since it is difficult for the user to move while performing imaging, it is more realistic for the user to independently perform imaging only at the main point.
Here, the smartphone is given as an example of the portable video display device. However, any device having an equivalent or similar hardware configuration or software configuration can implement the operation of this embodiment. For example, the invention may be applied to notebook PCs or tablet PCs.
In Embodiment 9, a video display system that is used by a plurality of users wearing the HMDs in cooperation with each other will be described.
The HMD 1a worn by the user 2a is connected to the external server 6 through a wireless router 4a and the network 5. In addition, the HMD 1b worn by the user 2b is connected to the same server 6 through a wireless router 4b and the network 5. When the distance between the HMD 1a (user 2a) and the HMD 1b (user 2b) is short, the wireless routers 4a and 4b can be shared.
In this case, the HMDs 1a and 1b collect the trajectories of the users 2a and 2b, respectively. Then, trajectory information of the user 2a wearing the HMD 1a and trajectory information of the user 2b wearing the HMD 1b are stored in the common server 6. Then, the trajectory information of the two users is read from the server 6 and is displayed as two avatars which are virtual images of the two users by the HMDs 1a and 1b, respectively. Therefore, it is possible to refer to each trajectory using the avatars.
As described above, according to Embodiment 9, a plurality of users can mutually check the trajectories of others. As a result, as an incidental effect, when another user goes out of sight during trajectory collection, it is possible to immediately know the position of another user.
As a practical aspect of this embodiment, portable information terminals may perform communication therebetween using a near field communication system, such as Bluetooth (registered trademark), without using the server 6 through the wireless router 4 or the network 5. For example, in a case in which a parent moves together with a child who carries a smartphone and the child goes missing, it is possible to immediately search for the child with the HMD or the smartphone of the parent.
Examples of the embodiments of the invention have been described above using Embodiments 1 to 9. However, it goes without saying that the configuration for achieving the technology of the invention is not limited to the above-described embodiments and various modification examples can be considered. For example, a portion of the configuration of a certain embodiment may be replaced with the configuration of another embodiment, and the configuration of another embodiment may be added to the configuration of a certain embodiment. All of these configurations belong to the scope of the invention. In addition, the numerical values and the messages appearing in the sentences and the drawings are just examples, and the effects of the invention may not be impaired even when different numerical values and messages are used.
Some or all of the above-mentioned functions of the invention may be implemented by hardware. For example, some or all of the functions are designed by integrated circuits. In addition, some or all of the functions may be implemented by software. For example, a microprocessor unit or the like interprets and executes programs for implementing each function or the like to implement the functions. Hardware and software may be used together. The software may be stored in the various programs 41 of the HMD 1 in advance at the time of product shipment. After the product is shipped, the software may be acquired from, for example, various server devices on the Internet. Further, the software provided by, for example, a memory card or an optical disk may be acquired.
In addition, the control lines and the information lines illustrated in the drawings are considered necessary for explanation and do not necessarily indicate all of the control lines and the information lines in the product. In practice, it can be considered that almost all configurations are connected to each other.
1 HMD (video display device)
4 Wireless router
9 Smartphone (video display device)
10 Main control unit
40 Storage unit
50 Sensor unit
51 GPS sensor
52 Geomagnetic sensor
53 Distance sensor
56 Altitude sensor
60 Communication processing unit
70 Image processing unit
71 Imaging unit
72 Display unit
80 Voice processing unit
90 Operation input unit
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2019/007184 | 2/26/2019 | WO | 00 |
