Patent Application
20170059871
The present invention relates to an information processing device that includes a head mounted display to enable head tracking display.
An information processing device (e.g., a video game device) has been known that enables, by using a head mounted display (HMD) incorporating a motion sensor, head tracking display (a function of following the orientation of the head of an operator and displaying a facing image on the head mounted display) of a global image (e.g., 360-degree panoramic image) (e.g., see Patent Document 1).
According to such an information processing device, an operator can enjoy various types of games (e.g., shooter games, racing games, sports games, and the like) while changing a visual field in any direction in a virtual space that is constituted by a global image (e.g., a full spherical panoramic image, a hemispherical panoramic image, a 360-degree horizontal panoramic image, a panoramic image in a given area thereof, or the like) that is prepared in advance in accordance with the game type.
In the information processing device of this kind, in order to enjoy play with more pleasure, an operation unit having a good operability (e.g., a hand-held remote controller) is required. This is because, although an operator wearing a head mounted display can be absorbed in a virtual space, the operator is completely isolated from a real space, which makes the operator unable to see his or her own hands manipulating the operation unit.
Moreover, in the information processing device of this kind, the operator can freely change a visual field by turning in any direction (longitudinal and lateral directions and further a vertical direction) and taking any inclined posture, and thus the operation unit has a demand for a more reliable operability in order to accurately indicate any direction and accurately indicate any motion, even in the visual field or the virtual space provided in such a manner.
In this regard, in the device described in Patent Document 1, a technique is employed in which a marker attached to a hand-held stick-shaped operation unit is captured by a camera, and a motion of the operation unit is recognized using an image acquired by the camera, but with such a technique, when the operator turns his or her back on the camera, and the marker disappears in the visual field of the camera, it is evident that the motion of the operation unit is no longer recognizable.
In addition, in the information processing device of this kind, it is possible in principle to progress a game using the whole area of a virtual space constituted by a global image, but what an operator can see is limited to an image (a space) in a given angle-of-view frame in the virtual space, the given angle-of-view frame being specified by the orientation of the head, and thus in order to indicate a direction or a movement using the stick-shaped operation unit, a special consideration of the mutual relation between both the spaces is requested.
The inventor obtained a finding that, at this point, it is possible to enjoy play with more pleasure by enabling the simultaneous display or alternatively switching display of an image in a given angle-of-view frame facing the orientation of the head of the operator, and an image in a given angle-of-view frame facing fixed direction such as a forward, rearward, rightward, and leftward direction or an image in a given angle-of-view frame facing a direction indicated by the stick-shaped operation unit.
The present invention is made in view of the above-mentioned technical background and has an objective to improve operability in information input in an information processing device that includes the head mounted display of this kind to enable head tracking display.
The other objectives, operations, and advantages of the present invention will become readily apparent to those skilled in the art upon referencing to the following description herein.
The above-mentioned technique problems can be solved by an information processing device that includes a head mounted display with the following configuration.
That is, the information processing device includes: an operation unit that is hand-held or wearable; a head mounted display that is mounted to the head of an operator in such a manner as to be disposed in front of the eyes of the operator; a first motion sensor that can detect a motion of the head of the operator; and an information processing unit that displays, on the display, an image in a predetermined angle-of-view frame in a preset global image, the predetermined angle-of-view frame corresponding to the orientation of the head of the operator identified using the first motion sensor, and executes a specific process corresponding to an indication of the operator identified by a motion of the operation unit, wherein the motion of the operation unit is detected using a second motion sensor incorporated in the operation unit.
With such a configuration, the information processing unit is a unit that executes a specific process of fulfilling a head tracking displaying function for a global image using a detection output from a first motion sensor (e.g., incorporated in a head mounted tool) simultaneously with recognizing and dealing with an indication of the operator using a detection output from a second motion sensor (incorporated in an operation unit), and thus the indication of the operator is not missed or misidentified depending on the orientation of the operator like a conventional device that recognizes the indication of the operator using a marker and a camera. Therefore, the operator can give an appropriate indication to the information processing unit using a motion of the operation unit (e.g., position, posture, and/or changes therein) from any posture or rotation angle, while freely looking around in the global image, and thus it is possible to significantly improve the operability of the information processing device of this kind.
Note that the “head mounted display” here is not limited to a display incorporated in a head mounted tool to be described later (including the display of a smartphone) and broadly means a display retained in front of the eyes of an operator without being supported by a hand, such as a display to be put on ears (e.g., a spectacles-type display), a display supported by a head band, and a display supported by a headgear.
According to a preferred embodiment of the present invention, when the indication of the operator indicates a certain direction by the motion of the operation unit (e.g., position, posture, and/or changes therein), the specific process may he a process of drawing a predetermined cursor on the global image in the indicated direction, or a process of comparing an indicated position on the global image in the indicated direction with a target position disposed in advance on the global image and determining that a predetermined event has occurred at least on condition that the indicated position coincides with the target position within a predetermined allowable range.
With such a configuration, by indicating a certain direction in a real space using the motion of the operation unit, it is possible to draw a predetermined cursor on the global image in the indicated direction in a virtual space regardless of whether or not any area in the global image is displayed on the display, or to compare an indicated position on the global image in the indicated direction with a target position disposed in advance on the global image and determine that a predetermined event has occurred at least on condition that the indicated position coincides with target position within a predetermined allowable range.
According to a preferred embodiment of the present invention, when the operator indicates a movement of a certain object by the motion of the operation unit, the specific process may be a process of drawing a figure corresponding to a path of the movement of the object at a corresponding position on the global image, or a process of comparing the path of the movement of the object with a target path disposed in advance on the global image and determining that a predetermined event has occurred at least on condition that the path of the movement of the object coincides with a target path within a predetermined allowable range.
With such a configuration, by indicating a movement of a certain object in the real space using the motion of the operation unit, it is possible to draw a figure corresponding to the path of the movement of the object at a corresponding position on the global image in the virtual space, or to compare the path of the movement of the object with a target path disposed in advance on the global image and determine that predetermined event has occurred at least on condition that the path of the movement of the object coincides with a target path in a predetermined allowable range.
According to a preferred embodiment of the present invention, the information processing unit is a unit that executes a computer program for implementing a shooter game, where the operation unit has an exterior shape in the form of a predetermined shooting weapon, and when the indication of the operator indicates a direction of a sight by the motion of the operation unit by the operator, the specific process may be a process of drawing a predetermined sight mark on the global image in the indicated direction, or a process of comparing a sight position on the global image in the indicated direction with a predetermined target position disposed in advance on the global image and determining that a hitting event has occurred at least on condition that the sight position coincides with the predetermined target position within a predetermined allowable range.
With such a configuration, by indicating a direction of the sight using a motion of the shooting weapon constituting the operation unit in the real space, it is possible to draw a predetermined sight mark on the global image in the indicated direction in the virtual space, or to compare a sight position on the global image in the indicated direction with a predetermined target position disposed in advance on the global image and determine that a hitting event has occurred at least on condition that the sight position coincides with the target position within a predetermined allowable range.
According to a preferred embodiment of the present invention, the information processing unit is a unit that executes a computer program for implementing a racing game, where the operation unit has an exterior shape in the form of a steering tool of a movable body, (e.g., a racing car, a motorcycle, a motorboat, a bicycle, or the like) and when the indication of the operator indicates a direction in which the movable body is to travel, by the motion of the operation unit, the specific process may be a process of drawing a figure corresponding to the movable body on the global image in such a manner that the figure travels in the indicated direction.
With such a configuration, by an operator indicating a direction in which the movable body is to travel, using a motion of the steering tool constituting the operation unit in the real space, it is possible to draw a figure corresponding to the movable body on the global image such that the movable body travels in the indicated direction in the virtual space.
According to a preferred embodiment of the present invention, the racing game is a car racing game, where the steering tool has an exterior shape in the form of a steering wheel of a racing car, and when the indication of the operator indicates a direction in which the racing car is to travel, by the motion of the operation unit, the specific process may be a process of drawing a figure corresponding to the racing car on the global image in such a manner that the figure travels in the indicated direction.
With such a configuration, by an operator indicating a direction in which the racing car is to travel, using a motion of a steering wheel constituting the operation unit in the real space, it is possible to draw a figure corresponding to the racing car on the global image such that the racing car travels in the indicated direction in the virtual space.
According to a preferred embodiment of the present invention, the information processing unit is a unit that executes a computer program for implementing a sports game involving an action of hitting or returning a ball with a ball-hitting tool such as a bat and a paddle, and when the indication of the operator indicates a swing movement of the ball-hitting tool by the motion of the operation unit by the operator, the specific process may be a process of drawing a figure corresponding to a path of the movement of the ball-hitting tool on the global image at a corresponding position, or a process of comparing the path of the movement of the ball-hitting tool with a target path disposed in advance on the global image and determining that a ball-hitting event has occurred at least on condition that the path of the movement of the ball-hitting tool coincides with the target path within a predetermined allowable range.
With such a configuration, by indicating a swing movement of the ball-hitting tool, using a motion of the ball-hitting tool constituting the operation unit in the real space, it is possible to draw a figure corresponding to the path of the movement of the ball-hitting tool, in the virtual space at the corresponding position on the global image, or by indicating a target path disposed on the panoramic image, using a motion of the ball-hitting tool constituting the operation unit in the real space, it is possible to cause a ball-hitting event to occur, the ball-hitting event being to be a condition for progressing the game.
According to a preferred embodiment of the present invention, the information processing unit may further include a function of displaying, on the display, an image in a predetermined angle-of-view frame in the global image, the predetermined angle-of-view frame corresponding to a direction indicated by the operation unit that is identified using the second motion sensor, instead of an image corresponding to the orientation of the head of the operator or together with an image facing the orientation of the head of the operator.
With such a configuration, by an operator indicating any direction with the operation unit using the second motion sensor, it is possible to freely make a search in any direction on the global image as if to light the direction by a searchlight.
According to a preferred embodiment of the present invention, the information processing unit may further include a function of displaying images in one or more predetermined angle-of-view frames in the global image, the one or more predetermined angle-of-view frames corresponding to one or more of directions such as a rightward, leftward, rearward, upward, and downward direction with respect to the orientation of the head of the operator identified using the first motion sensor, in one or more predetermined windows on the display together with an image corresponding to the orientation of the head of the operator.
With such a configuration, an operator can simultaneously observe images in one or more predetermined angle-of-view frames facing one or more directions such as rightward, leftward, rearward, upward, and downward direction with respect to the orientation of the head of the operator identified using the first motion sensor, in an image corresponding to the global image, and it is thus possible to perform the head tracking display while observing not only the front of the operator but also the situation in the other directions that are predetermined.
According to the present invention and a preferred embodiment thereof, the second motion sensor may include one or more of a triaxial acceleration sensor, a triaxial angular velocity sensor, and a triaxial geomagnetic sensor.
With such a configuration, for example, with the combination of the triaxial acceleration sensor and the triaxial geomagnetic sensor, the combination further including the triaxial angular velocity sensor, or the like, it is possible to detect the motion of the operation unit (e.g., position, orientation, and/or changes therein) with high accuracy as needed using the second motion sensor.
According to the present invention and a preferred embodiment thereof, the display retained on the head mounted tool, the first motion sensor, and the information processing unit are substituted with corresponding functions of a commercial smartphone.
With such a configuration, it is possible to implement the information processing device that includes the head mounted display according to the present invention at low cost only with the operation unit, the head mounted tool, and the smartphone, without separately including a personal computer or a dedicated game information processing device.
According to the present invention and a preferred embodiment thereof, the head mounted tool may include: a main body with a window opening that opens in front of eyes of one the wearer; a window plate that is hinged in such a manner as to open and close the window opening of the main body and open outward; a retainer that makes a smartphone retained on an inner side of the window plate with the screen of the smartphone turned inward; and an optical system that is interposed between the eyes of the wearer and the smartphone in a state where the window plate is closed.
With such a configuration, it is possible to configure the video game device according to the present invention at low cost only with the operation unit, the head mounted tool, and the smartphone, and moreover to make it easy to attach/detach the smartphone to/from the head mounted tool.
According to the present invention and a preferred embodiment thereof, the global image includes a full spherical panoramic image, a hemispherical panoramic image, a 360-degree panoramic image, or a panoramic image of a given area thereof.
With such a configuration, it is possible to make an optimal design in accordance with the dimensions of the virtual space requested to the game, the grade of a product, or the like.
The present invention seen from another aspect is a computer program causing a computer to function as an information processing unit in an information processing device, the information processing device comprising: a head mounted display that is mounted to a head of an operator in such a manner as to be disposed in front of eyes of the operator; a first motion sensor that can detect at least an orientation of the head of the operator; an operation unit in which a second motion sensor is incorporated, the operation unit being hand-held or wearable; and an information processing unit, wherein the information processing unit includes: a unit for displaying, on the display, an image in a predetermined angle-of-view frame in a preset global image, the predetermined angle-of-view frame corresponding to the orientation of the head of the operator identified using the first motion sensor; and a unit for executing a specific process corresponding to an indication of the operator identified by the motion of the operation unit that is detected using the second motion sensor.
According to a computer program having such a configuration, it is possible to cause the computer to function as an information processing device that constitutes the device according to the present invention.
An information processing device according to a more specific embodiment of the present invention may include: a head mounted tool; a smartphone that is in the head mounted tool and is disposed such that a display screen of the smartphone faces a front of eyes of a wearer through a predetermined optical system, the smartphone including a first motion sensor incorporated therein; and an operation unit that includes a second motion sensor incorporated therein, is hand-held or wearable, and further is to be in a wireless or wired connection with the smartphone, wherein the smartphone may include: a unit for displaying, on the display, an image in a predetermined angle-of-view frame in a preset global image, predetermined angle-of-view frame corresponding to the orientation of the head of the operator identified using the first motion sensor; and a unit for executing a specific process corresponding to an indication of the operator identified by the motion of the operation unit that is detected using the second motion sensor.
With such a configuration, it is possible to implement an inexpensive device constituted by the operation unit, the head mounted tool, and the smartphone only by installing a predetermined application program to the smartphone, without separately including a personal computer or a dedicated game information processing device.
At this point, the operation unit may be a unit having an exterior shape in a form of a tool that is used in a virtual space perceived through the display.
With such a configuration, an operator can indicate any direction or movement in the virtual space without incongruity while being absorbed in a virtual space through a picture on the display screen, by actually touching in the real space with skin the operation unit having an exterior shape that matches the virtual space.
At this point, the operation unit may have an exterior shape in a form of a gun and include: an operating element that is positioned in a trigger portion; an operation information generation unit that generates operation information corresponding to an operation of the operating element in response to the operation of the operating element; and a communication unit that transmits the operation information to the smartphone in a wired or wireless manner.
With such a configuration, it is possible to easily perform an action such as aiming a sight to any target and shooting the target in the virtual space through a natural action of taking aim with a gun, while being absorbed in the virtual space through a picture on the display screen, by actually touching in the real space with skin a gun-shaped operating element having an exterior shape that matches the game.
At this point, the second motion sensor may include one or more of a triaxial acceleration sensor, a triaxial angular velocity sensor, and a triaxial geomagnetic sensor.
With such a configuration, for example, with the combination of the triaxial acceleration sensor and the triaxial geomagnetic sensor, the combination further including the triaxial angular velocity sensor, or the like, it is possible to detect the position, orientation, and/or changes therein of the operation unit with high accuracy as needed using the second motion sensor.
At this point, the head mounted tool may include: a main body with a window opening that opens in front of eyes of one the wearer; a window plate that is hinged in such a manner as to open and close the window opening of the main body and open outward; a retainer that retains a smartphone on an inner side of the window plate with the screen of the smartphone facing inward; and an optical system that is interposed between the eyes of the wearer and the smartphone in a state where the window plate is closed.
With such a configuration, it is possible to implement the head mounted tool that makes it easy to attach or detach the smartphone and can cause the display screen of the smartphone to function as the display unit of the head mounted display.
The present invention seen from another aspect can he understood as an operation unit generally applicable to the information processing device of this kind. That is, the operation unit includes a main body that has an exterior shape in a form of a gun, wherein the main body includes: an operating element that is positioned in a trigger portion; an operation information generation unit that generates corresponding operation information in response to the operation of the operating element; motion sensor that includes one or more of a triaxial acceleration sensor, a triaxial angular velocity sensor, and a triaxial geomagnetic sensor; a motion information generation unit that generates motion information based on an output from the motion sensor; and a communication unit that sends out the operation information generated by the operation information generation unit and the motion information generated by the motion information generation unit to the outside in a wired or a wireless manner.
With such configurations, it is possible to provide an operating element that has a good operability and is suitable to the information processing device of this kind that enables head tracking display in a virtual space.
According to the present invention, it is possible to improve operability in information input in an information processing device that includes the head mounted display of this kind to enable head tracking display.
Hereinafter, there will be described in detail, with reference to the accompanying drawings, a preferable embodiment of an information processing device that includes a head mounted display according to the present invention, enabling head tracking display.
The operation unit 10 includes one or more operating elements (to be described later in detail with reference to
The head mounted tool 20 is mounted on a head 41 of the operator 40 such that a predetermined display device (not illustrated) is disposed in front of the operator 40's eyes, and incorporates a first motion sensor (to be described later in detail with reference to
The information processing unit 30 extracts image information corresponding to an image in a predetermined angle-of-view frame G (to be described later in detail with reference to
Additionally, the operation unit 10 newly includes a second motion sensor built therein (to be described later with reference to
Furthermore, the information processing unit 30 is arranged to have a function of reflecting an indicating motion of the operator, which is identified using the second motion sensor, on panoramic image in a predetermined mode, as will be described later with reference to the flow charts illustrated in
Here, the communication between the operation unit 10, the head mounted tool 20, and the information processing unit 30 may be wired communication using an electric cord or an optical fiber, or wireless communication such as Bluetooth and infrared communication.
It is obvious that the functions of the information processing unit 30 can be implemented by a personal computer (hereafter, abbreviated as a “PC”), an information processing device dedicated to game, or the like, and additionally by a portable intelligent information terminal (hereafter, referred to as a “smartphone”). At this point, by incorporating the smartphone itself into the head mounted tool 20 to make the display screen of the smartphone to function as a display device, it is possible to implement a video game device according to the present invention with a very simple configuration at low cost by connecting the operation unit 10 and the head mounted tool 20 with the smartphone incorporated therein through wired communication or wireless communication, as illustrated in
Next, some specific examples of the operation unit will be described.
The triaxial acceleration sensor 1603 is, as is well known to those skilled in the art, a sensor with which accelerations in three directions, an X axis, a Y axis, and a Z axis, are measured by one device, is able to detect a three dimensional acceleration can, and is available for the measurement of gravity (static acceleration). Many triaxial acceleration sensors are MEMS sensors, which are reduced in size by “Micro Electro Mechanical Systems (MEMS, also referred to as micromachines)”, in which a microscopic mechanical structure is integrated on a silicon substrate by micromachining techniques such as semiconductor manufacturing techniques and laser beam machining technique. The MEMS triaxial acceleration sensors are capable of measurement within a range of ± a few g's and referred to as a “low g” type, which can follow changes in acceleration ranging from 0 Hz to a few hundred Hz. The term “0 Hz” in this case means a state where only a gravitational acceleration is applied to a sensor, when the sum of X-axis, Y-axis, and Z-axis accelerating vectors can result in the measurement of a direction with respect to the ground. The MEMS triaxial acceleration sensors are roughly categorized into three types, a semiconductor piezoresistive triaxial acceleration sensor, an electrostatic triaxial acceleration sensor, and a thermal (gas temperature profile type) triaxial acceleration sensor, which differ in method of measuring acceleration. The semiconductor piezoresistive triaxial acceleration sensors measure acceleration by detecting the distortion of a diaphragm that occurs when the acceleration acts on a weight. The electrostatic triaxial acceleration sensor measures acceleration by making use of changes in electrostatic capacitance, and the thermal (gas temperature profile type) triaxial acceleration sensor measures acceleration by making use of the displacement of gas heated by a heater.
The triaxial angular velocity sensor 1604 is, as is well known to those skilled in the art, a kind of an inertial sensor that performs the measurement of a rotation angular velocity in orthogonal three axes (X axis, Y axis, and Z axis) and is also referred to as a gyro sensor. The angular velocity sensor measures the motion of rotation, which cannot be sensed by an acceleration sensor. Gyro sensors can be categorized according to methods for sensing the rotation. At present, gyro sensors most commonly installed in consumer equipment are IC-type vibration gyro sensors using a MEMS (Micro Electro Mechanical System) technique. An inertial sensor using the MEMS technique is configured by, as its name suggests, a technique that combines an element performing mechanical movement and an electronic circuit processing a signal from the element, so as to sense movement. The types of vibration gyro sensor include an electrostatic capacitive type that uses a silicon and a piezo type that uses a crystal or the other piezoelectric materials. Gyro sensors other than the vibration type include a geomagnetic type, an optical type, and a mechanical type. The three axes are typically defined as vertical, lateral, and longitudinal axes, and in many cases the vertical axis is referred to as a “yaw axis”, the lateral axis is referred to as a “pitch (pitching) axis”, and the longitudinal axis is referred to as a “roll axis”. All the vibration gyro sensors make use of Coriolis force (deflecting force) to sense rotation.
The triaxial geomagnetic sensor 1605 is, as is well known to those skilled in the art, a kind of a magnetic sensor and refers to a sensor that senses the direction of geomagnetism and calculates an orientation in the form of values on orthogonal three axes (X axis, Y axis, and Z axis). That is, this sensor includes first and second magnetic sensors for a longitudinal direction and a lateral direction, respectively, as well as a third geomagnetic sensor that detects geomagnetism in a vertical direction. Therefore, even when an electronic compass is inclined at an angle at which an operating element is held, the grasp of degrees at which the electronic compass is inclined enables the calculation of geomagnetism in a horizontal direction by subtracting the inclination and enables the display of a correct orientation. There are three known types of magnetic sensor element that is used in a geomagnetic sensor IC incorporated in a triaxial geomagnetic sensor. They are a magneto-resistive (MR) element, a magneto-impedance (MI) element, and a Hall element. The MR element makes use of the MR effect of, for example, a soft magnetic metallic material such as permalloy (NiFe) the resistance value of which changes with the intensity variation of an external magnetic field. The MI element makes use of the MI effect of an amorphous wire the impedance of which varies with the variation in the external magnetic field, where pulse current is caused to flow through the wire. The Hall element measures the variation in the external magnetic field by detecting a potential difference that develops by Hall effect in a semiconductor. A geomagnetic sensor IC typically incorporates such a magnetic sensor element, as well as a driving circuit, an amplifier circuit, and the like.
Next,
Next,
Next,
In
Next, the configuration of the head mounted tool 20 will be described. In this example, as described above, the head mounted tool 20 incorporates the smartphone 50, which includes an image display unit 51 functioning as a display unit of the head mounted display (HMD). In addition, the first motion sensor is constituted by a triaxial acceleration sensor 506, a triaxial angular velocity sensor 507, and a triaxial geomagnetic sensor 508 that are incorporated in the smartphone 50 (see
When the operator 40 wears the head mounted tool 20 with the smartphone 50 incorporated therein on the head 41, the display screen 51 of the smartphone 50 comes face-to-face with the eyes of the operator 40 through an optical system 28 that is retained by a partition plate 27, as illustrated in.
The control unit 501 includes a CPU or the like (including a System-on-Chip (SoC), a Micro Control Unit (MCU), a Field-Programmable Gate Array (FPGA), and the like) that executes a program for controlling various actions and the like of the smartphone.
The storage unit 502 is constituted by a ROM, a RAM, a storage, or the like, in which a program for fulfilling a function as a video game device to he described later, as well as a downloading program selection program for supporting the download of a game program through communication, and various kinds of data are saved.
The display 503 includes a display device such as a liquid crystal display, an organic EL display, and an inorganic EL display, displaying characters, figures, symbols, and the like. In connection with the present invention, the display 503 also functions as a display unit of a head mounted display (HMD). That is, a video constituting a video game is displayed on the display screen of the display 503. The video is preferably subjected to 3D processing using a well-known technique. In order to perform 3D display, the landscape display screen of the display 503 is divided into two sub areas, which display two respective images that have parallax.
The communication unit 514 performs wireless communication with an external device over a LAN, the Internet, and the like. There are wireless communication. standards such as 2G, 3G, 4G, IEEE 802.11, and Bluetooth®. Furthermore, the communication unit 514 has a built-in function of performing Bluetooth communication with the operation unit 10, 10A, or 10B, in connection with the present invention.
The speaker unit 517 outputs music or the like from the smartphone, and the speaker unit 517 is used to generate game sound, in connection with the present invention. The triaxial acceleration sensor 506 detects the direction and magnitude of acceleration that acts on a smartphone housing, the triaxial angular velocity sensor 507 detects the angle and the angular velocity of the smartphone housing, and the triaxial geomagnetic sensor 508 detects the direction of geomagnetism. The three sensors 506, 507, and 508 constitute the first motion sensor in the present invention. Then, based on the outputs from the three sensors 506, 507, and 508, the orientation of the head 41 of the operator 40 is detected.
Additionally, the vibration unit 513 is used in various application programs or the like, and can be constituted by, for example, a vibration speaker, a vibration mechanism using an electromagnetic coil, various vibrating motors, or the like. The camera unit 518 is connected to, for example, an in-camera, an out-camera, or the like provided on the front face or the back face of a terminal device housing, and is used to pick up a still image or a movie. The connector 519 is a terminal used for connection with the other device. The terminal may be a general-purpose terminal such as a USB, and an HDMI®. The audio output unit 515 performs audio output at the time of making a telephone call or the like using the smartphone. The audio input unit 516 (e.g., a microphone) performs audio input at the time of making a telephone call or the like using the smartphone. The inputting operation unit 504 is a touch-type inputting operation unit that detects a contact of a finger, a pen, a stylus pen, or the like to make various inputs. The GPS (Grand Positioning System) 509 detects the position of the smartphone. The illuminance sensor 510 detects an illuminance. Note that the illuminance refers to the intensity, brightness, or luminance of light. The timer 511 measures time. Note that the timer may be provided integrally with the CPU or the like. The battery 512 is charged to function as a power source of the smartphone.
In
Subsequently, a series of processes (steps 202 to 209) are repeatedly executed every predetermined cycle (e.g., every one frame time). First, in step 202, motion information and operation information generated by the operation unit 10 are read through Bluetooth communication with the operation unit 10. In subsequent step 203, the orientation of the head 41 of the operator 40 is determined based on the motion information from the first motion sensor (the triaxial acceleration sensor 506, the triaxial angular velocity sensor 507, and the triaxial geomagnetic sensor 508) incorporated in the smartphone 50. In subsequent step 204, operation details of the operation unit (about how the operating elements 13, 14, 15 are operated) are determined based on the operation information read from the operation unit 10. In subsequent step 205, determination as to whether the operation unit indicates a “direction” or a “movement” is made based on the motion information read from the operation unit 10. Afterward, an object updating process (step 206), an object drawing process (step 207), and a display image generating process (step 208) conforming to the type of the game are executed in order.
Here, the object updating process (step 206) means, as is well known to those skilled in the art, the execution of required update processes (e.g., updating an opponent, updating a player character or vehicle, updating a background, updating an obstacle, updating an effect, and the like) in accordance with the type of the game. Similarly, the object drawing process (step 207) means the execution of required drawing processes (e.g., drawing an opponent, drawing a player character or vehicle, drawing a background, drawing an obstacle, drawing an effect, and the like). Furthermore, the display image generating process (step 208) is a process for fulfilling a head tracking displaying function, which basically includes the extraction of an image in a predetermined angle-of-view frame G (see
That is, in the case where the indication is determined to be a “direction” (“DIRECTION” in step 301), first in step 302, sight coordinates are specified on a panoramic image in the indicated “direction”. In subsequent step 303, predetermined target coordinates on the panoramic image are read. In subsequent step 304, the sight coordinates specified in such a manner and the read target coordinates are compared. In subsequent step 305, it is determined whether or not the sight coordinates coincides with the target coordinates within a predetermined allowable range. When the coincidence is here determined (YES in step 305), the presence/absence of a trigger operation (an operation of pulling a trigger) is determined in subsequent step 306 based on the operation information that is read in advance from an operating element. When it is determined here that a trigger operation is present (YES in step 306), a hitting determination is made in subsequent step 307. In contrast, when the coincidence is out of the allowable range (NO in step 305), or when the coincidence is within the allowable range (YES in step 305), but the trigger operation is determined to be absent (No in step 306), the hitting determination is not made. The hitting determination here means, in a shooter game, for example, a bullet shot from a gun by an operator hitting an opponent.
In contrast, in the case where the indication is determined to be a “movement” (“MOVEMENT” in step 301), first in step 308, a path of the indicated “movement” on a panoramic image is specified. In subsequent step 309, a predetermined target path on the panoramic image is read. In subsequent step 310, the specified movement path and the read target path are compared. In subsequent step 311, it is determined whether or not the specified movement path coincides with the read target path within a predetermined allowable range. When the coincidence is here determined (YES in step 311), it is determined that a predetermined event occurs. In contrast, when the coincidence is out of the allowable range (NO in step 311), an event occurrence determination is not made. The predetermined event means, in a table tennis game, a baseball game, a golf game, or the like involving a ball-hitting action, for example, that a swing action of an operator coincides with a predetermined target swing action within an allowable range, and a ball-hitting action (e.g., a table tennis paddle striking a ball, a baseball bat striking a ball, a golf club striking a ball, and the like) are properly performed.
That is, in a shooter game, for example, as illustrated in
In contrast, in a sports game involving a ball-hitting action, as illustrated in
When the display mode is the first mode (“FIRST MODE” in step 501), a head facing image is extracted in step 502 from the panoramic image in accordance with the orientation of the head of an operator that is determined based on motion information acquired from the first motion sensor incorporated in the smartphone. In this case, the head facing image extracted in such a manner is determined as an output image being kept unchanged.
When the display mode is the second mode (“SECOND MODE” in step 501), a head facing image is extracted in step 504 from a panoramic image, as in step 502. In subsequent step 505, leftward, rightward, and rearward facing images are extracted, with the orientation of the head defined as a criterion. In subsequent step 506, the four images (the head facing image, the leftward image, the rightward image, and the rearward image) are composed so as to be projected on the same display screen. In this case, the composed image generated in such a manner is determined as an output image being kept unchanged.
When the display mode is the third mode (“THIRD MODE” in step 501), a head facing image is extracted in step 508 from a panoramic image, as in step 502. In subsequent step 509, an image in the direction indicated by an operating element is extracted from the panoramic image in accordance with the direction indicated by an operator that is determined based on motion information acquired from the second motion sensor incorporated in the operating element. In subsequent step 510, the head facing image and the image in the direction indicated by the operating element are composed so as to be projected on the same display screen.
In subsequent step 511, it is determined which image is specified from among “head facing”, “composition”, and “orientation of operation unit”, by the operator operating an operating element of the operating element. Here, when the operator specifies “head facing”, the head facing image is determined in step 512 as an output image being kept unchanged. Alternatively, when the operator specifies “composition”, the composed image of the head facing image and the image in the direction indicated by the operating element is determined in step 513 as an output image being kept unchanged. Furthermore, when the operator specifies “operating element indicating direction”, the image in the direction indicated by the operating element is determined in step 514 as an output image being kept unchanged.
Referring back to
[Mutual Relation between Detection Coordinate Spaces of First and Second Motion Sensors]
As illustrated in
As is obvious from these drawings, for example, when the operator 40 turns the whole body thereof over horizontal 360 degrees, squarely holding the gun-shaped operation unit 10, a sight mark 61 is drawn on a panoramic image on the extension of the sight of the gun-shaped operation unit 10. Therefore, the operator 40 manipulates the gun-shaped operation unit 10 at will while turning the line of sight in any direction in the virtual space constituted by the 360-degree panoramic image and the hemispherical panoramic image, and adjusts the sight to target graphics 60a and 60b, enabling the pleasure in the shooter game to be increased.
It is assumed that there are a terminal device (a smartphone) and a gun device (a gun-shaped operation unit) in one world coordinate space, and a motion sensor is incorporated in each device. Here, each motion sensor is constituted by a triaxial acceleration sensor, a triaxial angular velocity sensor, and a triaxial geomagnetic sensor. Each device can calculate the basic posture thereof using the triaxial acceleration sensor and the triaxial geomagnetic sensor. The posture of the device after moving is calculated by correcting the above-mentioned basic posture using the sensing result of the triaxial angular velocity sensor.
Using the triaxial acceleration sensor and the triaxial angular velocity sensor incorporated in each device, the posture of the terminal device and the posture of the gun device are calculated, and the postures are defined as the basic postures of the respective devices. Note that, even without the triaxial angular velocity sensor, posture control is possible using the triaxial acceleration sensor and the triaxial geomagnetic sensor.
sensor_pose=Quaternion(−_mAccel.Pose.z, _mAccel.Pose.y, −_mAccel.Pose.w);
Quaternion rotation =this.transform.rotation;
Quaternion qx=Quaternion.AngleAxis(−gyro.RotZ, Vector3.1
Quaternion qy=Quaternion.AngleAxis(−gyro.RotY, Vector3.up);
Quaternion qz=Quaternion,AngleAxis(−gyro.RotX, Vector3.forward);
q=(qz*qy*qx);
rotation=rotation*q;
Quaternion sensor_pose=clSingleton<clBLESensorManager>.instance.SenserPose;
this.transform.rotation=Quaternion.Slerp(rotation, sensor_pose, 0.04f);
Using the triaxial acceleration sensor and the triaxial geomagnetic sensor, the magnetic north direction of each object is calculated from the triaxial geomagnetic sensor.
At this point, to calculate the magnetic north of a gun object, components from the triaxial geomagnetic sensor contains inclination information, and thus the geomagnetic sensor information is corrected to compensate for the inclination of the terminal calculated from the acceleration, so as to extract a horizontal geomagnetism component.
Vector3b=Vector3(−mAccel.Gravity.x, _mAccel.Gravity.y, _mAccel.Gravity.z);
b=b.normalized;
Vector3 a=Vector3 (0f, 1f, 0f);
a=a.normalized;
float value=a.x*b.z+ay*b.y+a.z b,z;
float gravity_angle=Mathf.A cos(value/a.magnitude* b.magnitude);
Vector 3 axis=Vector 3.Cross(a, b);
axis=axis.normalized
Quaternion aq=Quaternion.AngleAxis(−gravity_angle* Mathf.Rad2Deg, axis);
Vector3 geom_v=new Vector3(−_mGeom.Z, _mGeom.Y, −_mGeom.X);
geom_v=geom_v.normalized;
Quaternion gq=Quaternion(−aq.z, −aq.x, aq.y, aq,w);
float m=Mathf.Sqrt(Mathf.Pow(geom.x, 2)+Mathf.Pow(geom.y, 2)+Mathf.Pow(geom,z, 2));
geom_height=Mathf.A sin(geom.z/m);
geom_height_head=Mathf.A cos(geom.x/ (m*Mathf.Cos(geom_height)));
The magnetic north direction of each object calculated in Second Step and the basic posture of each object are composed. At this point, the posture and the orientation of each object in the world space can be found.
sensorRotation=q_head*pose;
The world coordinates of an end point, being a given distance away in a forward direction from a current posture of the gun object, is calculated based on posture information on the gun object calculated in Third Step. Note that if the above calculated end point exists in the visual field of a camera of a terminal object, the world coordinates are converted into viewport coordinates of the camera (range: 0.0 to 1.0). The coordinates in a 2D coordinate system are used as 2D coordinates of the sight of the gun on the camera. When the viewport coordinates are out of the range of 0.0 to 1.0, the sight is determined to be out of screen and is not displayed.
Vector3ray_s_pos=gameObject.transform.position; _gunRay,SetPosition(0, ray_s_pos);
Vector3 ray_e_pos=gameObject.transform.forward* kRayLength;
Vector2 v2=ViewPortPos(r_e_pos);
Vector3 target_pos=_basePos+new Vector3((v2.x−0.5f), (v2.y−0.5f), 0.0f);
this.transform,position=Vector3.MoveTowards(this.transform,position, target_pos, 0.5f);
[Relation between Action by Operator and Picture on Display Screen]
At this point, as illustrated in the left half of
Next,
In contrast, as illustrated in the left half of
Next,
In contrast, as illustrated in the left half of
Next,
In this example, three small windows 51a, 51b, and 51c are disposed on the display screen 51. That is, the rearward display window 51a for displaying an image in a predetermined rearward angle-of-view frame is disposed in an upper portion of the display screen 51, the leftward display window 51b for displaying an image in a predetermined leftward angle-of-view frame is disposed in a lower left portion of the display screen 51, and the rightward display window 51c for displaying an image in a predetermined rightward angle-of-view frame is disposed in a lower right portion of the display screen 51.
At this point, as illustrated in the left half of
Next,
At this point, as illustrated in the left half of
Next,
At this point, as illustrated in the left half of
Next,
At this point, as illustrated in the left half of
As seen from above, in the present invention, in the state where the detection coordinate space of the first motion sensor (incorporated in the smartphone) is aligned with the detection coordinate space of the second motion sensor (incorporated in the operating element), the position, posture, and changes therein of the operating element that are determined based on the motion information from the second motion sensor are projected on the panoramic image in the form of various corresponding figures, and it is thus possible to enjoy the video game while manipulating various tools with sensation similar to that in the real space, in the virtual game space formed by the panoramic image.
Further, in the above embodiment, the information processing device according to the present invention, is implemented as a video game device, but the application of the device according to the present invention is not limited to this, and the information processing device can be implemented as a device to execute any computer software.
That is, the electric circuitry inside the stick-shaped operation unit 66, which is not illustrated, includes a control unit that is constituted by a microprocessor, an ASIC, or the like, and performs integrated control on the entire operating element, a communication unit that performs short range communication such as Bluetooth, three sensors that constitute the second motion sensor, namely, a triaxial acceleration sensor, a triaxial angular velocity sensor, and a triaxial geomagnetic sensor, a battery, a storage unit, and a switch that is turned on or off interlocking with a push button. Then, the above-mentioned action is achieved by the control unit executing the same processing as that illustrated in the flow chart of
According to the present invention, in an information processing device that includes a head mounted display of this kind to enable head tracking display, it is possible to improve operability in information input.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2015-169855 | Aug 2015 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2015/083910 | 12/2/2015 | WO | 00 |
