Patent Application
20210165558
The present invention relates to a display device, a display method, and a display program, in particular, relates to a display device, a display method, and a display program that present a state where an article is virtually placed in a room space to a user.
In the related art, there is a technique of virtually placing articles such as furniture in the room space and presenting the articles to the user.
JP2015-165420A discloses that the inside of the room is three-dimensionally measured to generate a virtual space modeled three-dimensionally, and the layout simulation of placing other articles such as furniture in the virtual space may be performed.
JP2003-085356A discloses that a device absent in the home is selected as a device to be purchased, and the position of a candidate for the device to be purchased can be moved by the user with the mouse to a desired position on the virtual home screen.
JP2017-519259A discloses that furniture in the virtual room is photo-realistically described such as image capture, or is similarly described, and the description includes the active/inactive state of the device.
In a case where the purchase of the article to be placed in the room is considered, such as furniture, home appliances, and interiors, by placing a virtual object representing the article in the virtual room space and presenting the virtual object to the user, the user can confirm the atmosphere or harmony in a case where the article is placed in the room, or the suitability of the article with respect to the size or space of the room, in advance.
However, in the related art, securing the placement space of the purchase consideration article is hindered due to the object placed in the real room, and thus the placement that maximizes the original attractiveness or value of the purchase consideration article cannot be proposed.
The present invention has been made in view of such problems, and an object thereof is to provide a display device, a display method, and a display program that can propose the attractiveness or value of a placement consideration article to a user by eliminating limitation in a virtual space of a layout of the placement consideration article due to an object actually placed in a room.
A display device according to a first aspect of the present invention comprises an image acquiring unit that acquires an image obtained by imaging an inside of a room, a measuring unit that measures space information regarding the room from the image acquired by the image acquiring unit, a placement consideration article selecting unit that allows a user to select a placement consideration article that is an article to be considered for new placement in the room, a generating unit that generates, based on the image and the space information, a virtual room space that is a space virtually representing the inside of the room, a virtual real space object virtually representing one or more articles placed in the room, and a virtual placement consideration article object virtually representing the placement consideration article selected by the user by the placement consideration article selecting unit in the virtual room space, a placing unit that decides optimum placement of the virtual placement consideration article object in the virtual room space, and decides placement of the virtual real space object in the virtual room space, and a display controller that displays the virtual room space in which the virtual real space object and the virtual placement consideration article object are placed by the placing unit on a display unit, in which in a case where optimum placement of the virtual placement consideration article object is not hindered even though the virtual real space object is placed at a virtual real space object initial position that is a position in the virtual room space corresponding to a position in the room of the one or more articles placed in the room, the placing unit places the virtual real space object at the virtual real space object initial position, and in a case where optimum placement of the virtual placement consideration article object is hindered due to placement of the virtual real space object at the virtual real space object initial position, the placing unit moves the virtual real space object from the virtual real space object initial position.
In the display device according to a second aspect of the present invention, in a case where optimum placement of the virtual placement consideration article object is hindered due to placement of the virtual real space object at the virtual real space object initial position, the placing unit may move the virtual real space object to an outside of the virtual room space.
In the display device according to a third aspect of the present invention, in a case where the virtual real space object is moved to the outside of the virtual room space, the display controller may do not display the moved virtual real space object on the display unit.
In the display device according to a fourth aspect of the present invention, in a case where optimum placement of the virtual placement consideration article object is hindered due to placement of the virtual real space object at the virtual real space object initial position, the placing unit may move the virtual real space object in the virtual room space.
In the display device according to a fifth aspect of the present invention, the display controller may display the virtual real space object in the virtual room space displayed on the display unit at a position optionally instructed by the user excluding a displayed position of the virtual placement consideration article object, and in a case where the position optionally instructed by the user is outside the virtual room space, may do not display the virtual real space object on the display unit.
The display device according to a sixth aspect of the present invention may further comprise an article database that stores information regarding one or more articles used for the placement consideration article selecting unit to allow the user to select the placement consideration article, in which the placement consideration article selecting unit allows the user to select information regarding any one or more articles from the information regarding the one or more articles stored in the article database.
In the display device according to a seventh aspect of the present invention, the information regarding the article may include information regarding an optimum layout of the article in the virtual room space, and the display controller may decide a virtual placement consideration article object initial position in the virtual room space based on the information regarding an optimum layout of the placement consideration article selected by the user, and may display the virtual placement consideration article object at the virtual placement consideration article object initial position.
In the display device according to an eighth aspect of the present invention, the article database may store a virtual placement consideration article object corresponding to one or more placement consideration articles, and the generating unit may extract a virtual placement consideration article object corresponding to the placement consideration article selected by the user from the article database to generate the virtual placement consideration article object.
In the display device according to a ninth aspect of the present invention, the generating unit may generate the virtual real space object corresponding to the one or more articles placed in the room based on a subject image of the one or more articles included in the image.
The display device according to a tenth aspect of the present invention may further comprise an object database that stores the virtual real space object corresponding to the one or more articles, in which the generating unit may extract the virtual real space object corresponding to the one or more articles placed in the room from the object database to generate the virtual real space object.
In the display device according to an eleventh aspect of the present invention, the virtual room space may include a wall, a floor, and a ceiling.
In the display device according to a twelfth aspect of the present invention, the one or more articles placed in the room may include at least one of a window, a curtain, or furniture.
In the display device according to a thirteenth aspect of the present invention, the placement consideration article may include a product that can be purchased by electronic commerce.
A display method according to a fourteenth aspect of the present invention, executed by a computer, comprises a step of acquiring an image obtained by imaging an inside of a room, a step of measuring space information regarding the room from the acquired image, a step of allowing a user to select a placement consideration article that is an article to be considered for new placement in the room, a step of generating, based on the image and the space information, a virtual room space that is a space virtually representing the inside of the room, a virtual real space object virtually representing one or more articles placed in the room, and a virtual placement consideration article object virtually representing the placement consideration article selected by the user in the virtual room space, a step of deciding optimum placement of the virtual placement consideration article object in the virtual room space, and deciding placement of the virtual real space object in the virtual room space, and a step of displaying the virtual room space in which the virtual real space object and the virtual placement consideration article object are placed, in which in a case where optimum placement of the virtual placement consideration article object is not hindered even though the virtual real space object is placed at a virtual real space object initial position that is a position in the virtual room space corresponding to a position in the room of the one or more articles placed in the room, the virtual real space object is placed at the virtual real space object initial position, and in a case where optimum placement of the virtual placement consideration article object is hindered due to placement of the virtual real space object at the virtual real space object initial position, the virtual real space object is moved from the virtual real space object initial position.
A display program according to a fifteenth aspect of the present invention allows a computer to execute the display method according to the fourteenth aspect.
According to the present invention, in a case where optimum placement of the article selected by the user is decided, the object in the real space which may hinder the placement of the article can be removed from the virtual room space. Therefore, in a case where the user considers new placement of the product to be purchased in the room, it is possible to propose the attractiveness or value of the product more than that of the article existing in the room to the user, and to reduce the loss of the sale opportunity of the product. Also, the total coordination of the room including the product to be purchased is possible, and cross-selling of the related products is possible.
The image acquiring unit 11 acquires an image obtained by imaging the inside of the room by a user using a device, such as a 3D camera, that acquires space information. The 3D camera is a device in which an infrared sensor is attached to a twin-lens camera or a single-lens camera, and can measure a distance to a target object. The 3D camera includes a device that acquires the space information by combining the single-lens camera and the motion sensor, or by analyzing the information obtained by the single-lens camera.
The space measuring unit 12 measures the space information regarding the room from the image acquired by the image acquiring unit 11. For example, the space measuring unit 12 measures a width between the right and left walls, a ceiling height, and a depth which are dimensions of the room space. The space measuring unit 12 measures the shape and the position coordinate of one or a plurality of articles placed in the room space.
The product selecting unit 13 allows the user to select a placement consideration article that is an article to be considered for new placement in the room. For example, the product selecting unit 13 outputs signals for displaying a product selection screen for the user to select information regarding any one or more articles from the information regarding one or more articles stored in the product database 18 on the display unit 14 to the display controller 15.
The virtual object generating unit 16 generates the virtually represented room space and the virtually represented object based on the room image acquired by the image acquiring unit 11 and the space information measured by the space measuring unit 12. The virtual object generating unit 16 generates the virtual room space that is a space virtually representing the inside of the room based on the room image acquired by the image acquiring unit 11 and the space information measured by the space measuring unit 12. The virtual room space includes a wall, a floor, and a ceiling. The virtual object generating unit 16 generates the virtual real space object which virtually represents one or more articles actually placed in the imaged room. The virtual object generating unit 16 generates the virtual real space object corresponding to the one or more articles placed in the room based on a subject image of the one or more articles included in the room image acquired by the image acquiring unit 11. The virtual object generating unit 16 extracts the virtual real space object corresponding to the one or more articles placed in the room from the object database 19 to generate the virtual real space object. Further, the virtual object generating unit 16 generates a virtual placement consideration article object which virtually represents a placement consideration article selected by the user by the product selecting unit 13 in the virtual room space. The virtual object generating unit 16 extracts the virtual placement consideration article object corresponding to the placement consideration article selected by the user from the product database 18 to generate the virtual placement consideration article object.
The layout deciding unit 17 places the virtual real space object and the virtual placement consideration article object in the virtual room space. For example, the layout deciding unit 17 places the virtual placement consideration article object at an optimum position for the virtual placement consideration article object in the virtual room space. The layout deciding unit 17 places the virtual real space object at the position (virtual real space object initial position) in the virtual room space, which corresponds to the position in which the article is placed on the room image acquired by the image acquiring unit 11. Also, for example, the layout deciding unit 17 places the virtual placement consideration article object at the position in the virtual room space optionally instructed by the user.
The display unit 14 displays the virtual room space in which the virtual real space object and the virtual placement consideration article object are placed by the layout deciding unit 17.
The display controller 15 moves the virtual real space object and the virtual placement consideration article object in the virtual room space which are displayed by the display unit 14 from the position in which the objects are placed by the layout deciding unit 17 to the position optionally instructed by the user.
The product database 18 stores information regarding one or more articles used for the product selecting unit 13 to allow the user to select the placement consideration article. The information regarding the article includes information regarding the optimum layout of the article in the virtual room space. The product database 18 stores the virtual placement consideration article object corresponding to one or more placement consideration articles.
The object database 19 stores the virtual real space object corresponding to one or more articles.
The image processing device 100 is configured by a mobile information terminal such as a smartphone, a tablet terminal, or a digital camera, or a personal computer. The camera used by the user in a case of imaging the inside of the room, the product database 18 and the object database 19 may or may not be integrated with the image processing device 100. The product database 18 and the object database 19 are provided in another personal computer or a server, and the image processing device 100 may be accessible thereto via a network.
In the present specification, “purchase” means a paid transfer, and “product” means a paid-transferred article, but the same processing as for the product can be performed with respect to the room space in which free-transferred article is to be installed.
In step S1, the image acquiring unit 11 acquires an image (room image) in which the inside of the room is captured. The room image is, for example, a three-dimensional image captured by the 3D camera. The room space is a room space in which a product is newly purchased and installed.
In step S2, the space measuring unit 12 measures the dimensions of the room space, and the shape and the position coordinate of the existing article which is one or a plurality of articles actually placed in the room space from the room image acquired by the image acquiring unit 11.
In step S3, the virtual object generating unit 16 which is a part of the generating unit generates the objects (virtual real space objects) OBJ2-1, OBJ2-2, and OBJ2-3 in the virtual room space S2, which correspond to the shapes of the real space objects OBJ1-1, OBJ1-2, and OBJ1-3 of one or a plurality of existing articles placed in the room space S1 measured by the space measuring unit 12, based on the object database 19 (see
For example, the virtual object generating unit 16 estimates the article name of the article corresponding to the article (same or similar article) based on the measured shape of the existing article by machine learning or the like. The virtual object generating unit 16 specifies and extracts the object data including the image data of the article corresponding to the estimated article name from the list of the object data of the article stored in the object database 19. The extracted object data includes the virtual real space object.
Alternatively, the virtual object generating unit 16 may generate the virtual real space object of the existing article by simplifying the article portion by reducing the color of the subject image of the existing article measured from the image or approximating the shape with a polygonal shape.
In step S4, the product selecting unit 13 selects the product name of any product (placement consideration article) from the product database 18 based on the operation by the user.
In step S5, the layout deciding unit 17 functioning as the placing unit decides the placement of the virtual placement consideration article object corresponding to the product selected in step S4 in the virtual room space S2, and the placement of the virtual real space object corresponding to the existing article measured in the room space S1 in the virtual room space S2. The details of the processing will be described below.
In step S6, the display controller 15 displays the virtual room space S2 in which the virtual placement consideration article object corresponding to the selected product name and the virtual real space object corresponding to the existing article measured in the room space S1 are placed on the display unit 14 in accordance with the placement decided by the layout deciding unit 17.
In step S7, the display controller 15 moves the displayed position of the virtual placement consideration article object of the product corresponding to the selected product name and/or the virtual real space object corresponding to one or a plurality of existing articles to any position in the virtual room space S2 in accordance with the operation of the user.
In step S100, the layout deciding unit 17 determines whether each existing article can be moved. For example, as shown in
In the room space S1 shown in
In step S101, the layout deciding unit 17 creates the virtual room space S2 that is a virtual room space in which the movable article is eliminated from the room image. The virtual room space S2 is a virtual space in which the measured three-dimensional real space, that is, the room space is converted into the coordinate of the room space represented by the three-dimensional computer graphics (CG).
In step S102, the layout deciding unit 17 decides the optimum placement of the object (virtual placement consideration article object) IT of the product selected in step S4 in the empty space V1 of the virtual room space S2. That is, the layout deciding unit 17 decides the virtual placement consideration article object initial position that is a position in which the virtual placement consideration article object is initially displayed in the virtual room space.
In step S103, the layout deciding unit 17 decides the placement of the virtual real space object corresponding to the movable article in the virtual room space S2 in a range in which the optimum placement of the virtual placement consideration article object IT is not hindered.
On the other hand, in a case where the virtual real space object OBJ2-1 of the movable article is placed at the virtual real space object initial position corresponding to the original position of the movable article in the room image, it hinders the position of the virtual placement consideration article object IT of the product (that is, the virtual real space object OBJ2-1 and the virtual placement consideration article object IT overlap each other). Therefore, the coordinates of the virtual real space object OBJ2-1 of the movable article are not provided, and an elimination state is decided to be the placement of the virtual real space object OBJ2-1. However, even in a case where both hinder each other, as long as the hindrance is eliminated by finely adjusting the coordinates corresponding to the original position of the virtual real space object OBJ2-1, the position after fine adjustment may be decided to be the placement of the virtual real space object OBJ2-1. That is, in a case where optimum placement of the virtual placement consideration article object IT is hindered due to placement of the virtual real space object OBJ2-1 at the virtual real space object initial position, the layout deciding unit 17 moves the virtual real space object OBJ2-1 from the virtual real space object initial position. Particularly, in a case where optimum placement of the virtual placement consideration article object IT is hindered due to placement of the virtual real space object OBJ2-1 at the virtual real space object initial position, the layout deciding unit 17 moves the virtual real space object OBJ2-1 to the outside of the virtual room space S2. Alternatively, in a case where optimum placement of the virtual placement consideration article object IT is hindered due to placement of the virtual real space object OBJ2-1 at the virtual real space object initial position, the layout deciding unit 17 moves the virtual real space object OBJ2-1 in the virtual room space S2. In a case where the virtual real space object OBJ2-1 is moved to the outside of the virtual room space S2, the display controller 15 does not display the moved virtual real space object OBJ2-1 on the display unit 14.
As shown in
As described above, in a case where the user decides the optimum placement in the room space of the product optionally selected from the product database 18, the virtual real space object OBJ2-1 corresponding to the real space object OBJ1-1 of the existing article in the room space, which hinders the optimum placement of the product can be eliminated from the virtual room space S2. Therefore, in a case where the user considers new placement of the product to be purchased in the room space S1, it is possible to propose the attractiveness or value of the product to be newly purchased to the user without the influence of the existing article (movable article) in the room space S1 and to reduce the loss of the sale opportunity of the product. Also, the total coordination of the room including the product to be purchased or existing article in the room space is possible, and cross-selling of the related products is possible.
From the above description, the display device according to the following supplementary note 1 can be grasped.
[Supplementary Note 1]
A display device comprising a processor, in which the processor acquires an image obtained by imaging an inside of a room, measures space information regarding the room from the acquired image, allows a user to select a placement consideration article that is an article to be considered for new placement in the room, generates, based on the image and the space information, a virtual room space that is a space virtually representing the inside of the room, a virtual real space object virtually representing one or more articles placed in the room, and a virtual placement consideration article object virtually representing the placement consideration article selected by the user by the placement consideration article selecting unit in the virtual room space, decides optimum placement of the virtual placement consideration article object in the virtual room space, and decides placement of the virtual real space object in the virtual room space, and displays the virtual room space in which the virtual real space object and the virtual placement consideration article object are placed, and in a case where optimum placement of the virtual placement consideration article object is not hindered even though the virtual real space object is placed at a virtual real space object initial position that is a position in the virtual room space corresponding to a position in the room of the one or more articles placed in the room, the virtual real space object is placed at the virtual real space object initial position, and in a case where optimum placement of the virtual placement consideration article object is hindered due to placement of the virtual real space object at the virtual real space object initial position, the virtual real space object is moved from the virtual real space object initial position.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2018-168944 | Sep 2018 | JP | national |
This application is a Continuation of PCT International Application No. PCT/JP2019/027466 filed on Jul. 11, 2019, which claims priority under 35 U.S.C § 119(a) to Japanese Patent Application No. 2018-168944 filed on Sep. 10, 2018. Each of the above application(s) is hereby expressly incorporated by reference, in its entirety, into the present application.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2019/027466 | Jul 2019 | US |
| Child | 17170102 | US |
