Patent Application
20170352091
The field of the invention relates to methods for generating a 3D virtual body model of a person combined with a 3D garment image, as well as to related devices, systems and computer program products.
When selling clothes, clothing shops or stores tend to display a sample of the clothes on mannequins so that customers may view the sample of the clothes in a way that mimics how the clothes might look on the customer. Such a viewing is inherently a 3D experience, because a viewer can move through the shop or store, or move around the mannequin, while looking at the clothed mannequin, so as to view the garment on the mannequin from various perspectives. Displaying clothing from different perspectives is a highly desirable goal: fashion houses use models who walk up and down a catwalk to display the items of clothing. When a model walks up and down a catwalk, a viewer is automatically presented with a large number of perspectives of the items of clothing, in 3D. However, using fashion models to display items of clothing at a fashion show is a time consuming and an expensive undertaking.
It is known to show items of clothing on a 3D body model on a computer screen. But it is desirable to provide a technical solution to the problem that showing items of clothing on a 3D body model on a computer screen does not replicate in a simple and low cost way the technical experience of viewing items of clothing on a mannequin while moving through a clothes shop or store, or while moving around the mannequin, or while viewing a model walking up and down a catwalk.
There are some aspects of shopping for clothes in which the available options are far from ideal. For example, if a user wants to decide what to buy, she may have to try on various items of clothing. When wearing the last item of clothing and viewing themselves in a mirror in a fitting room, the user then has to decide, from memory, how that item of clothing compares to other items of clothing she has already tried on. And because she can only try on one outfit at a time, it is physically impossible for the user to compare herself in different outfits at the same time. A user may also like to compare herself in an outfit near to another user (possibly a rival) in the same outfit or in a different outfit. But another user may be unwilling to participate in such a comparison, or it may be impractical for the other user to participate in such a comparison. It is desirable to provide an improved way of comparing outfits, and of comparing different users in different outfits.
It is known to show items of clothing on a 3D body model on a computer screen, but because of the relatively detailed view required, and because of the many options which may be necessary to view a desired item of clothing on a suitable 3D body model, and because of typically the requirement to register with a service which offers viewing of garments on 3D body models, mobile computing devices have hitherto been relatively unsuitable for such a task. It is desirable to provide a method of viewing a selected item of clothing on a 3D body model on a mobile computing device which overcomes at least some of these problems.
WO2012110828A1, GB2488237A and GB2488237B, which are incorporated by reference, disclose a method for generating and sharing a 3D virtual body model of a person combined with an image of a garment, in which:
(a) the 3D virtual body model is generated from user data;
(b) a 3D garment image is generated by analysing and processing multiple 2D photographs of the garment; and
(c) the 3D garment image is shown super-imposed over the 3D virtual body model. A system adapted or operable to perform the method is also disclosed.
EP0936593B1 discloses a system which provides a full image field formed by two fixed sectors, a back sector and a front sector, separated by a mobile part sector formed by one or more elements corresponding to the rider clothing and various riding accessories. The mobile part sector, being in the middle of the image, gives a dynamic effect to the whole stamping thus creating a macroscopic, dynamical, three-dimensional sight perception. To obtain the correct sight view of the mark stamping a scanner is used to receive three-dimensional data making part of the physical model: motorcycle and rider. Subsequently the three-dimensional data at disposal as well as the mark stamping data are entered in a computer with a special software, then the stated data are processed to obtain a complete image of the deforming stamping as the said image gets the characteristics of the data base or surface to be covered. The image thus obtained is applied in the curved surface without its sight perception getting altered.
According to a first aspect of the invention, there is provided a method for generating a 3D virtual body model of a person combined with a 3D garment image, and displaying the 3D virtual body model of the person combined with the 3D garment image on a screen of a computing device, the computing device including a sensor system, the method including the steps of:
(a) generating the 3D virtual body model;
(b) generating the 3D garment image for superimposing on the 3D virtual body model;
(c ) superimposing the 3D garment image on the 3D virtual body model;
(d) showing on the screen the 3D garment image superimposed on the 3D virtual body model;
(e) detecting a position change using the sensor system, and
(f) showing on the screen the 3D garment image superimposed on the 3D virtual body model, modified in response to the position change detected using the sensor system.
An advantage is that a user is provided with a different view of a 3D garment superimposed on a 3D virtual body model, in response to modifying their position, which technically is similar to a user obtaining a different view of a garment on a mannequin, as the user moves around the mannequin. The user may alternatively tilt the computing device, and be provided with a technically similar effect.
The method may be one wherein the modified 3D garment image superimposed on the 3D virtual body model shown on the screen is modified in perspective.
The method may be one wherein 3D virtual body model image modification is provided using a sequence of pre-rendered images. An advantage is that the required computing time between position change and providing the modified image is reduced.
The method may be one wherein the 3D virtual body model is shown to rotate by use of a progressive sequence of images depicting the 3D virtual body model at different angles.
The method may be one wherein the position change is a tilting of the screen surface normal vector. An advantage is that a user does not have to move; instead they can simply tilt their computing device.
The method may be one wherein the sensor system includes an accelerometer. The method may be one wherein the sensor system includes a gyroscope. The method may be one wherein the sensor system includes a magnetometer.
The method may be one wherein the a user is given the feeling of being able to move around the sides of the 3D virtual body model by tilting the computing device.
The method may be one wherein the sensor system includes a camera of the computing device. A camera may be a visible light camera. A camera may be an infra red camera.
The method may be one wherein the sensor system includes a pair of stereoscopic cameras of the computing device. An advantage is improved accuracy of position change detection.
The method may be one wherein the position change is a movement of a head of a user. An advantage is that technically the user moves in a way that is the same or similar to how they would move to view a real object from a different angle.
The method may be one wherein the position change is detected using a head tracker module.
The method may be one wherein the user is given the feeling of being able to move around the sides of the 3D virtual body model by moving their head around the computing device.
The method may be one wherein the images and other objects on the screen move automatically in response to user head movement.
The method may be one wherein the computing device is a mobile computing device.
The method may be one wherein the mobile computing device is a mobile phone, or a tablet computer, or a head mounted display. A mobile phone may be a smartphone.
The method may be one wherein the mobile computing device asks a user to rotate the mobile computing device, in order to continue. An advantage is that the user is encouraged to view the content in the format (portrait or landscape) in which it was intended to be viewed.
The method may be one wherein the computing device is a desktop computer, or a laptop computer, or a smart TV, or a head mounted display. Use of a smart TV may include use of an active (shuttered glasses) 3D display, or of a passive (polarising glasses) 3D display.
The method may be one wherein the 3D virtual body model is generated from user data.
The method may be one wherein the 3D garment image is generated by analysing and processing one or multiple 2D photographs of a garment.
The method may be one wherein the screen shows a scene, in which the scene is set with the midpoint of the 3D virtual body model's feet as the pivot point, so the user is given the impression of moving around the model to see the different angles.
The method may be one wherein a scene consists of at least three images: the 3D body model, a distant background, and a floor.
The method may be one wherein background images are programmatically converted into a 3D geometry.
The method may be one wherein a distant part of the background is placed independently of the floor section, with the distant image placed as a vertical plane, and the floor image oriented such that the top of the floor image is deeper than the bottom of the floor image.
The method may be one wherein the background and floor images are separated, by dividing a background image at a horizon line.
The method may be one wherein a depth value for each background image is set and stored in metadata for a resource of the background image.
The method may be one wherein within the screen, a scene is presented within a frame to keep it separate from other features, and the frame crops the contents so that when zoomed in or rotated significantly, edge portions of the scene are not visible.
The method may be one wherein a stereo vision of the 3D virtual body model is created on a 3D display device, by generating a left-eye/right-eye image pair with 3D virtual body model images rendered in two distinct rotational positions.
The method may be one wherein the 3D display device is an active (shuttered glasses) 3D display, or a passive (polarising glasses) 3D display.
The method may be one wherein the 3D display device is used together with a smart TV.
The method may be one wherein a user interface is provided including a variety of settings to customize sensitivity and scene appearance.
The method may be one wherein the settings include one or more of: iterate through available background images, iterate through available garments for which images are stored, set a maximum viewing angle, set a maximum virtual avatar image rotation to be displayed, set an increment by which the virtual avatar image should rotate, set an image size to be used, zoom in/out on the virtual avatar and background section of a main screen.
The method may be one wherein when a 3D textured geometry of the 3D virtual body model and the 3D garment dressed on the 3D virtual body model are all present, generating a render with a rotated 3D virtual body model is implemented by applying a camera view rotation along the vertical axis during the rendering process.
The method may be one wherein when 2D garment models are used for outfitting, generating a rotated version of 2D garment models involves first approximating the 3D geometry of the 2D garment model based on assumptions, performing a depth calculation and finally a corresponding 2D texture movement is applied to the image in order to emulate a 3D rotation.
The method may be one wherein for a 2D torso-based garment model with a single 2D texture cut-out or silhouette, the 3D geometry model of the garment is approximated by applying the following simplifications: around the upper body, the garment closely follows the geometry of the underlying body shape; around the lower body, the garment approximates to an elliptic cylinder with varying axis lengths, centred at the origin of the body.
The method may be one including the steps of: generating a smooth 3D mesh with faces from a point cloud of vertices given by depth approximations at each pixel, and generating a final normalised depth map of the garment for a required view.
The method may be one wherein the depth map is used to calculate the extent to which a given point on the garment texture needs to move in the image in order to simulate an out-of-plane rotation about the vertical axis.
The method may be one wherein an underlying head and neck base geometry of the user's 3D body shape model is used as an approximate 3D geometry and modeling a 3D rotation of the head sprite/hairstyle from a single 2D texture image using an approach of 2D texture morphing and morph field extrapolation is performed.
According to a second aspect of the invention, there is provided a computing device including a screen, a sensor system and a processor, the computing device configured to generate a 3D virtual body model of a person combined with a 3D garment image, and to display the 3D virtual body model of the person combined with the 3D garment image on the screen, in which the processor:
(a) generates the 3D virtual body model;
(b) generates the 3D garment image for superimposing on the 3D virtual body model;
(c) superimposes the 3D garment image on the 3D virtual body model;
(d) shows on the screen the 3D garment image superimposed on the 3D virtual body model;
(e) detects a position change using the sensor system, and
(f) shows on the screen the 3D garment image superimposed on the 3D virtual body model, modified in response to the position change detected using the sensor system.
The computing device may be further configured to perform a method of any aspect of the first aspect of the invention.
According to a third aspect of the invention, there is provided a system including a server and a computing device in communication with the server, the computing device including a screen, a sensor system and a processor, the server configured to generate a 3D virtual body model of a person combined with a 3D garment image, and to transmit to the computing device an image of the 3D virtual body model of the person combined with the 3D garment image, in which the server:
(a) generates the 3D virtual body model;
(b) generates the 3D garment image for superimposing on the 3D virtual body model;
(c) superimposes the 3D garment image on the 3D virtual body model;
(d) transmits the image of the superimposed the 3D garment image on the 3D virtual body model to the computing device;
and in which the computing device:
(e) shows on the screen the 3D garment image superimposed on the 3D virtual body model;
(f) detects a position change using the sensor system, and
(g) transmits to the server a request for a 3D garment image superimposed on the 3D virtual body model, modified in response to the position change detected using the sensor system;
and in which the server
(h) transmits an image of the superimposed the 3D garment image on the 3D virtual body model to the computing device, modified in response to the position change detected using the sensor system;
and in which the computing device:
(i) shows on the screen the 3D garment image superimposed on the 3D virtual body model, modified in response to the position change detected using the sensor system.
The system may be further configured to perform a method of any aspect according to the first aspect of the invention.
According to a fourth aspect of the invention, there is provided a computer program product executable on a computing device including a processor, the computer program product configured to generate a 3D virtual body model of a person combined with a 3D garment image, and to provide for display the 3D virtual body model of the person combined with the 3D garment image, in which the computer program product is configured to:
(a) generate the 3D virtual body model;
(b) generate the 3D garment image for superimposing on the 3D virtual body model;
(c) superimpose the 3D garment image on the 3D virtual body model;
(d) provide for display on a screen the 3D garment image superimposed on the 3D virtual body model;
(e) receive a detection of a position change using a sensor system, and
(f) provide for display on the screen the 3D garment image superimposed on the 3D virtual body model, modified in response to the position change detected using the sensor system.
The computer program product may be further configured to perform a method of any aspect according to a first aspect of the invention.
According to a fifth aspect of the invention, there is provided a method for generating a plurality of 3D virtual body models, each 3D virtual body model combined with a respective different 3D garment image, and displaying the plurality of 3D virtual body models, each combined with the respective different 3D garment image, in a single scene, on a screen of a computing device, the method including the steps of:
(a) generating the plurality of 3D virtual body models;
(b) generating the respective different 3D garment images for superimposing on the plurality of 3D virtual body models;
(c) superimposing the respective different 3D garment images on the plurality of 3D virtual body models, and
(d) showing on the screen in a single scene the respective different 3D garment images superimposed on the plurality of 3D virtual body models.
Because a scene is provided in which respective different 3D garment images are superimposed on the plurality of 3D virtual body models, an advantage is that such a scene may be assembled relatively quickly and cheaply, which are technical advantages relative to the alternative of having to hire a plurality of models and clothe them in order to provide an equivalent real-life scene. A further advantage is that a user may compare herself in a particular outfit to herself in various other outfits, something which would be physically impossible, because the user cannot physically model more than one outfit at a time.
The method may be one wherein the plurality of 3D virtual body models is of a plurality of respective different people. An advantage is that a user may compare herself in a particular outfit to other users in her social group in various outfits, without having to assemble the real people and actually clothe them in the outfits, something those real people may be unavailable to do, or unwilling to do.
The method may be one wherein the plurality of 3D virtual body models is shown at respective different viewing angles.
The method may be one wherein the plurality of 3D virtual body models is at least three 3D virtual body models. An advantage is that more than two models may be compared at one time.
The method may be one wherein a screen image is generated using a visualisation engine which allows different 3D virtual body models to be modelled along with garments on a range of body shapes.
The method may be one wherein 3D virtual body models in a screen scene are distributed in multiple rows.
The method may be one wherein within each row the 3D virtual body models are evenly spaced.
The method may be one wherein the screen scene shows 3D virtual body models in perspective.
The method may be one wherein garments are allocated to each 3D virtual body model randomly, or pre-determined by user input, or as a result of a search by a user, or created by another user, or determined by an algorithm.
The method may be one wherein the single scene of a set of 3D virtual body models is scrollable on the screen. The method may be one wherein the single scene of a set of 3D virtual body models is horizontally scrollable on the screen.
The method may be one wherein a seamless experience is given by repeating the scene if the user scrolls to the end of the set of 3D virtual body models.
The method may be one wherein the single scene is providable in profile or in landscape aspects.
The method may be one wherein the screen is a touch screen.
The method may be one wherein touching an outfit on the screen provides details of the garments.
The method may be one wherein touching an outfit on the screen provides a related catwalk video.
The method may be one wherein the scene moves in response to a user's finger sliding horizontally over the screen.
The method may be one wherein with this operation, all the body models in the screen move with predefined velocities to generate the effect of a translational camera view displacement in a perspective scene.
The method may be one wherein by applying different sliding speeds to different depth layers in the scene, a perspective dynamic layering effect is provided.
The method may be one wherein a horizontal translation of each 3D virtual body model is inversely proportional to a depth of each 3D virtual body model in the scene.
The method may be one wherein when a user swipes, and their finger lifts off the touchscreen, the all layers gradually halt.
The method may be one wherein the scene switches to the next floor, upstairs or downstairs, in response to a user sliding their finger over the screen, vertically downwards or vertically upwards, respectively.
The method may be one wherein after the scene switches to the next floor, the 3D virtual body models formerly in the background come to the foreground, while the 3D virtual body models formerly in the foreground move to the background.
The method may be one wherein a centroid position of each 3D virtual body model follows an elliptical trajectory during the switching transformation.
The method may be one wherein in each floor, garments and/or outfits of a trend or a brand are displayable.
The method may be one wherein a fog model, with respect to the translucency and the depth of the 3D virtual body models, is applied to model the translucency of different depth layers in a scene.
The method may be one wherein the computing device includes a sensor system, the method including the steps of
(e) detecting a position change using the sensor system, and
(f) showing on the screen the 3D garment images superimposed on the 3D virtual body models, modified in response to the position change detected using the sensor system.
The method may be one wherein the modification is a modification in perspective.
The method may be one wherein the position change is a tilting of the screen surface normal vector.
The method may be one wherein the sensor system includes an accelerometer.
The method may be one wherein the sensor system includes a gyroscope.
The method may be one wherein the sensor system includes a magnetometer.
The method may be one wherein the sensor system includes a camera of the computing device. A camera may be a visible light camera. A camera may be an infra red camera.
The method may be one wherein the sensor system includes a pair of stereoscopic cameras of the computing device.
The method may be one wherein the position change is a movement of a head of a user.
The method may be one wherein the position change is detected using a head tracker module.
The method may be one wherein the images and other objects move automatically in response to user head movement.
The method may be one wherein the computing device is a mobile computing device.
The method may be one wherein the mobile computing device is a mobile phone, or a tablet computer, or a head mounted display.
The method may be one wherein the mobile computing device is a mobile phone and wherein no more than 3.5 3D virtual body models appear on the mobile phone screen.
The method may be one wherein the computing device is a desktop computer, or a laptop computer, or a smart TV, or a head mounted display. Use of a smart TV may include use of an active (shuttered glasses) 3D display, or of a passive (polarising glasses) 3D display.
The method may be one wherein the 3D virtual body models are generated from user data.
The method may be one wherein the 3D garment images are generated by analysing and processing one or multiple 2D photographs of the garments.
The method may be one wherein in the scene, a floor and a background are images that makes it look like the crowd is in a particular location.
The method may be one wherein a background and a floor can be chosen by the user or customized to match some garment collections.
The method may be one wherein a lighting variation on the background is included in the displayed scene.
The method may be one wherein a user can interact with the 3D virtual body models to navigate through the 3D virtual body models.
The method may be one wherein selecting a model allows the user to see details of the outfit on the model.
The method may be one wherein the user can try the outfit on their own 3D virtual body model.
The method may be one wherein selecting an icon next to a 3D virtual body model allows one or more of: sharing with others, liking on social media, saving for later, and rating.
The method may be one wherein the 3D virtual body models are dressed in garments and ordered according to one or more of the following criteria: Garments that are most liked; Garments that are newest; Garments of the same type/category/style/trend as a predefined garment; Garments that have the user's preferred size available; Garments of the same brand/retailer as a predefined garment; sorted from the most recently visited garment to the least recently visited garment.
The method may be one wherein a user can build up their own crowd and use it to store a wardrobe of preferred outfits.
The method may be one wherein a user interface is provided which is usable to display the results from an outfit search engine.
The method may be one wherein the method includes a method of any of aspect according to the first aspect of the invention.
According to a sixth aspect of the invention, there is provided a computing device including a screen and a processor, the computing device configured to generate a plurality of 3D virtual body models, each 3D virtual body model combined with a respective different 3D garment image, and to display the plurality of 3D virtual body models, each combined with the respective different 3D garment image, in a single scene, on the screen of the computing device, in which the processor:
(a) generates the plurality of 3D virtual body models;
(b) generates the respective different 3D garment images for superimposing on the plurality of 3D virtual body models;
(c) superimposes the respective different 3D garment images on the plurality of 3D virtual body models, and
(d) shows on the screen in a single scene the respective different 3D garment images superimposed on the plurality of 3D virtual body models.
The computing device may be configured to perform a method of any aspect according to a fifth aspect of the invention.
According to a seventh aspect of the invention, there is provided a server including a processor, the server configured to generate a plurality of 3D virtual body models, each 3D virtual body model combined with a respective different 3D garment image, and to provide for display the plurality of 3D virtual body models, each combined with the respective different 3D garment image, in a single scene, in which the processor:
(a) generates the plurality of 3D virtual body models;
(b) generates the respective different 3D garment images for superimposing on the plurality of 3D virtual body models;
(c) superimposes the respective different 3D garment images on the plurality of 3D virtual body models, and
(d) provides for display in a single scene the respective different 3D garment images superimposed on the plurality of 3D virtual body models.
The server may be configured to perform a method of any aspect according to a fifth aspect of the invention.
According to a eighth aspect of the invention, there is provided a computer program product executable on a computing device including a processor, the computer program product configured to generate a plurality of 3D virtual body models, each 3D virtual body model combined with a respective different 3D garment image, and to provide for display the plurality of 3D virtual body models, each combined with the respective different 3D garment image, in a single scene, in which the computer program product is configured to:
(a) generate the plurality of 3D virtual body models;
(b) generate the respective different 3D garment images for superimposing on the plurality of 3D virtual body models;
(c) superimpose the respective different 3D garment images on the plurality of 3D virtual body models, and
(d) provide for display in a single scene the respective different 3D garment images superimposed on the plurality of 3D virtual body models.
The computer program product may be configured to perform a method of any aspect according to a fifth aspect of the invention.
According to a ninth aspect of the invention, there is provided a method for generating a 3D virtual body model of a person combined with a 3D garment image, and displaying the 3D virtual body model of the person combined with the 3D garment image on a screen of a mobile computing device, in which:
(a) the 3D virtual body model is generated from user data;
(b) a garment selection is received;
(c) a 3D garment image is generated of the selected garment, and
(d) the 3D garment image is shown on the screen super-imposed over the 3D virtual body model.
The method may be one in which garment size and fit advice is provided, and the garment selection, including a selected size, is received.
The method may be one in which the 3D garment image is generated by analysing and processing one or multiple 2D photographs of the garment.
The method may be one in which an interface is provided on the mobile computing device for a user to generate a new user account, or to sign in via a social network.
The method may be one in which the user can edit their profile.
The method may be one in which the user can select their height and weight.
The method may be one in which the user can select their skin tone.
The method may be one in which the user can adjust their waist and hip size.
The method may be one in which the method includes a method for generating a plurality of 3D virtual body models, each 3D virtual body model combined with a respective different 3D garment image, and displaying the plurality of 3D virtual body models, each combined with the respective different 3D garment image, in a single scene, on the screen of the mobile computing device, the method including the steps of:
(a) generating the plurality of 3D virtual body models;
(b) generating the respective different 3D garment images for superimposing on the plurality of 3D virtual body models;
(c) superimposing the respective different 3D garment images on the plurality of 3D virtual body models, and
(d) showing on the screen in a single scene the respective different 3D garment images superimposed on the plurality of 3D virtual body models.
The method may be one in which an icon is provided for the user to ‘like’ an outfit displayed on a 3D body model.
The method may be one in which by selecting a 3D body model, the user is taken to a social view of that particular look.
The method may be one in which the user can see who created that particular outfit and reach the profile view of the user who created that particular outfit.
The method may be one in which the user can write a comment on that outfit.
The method may be one in which the user can ‘Like’ the outfit.
The method may be one in which the user can reach a ‘garment information’ view.
The method may be one in which the user can try the outfit on their own 3D virtual body model.
The method may be one in which because the body measurements for the user's 3D virtual body model are registered, the outfit is displayed as how it would look on the user's body shape.
The method may be one in which there is provided a scrollable section displaying different types of selectable garments and a section displaying items that the 3D virtual body model is wearing or has previously worn.
The method may be one in which the screen is a touch screen.
The method may be one in which the 3D virtual body model can be tapped several times and in so doing rotates in consecutive rotation steps.
The method may be one in which the user can select to save a look.
The method may be one in which after having saved a look the user can choose to share it with social networks.
The method may be one in which the user can use hashtags to create groups and categories for their looks.
The method may be one in which a parallax view is provided with 3D virtual body models belonging to the same category as a new look created.
The method may be one in which a menu displays different occasions; selecting an occasion displays a parallax crowd view with virtual avatars belonging to that particular category.
The method may be one in which a view is available from a menu in the user's profile view, which displays one or more of: a parallax view showing the outfits the user has created together with statistics showing the number of looks the user has, the number of likes on different outfits, the number of followers and how many people the user is following.
The method may be one in which selecting followers displays a list of all the people following the user together with the option to follow them back.
The method may be one in which there is provided an outfitting recommendation mechanism, which provides the user with a list of garments which are recommended to combine with the garment(s) the user's 3D virtual body model is wearing.
The method may be one in which recommendation is on an incremental basis and it is approximately modelled by a first-order Markov model.
The method may be one in which for each other user who has appeared in the outfitting history, the frequency of each other user's outfitting record is weighted based on the similarity of the current user and each other user; then the weights of all similar body shapes are accumulated for recommendation.
The method may be one in which a mechanism is used in which the older top-ranking garment items are slowly expired, tending to bring more recent garment items into the recommendation list.
The method may be one in which recommendations are made based on other garments in a historical record which are similar to a current garment.
The method may be one in which a recommendation score is computed for every single garment in a garment database, and then the garments are ranked to be recommended based on their recommendation scores.
The method may be one in which the method includes a method of any aspect according to a first aspect of the invention, or any aspect according to a fifth aspect of the invention.
According to a tenth aspect of the invention, there is provided a system including a server and a mobile computing device in communication with the server, the computing device including a screen, and a processor, in which the system generates a 3D virtual body model of a person combined with a 3D garment image, and displays the 3D virtual body model of the person combined with the 3D garment image on the screen of the mobile computing device, in which the server
(a) generates the 3D virtual body model from user data;
(b) receives a garment selection from the mobile computing device;
(c) generates a 3D garment image of the selected garment,
(d) superimposes the 3D garment image over the 3D virtual body model, and transmits an image of the 3D garment image superimposed over the 3D virtual body model to the mobile computing device,
and in which the mobile computing device
(e) shows on the screen the 3D garment image super-imposed over the 3D virtual body model.
The system may be configured to perform a method of any aspect according to a ninth aspect of the invention.
According to an eleventh aspect of the invention, there is provided a method for generating a 3D garment image, and displaying the 3D garment image on a screen of a computing device, the method including the steps of:
(a) for a 2D torso-based garment model with a single 2D texture cut-out or silhouette, the 3D geometry model of the garment is approximated by applying the following simplifications: around the upper body, the garment closely follows the geometry of the underlying body shape; around the lower body, the garment approximates to an elliptic cylinder with varying axis lengths, centred at the origin of the body;
(b) showing on the screen the 3D garment image.
An example implementation is in a digital media player and microconsole, which is a small network appliance and entertainment device to stream digital video/audio content to a high definition television set. An example is Amazon Fire TV.
The method may be one wherein the computing device includes a sensor system, including the steps of:
(c) detecting a position change using the sensor system, and
(d) showing on the screen the 3D garment image, modified in response to the position change detected using the sensor system.
The method may be one for generating a 3D virtual body model of a person combined with the 3D garment image, including the steps of:
(e) generating the 3D virtual body model;
(f) showing on the screen the 3D garment image on the 3D virtual body model.
The method may be one including the steps of: generating a smooth 3D mesh with faces from a point cloud of vertices given by depth approximations at each pixel, and generating a final normalised depth map of the garment for a required view.
The method may be one wherein the depth map is used to calculate the extent to which a given point on the garment texture needs to move in the image in order to simulate an out-of-plane rotation about the vertical axis.
The method may be one wherein an underlying head and neck base geometry of the user's 3D body shape model is used as an approximate 3D geometry and modeling a 3D rotation of the head sprite/hairstyle from a single 2D texture image using an approach of 2D texture morphing and morph field extrapolation is performed.
According to an twelfth aspect of the invention, there is provided a system including a server and a computing device in communication with the server, the computing device including a screen, a sensor system and a processor, the server configured to generate a 3D virtual body model of a person combined with a 3D garment image, and to transmit to the computing device an image of the 3D virtual body model of the person combined with the 3D garment image, in which the server:
(a) generates the 3D virtual body model;
(b) generates the 3D garment image for superimposing on the 3D virtual body model;
(c) superimposes the 3D garment image on the 3D virtual body model;
(d) transmits the image of the superimposed the 3D garment image on the 3D virtual body model to the computing device;
and in which the computing device:
(e) shows on the screen the 3D garment image superimposed on the 3D virtual body model;
(f) detects a position change using the sensor system, and
(g) transmits to the server a request for a 3D garment image superimposed on the 3D virtual body model, modified in response to the position change detected using the sensor system;
and in which the server
(h) transmits an image manipulation function (or parameters for one) relating to an image of the superimposed 3D garment image on the 3D virtual body model to the computing device, modified in response to the position change detected using the sensor system;
and in which the computing device:
(i) applies the image manipulation function to the image of the 3D garment image superimposed on the 3D virtual body model, and shows on the screen the 3D garment image superimposed on the 3D virtual body model, modified in response to the position change detected using the sensor system.
The system may be one configured to perform a method according to any aspect of the first aspect of the invention.
Aspects of the invention will now be described, by way of example(s), with reference to the following Figures, in which:
We introduce a number of user interfaces for virtual body shape and outfitting visualisation, size and fit advice, and garment style recommendation, which help improve users' experience in online fashion and e-commerce. As typical features, these user interfaces 1) display one or more 3D virtual avatars which are rendered by a body shape and outfitting visualisation engine, into a layout or scene with interactive controls, 2) provide users with new interactive controls and visual effects (e.g. 3D parallax browsing, parallax and dynamic perspective effects, stereo visualisation of the avatars), and 3) embed a range of different recommendation features, which will ultimately enhance a user's engagement in the online fashion shopping experience, help boost sales, and reduce returns.
As a summary, the following three user interfaces are disclosed:
A unified and compact user interface that integrates a user's body shape visualisation, outfitting, garment size and fit advice, and social network and recommendation features.
A user interface with a crowd of virtual avatars shown to the user. These people/avatars can be in different outfits, have different body shapes, and may be shown from different view angles. A number of visual effects (e.g. 3D parallax browsing) and recommendation features may be associated with this user interface. The user interface can for example be implemented on both a desktop computer and on a mobile platform.
This user interface generates a user experience in which one is given the feeling of being able to move around the sides of the virtual avatar for example by either moving one's head around the mobile phone, or simply turning the phone in one's hand. In an example, the user interface may be used to generate stereo image pairs of the virtual avatar in a 3D scene for 3D display.
Technical details and underlying algorithms to support the features of the above user interfaces are detailed in the remaining sections.
This document describes applications that may run on a mobile phone or other portable computing device. The applications or their user interfaces may allow the user to
The applications may be connected to the internet. A user may access all or some of the content also from a desktop application.
An application may ask a user to rotate a mobile device (eg. from landscape to portrait, or from portrait to landscape), in order to continue. Such a step is advantageous in ensuring that the user views the content in the most appropriate device orientation for the content to be displayed.
Section 1: The “Wanda” User Interface
The “Wanda” user interface is a unified and compact user interface which integrates virtual body shape visualisation, outfitting, garment size and fit advice, and social network and recommendation features. Major example product features of the Wanda user interface are detailed below.
1.1 Account Creation/Renewal
A first thing a user may have to do is to log on, such as to an app or in the user interface, and create a user. An example of a workflow of this process can be seen in
1.2 Edit Profile View
After signing up, the user may fill in a name and choose a username. See
1.3 Adding Measurements
Height, weight and bra size may be shown in a separate view which is reached from the edit profile view. See
Weight may be shown in either or both stones and kilos, and may be displayed in a scrollable list where the user taps and chooses relevant weight. The user may then automatically be taken to the bra size measurements which may be completed in the same manner as the previous two measurements. See
From the edit profile view, the user may reach the settings for adjusting skin tone to their virtual avatars. A selection of models with different skin tones are available where the user can choose whichever model suits them best. See
1.4 ‘All Occasions’ View
When finished with the profile and body shape settings, saving the profile may take the user to the ‘all occasions’ view. See
1.5 Parallax View
The parallax view can be scrolled horizontally where a variety of virtual avatars wearing different outfits are displayed.
Next to the virtual avatars there can be icons. One of the icons which may be available is for the user to ‘like’ an outfit displayed on a virtual avatar. In one implementation this is shown as a clickable heart icon together with the number of ‘likes’ that an outfit has received. See
There may be several different parallax views showing crowds of different categories. From any parallax view, a new look may be created such as by choosing to create a completely new look or to create a new look based on another virtual avatar's look. See for example
1.6 Viewing Someone Else's Look
By tapping on an outfit worn by a virtual avatar in a parallax view, the user may be taken to a social view of that particular look. For one implementation, see
As seen in
From the Garment information view, a clothes item may be selected which takes the user to a specific view regarding that garment. See
If the user selects different sizes, the app or user interface may tell the user how it thinks the garment will fit at the bust, waist, and hips. For example, the app or user interface could say that a size 8 may have a snug fit, a size 10 the intended fit and size 12 a loose fit. The same size could also fit differently over the different body sections. For example it could be snug over the hip but loose over the waist.
There are different ways for the user to create new looks. To create a new look from a social view, the user may tap the option to try the outfit on. See
From the same view, the user may reach an edit outfit view either by swiping left or by tapping one of the buttons displayed along the right hand side of the screen.
1.7 Edit Look View
From this view, as shown for example in
The section with selectable garments (eg.
To the side of the selectable garments there may be a selection of tabs related to garment categories, which may let the user choose what type of garments to browse through, for example coats, tops, shoes.
Once the user finishes editing with their outfit they can swipe from left to right to hide the edit view and better display the new edited outfit on the user's virtual avatar. See
The virtual avatar can be tapped several times and in so doing rotate in consecutive rotation steps, as illustrated for example in
The user can select to save the look. See
1.8 Menu
At the top of the screen there is a menu. One implementation of the menu is shown in
The menu also gives access to the user's liked looks where everything the user has liked is collected. See for example
There is access to the user's ‘my style’ section which is a parallax view showing looks that other users have created and which the user is following. The same feed will also show the user's own outfits mixed in with these other followed users' outfits. For one implementation, see
1.9 Profile View
Another view available from the menu is the user's profile view. The profile view may display a parallax view showing the outfits the user has created together with statistics showing the number of looks the user has, the number of likes on different outfits, the number of followers and how many people the user is following. An example of this is shown in
The area displaying the statistics can be tapped to get more information than just a number. For example, tapping on followers displays a list of all the people following the user together with the option to follow them back, or to unfollow (see eg.
In the profile view (eg.
1.10 Outfitting Recommendation
Associated with the ‘Wanda’ user interface, we introduce an outfitting recommendation mechanism, which provides the user with a list of garments which are recommended to combine with the garment(s) the user's virtual avatar is wearing.
We explore the historical data warehouse (e.g. the render logs), which stores a list of records containing pairwise information of: 1) the user identifier u, which can be used to look up user attribute data including body measurement parameters, demographic information, etc., and 2) the outfit combination O tried on, which is in the format of a set of garment identifiers {ga, gb, gc, . . . }. Examples of outfitting data record are given as follows:
In the outfitting model, we assume that the user adds one more garment to the current outfit combination on the virtual avatar each time. The recommendation is on an incremental basis and hence it can be approximately modelled by a first-order Markov model. To perform the recommendation, we first try to build an outfit relation map list M for all users who have appeared in the historical data. Each item in M will be in the format of
The outfit relation map list M is populated from the historical data H with the following Algorithm 1:
1 Initialize M={ }
2 For each record entry (user: u, outfit: O) in the historical data H:
3 For each subset S of the outfit combination O (including φ but excluding O itself):
4 For each garment g in O\S,
5 If an entry with keys {{outfit: S, garment: g}, {user: u, frequency: f}} already exists in M,
6 Update the entry with an incremental frequency f+1:
7 Else,
8 Insert a new entry {(outfit: S, garment: g), {user: u, frequency: 1}} to M.
This population process is repeated over all the users in the render history and can be computed offline periodically.
In the recommendation stage, we assume that a new user u* with the current outfit combination O* is trying to pick up a new garment in the virtual fitting room, where the new garment has appeared in the historical record. Recommendation score R(g′) for an arbitrary new garment g* not in the current outfit O* is computed by aggregating all the frequencies fu of the entries with the same outfit-garment keys (outfit O*, garment g*) in the list M for all existing users u in the historical data D using the following equations.
R(g′)=wg
The time weight wg
Given each user u who has appeared in the outfitting history, we weight the frequency of a user u's outfitting record based on the similarity of the current user u* and u. The similarity of two users u and u′ is defined as follows:
s(u,u′)=1/(1+d(b(u),b(u′))), (1.2)
where b(u) is a feature vector of user u (i.e. body metrics or measurements such as height, weight, bust, waist, hips, inside leg length, age, etc), and d (.,.) is a distance metric (e.g. Euclidean distance of two measurements vectors). We then accumulate the weights of all similar body shapes for recommendation.
For online fashion, it is preferable to recommend more recently available garment items. To achieve that, we could also weight the each garment candidate with its age t on the website by
w
g*,t=exp(−tg*/T), (1.3)
where tg* is the existing time of garment g*, and T is a constant decay window, usually set to 30 to 90 days. This mechanism will slowly expire the older top-ranking garment items and tend to bring more recent garment items into the recommendation list. If we constantly set wg*,t=1, no time weighting will be applied to the recommendation,
We can also generalise the formulation in Eq. (1.1) so that the algorithm can recommend a new garment g* which never appears in the historical record H. In that case, we may make recommendation based on the other garments in the historical record H which are similar to g* as the following equation (1.4) shows:
R(g′)=wg
where sg(g−,g) defines a similarity score between the garment g* and an existing garment g in the historical record H. The similarity score sg(g−,g) can be computed based on the feature distances (i.e. Euclidean distance, vector correlation, etc.) of garment image features and metadata, which may include but is not limited to colour, pattern, shape of the contour of the garments, garment type, fabric material,
We compute the recommendation score R(g) for every single garment g in the garment database, and then rank the garment to be recommended based on their recommendation scores. Two different ranking approaches can be used for generating the list of recommended garments.
1. Top-n: This is a deterministic ranking approach. It will simply recommend the top n garments with the highest recommendation scores.
2. Weighted-rand-n: It will randomly sample n garment candidates without replacement based on a sampling probability proportional to the recommendation scores R(g). This ranking approach introduces some randomness to the recommendation list.
Section 2: The “Crowd” User Interface
2.1 Overview of the User Interface
The “Crowd” user interface is a user interface in which a collection of virtual avatars are displayed. In an example, a crowd of people is shown to the user. These avatars may differ in any combination of outfits, body shapes, and viewing angles. In an example, these people are all wearing different outfits, have different body shapes and are shown from different angles. The images may be generated using (eg. Metail's) visualisation technology which allows different body shapes to be modelled along with garments on those body shapes. A number of visual effects and recommendation features may be associated with this user interface. The “Crowd” user interface may contain the following major example product features:
Please refer to
2.2 Effects with Respect to the “Crowd” User Interface and Mathematical Models
The user can explore the crowd by sliding their finger horizontally over the screen. With this operation, all the body models in the screen move with predefined velocities to generate the effect of a translational camera view displacement in a perspective scene. In the process, the camera eye position e and target position t are translated horizontally with the same amount from their original positions e0 and t0 respectively, while the camera direction remains unchanged.
e=e
0+(Δx,0,0)
t=t
0+(Δx,0,0) (2.1)
According to the principle of projective geometry, we can use the following formulations to model the constraints among the scale s of the virtual avatars, the sliding speed v of the body models, and the image ground height h of each layer i (i=0, 1, 2, . . . , L) under this camera transform. Assuming zi is the depth of virtual avatars in layer i (away from the camera centre), then the sliding speed vi, the scaling factor si, and the image ground height hi (i=0, 1, 2, . . . , L) are given by:
where z0, v0, s0 and h0 are the depth, the sliding speed, the scaling factor, and the ground height of the foreground (first) layer 0, respectively. hhorizon is the image ground height of the horizon line, which is at the infinite depth. By applying different sliding speeds vi to different depth layers i (i=0, 1, 2, . . . , L) in the scene according to equations (2.2), we can achieve a perspective dynamic layering effect. A simple mock implementation example is illustrated in
When the user tilts the mobile device left or right, we can mimick the effect of a weak view rotation targeted at the foreground body model. In this process, the camera eye position e is translated horizontally from their original positions e0, while the camera target position t remains unchanged, as the following equation (2.3) shows:
e=e
0+(Δx,0,0)
t=t0 (2.3)
Under a weak perspective assumption where the translation Δx is small and the vanishing points are close to infinite, we can use the following equation (2.4) to approximately model the horizontal translation Δxi of each background layer i (i=1, 2, . . . , L) under this camera transform and achieve a view change effect:
where z0 and zi is the depth of the foreground (first) layer and each background layer i (i=1, 2, . . . , L), respectively. In an implementation, the amount of the eye translation Δx is proportional to the output of the accelerometer in the mobile device, integrated twice with respect to time.
When the user slides their finger vertically over the screen, we could activate the following “Elevator effects” and/or the “Layer-swapping effects” in the “Crowd” user interface products:
1. Elevator effects
When the user slides their finger over the screen vertically, an elevator effect will be created to switch to the next floor (either upstairs or downstairs). Also, an effect of looking-up/looking-down under a small rotation will be mocked up during the process.
In each floor, garments and/or outfits of a trend or a brand can be displayed eg. as a recommendation feature.
Elevator effects may be generated based on the following formulations of homography transform. Let K be the 3×3 intrinsic camera matrix for rendering the body model, and R be the 3×3 extrinsic camera rotation matrix. The homography transform makes the assumption that the target object (the body model in our case) is approximately planar. The assumption is valid when the rotation is small. For an arbitrary point p in the original body model image which is represented in a 4d homogeneous coordinate, its corresponding homogeneous coordinate p′ in the weak-perspective transform image can thus be computed as:
p′=Hp=KR−1K−1p. (2.5)
2. Layer Swapping Effects
We can also implement layer swapping effects with a vertical sliding. After the sliding, the virtual avatars in the background now come to the foreground, while the foreground ones now move to the background instead. There may be an animated transition for the layer swapping.
We apply the fog model, i.e. a mathematical model with respect to the translucency (alpha value) and the depth of the virtual avatars, to model the translucency of different depth layers. Assume the cj is the colour of the fog (eg. in RGBA) and cb is the sample colour from the texture of the body model. After the processing, the processed sample colour c is computed as
c=fc
f+(1−f)cb ,(2.6)
where f is the fog compositing coefficient that is between 0 and 1. For the linear-distance fog model, f is determined by the distance of the object (i.e. the virtual avatar) z as
We select znear to be the depth z0 of the first layer so no additional translucency will be applied to the foremost body models.
The effect can be achieved by applying transformations for scale and translucency transition. The transition of virtual avatars can be computed using the combinations of the equation (2.2) for layer movement and equations (2.6), (2.7) for creating the fog model.
This effect animates the dynamic process of switching a nearby body model from the background to the foreground using an elliptical rotational motion. Mathematically, the centroid position p=(x,y) of the body model may follow an elliptical trajectory during the transformation. The transformation of the scale s and translucency colour c of the model may be in synchronisation with the sinusoidal pattern of the model centroid displacement. In combination with equations (2.1) and (2.3), the parametric equations for computing the model central position p=(x,y), the scale s, and the translucency colour c during the transformation may be as follows:
x=x
end−(xend−xstart)cos(πt/2),
y=y
start+(yend−ystart)sin(πt/2),
s=s
start+(send−sstart)sin(πt/2),
c=c
start+(cend−cstart)sin(πt/2), (2.8)
where t is between 0 and 1, and t=0 corresponds to the starting point of the transformation and t=1 corresponds to the ending point of the transformation.
The floor and the background can be plain or an image that makes it look like the crowd is in a particular location. The background and the floor can be chosen by the user or customized to match some garment collections, e.g. using a beach image as the background when visualising the summer collection in the “Crowd”. Intermediate depth layers featuring images of other objects may also be added. This includes but is not restricted to garments, pillars, snow, rain, etc.
We can also model a lighting variation on the background: e.g. a slow transition from bright in the centre of crowd to dark at the periphery of the crowd. As a mathematical model, the intensity of the light source I may be inversely correlated with the Euclidean distance between the current location p to the centre of the “Crowd” c (in the camera coordinate system) as the example of equation (2.9) shows:
I=I
max/(1+γ∥p−c∥2), (2.9)
where γ is a weighting factor that adjusts the attenuation of the light.
The user can interact with the crowd to navigate through it. Some examples of such interaction are:
Clicking on icons by each model in the crowd brings up other features including, but not limited to, sharing with others, liking on social media, saving for later, and rating (see
2.3 Recommendation Mechanisms
We can arrange the garments and the outfits of those neighbouring background body models in the “Crowd” by some form of ranking recommendation mechanism (see
Examples of ranking mechanisms when placing avatars in the crowd are illustrated in
Several further recommendation algorithms may be provided based on the placements of body models in the “Crowd” user interface, as described below.
We can recommend a user those outfits which are published on the social network by her friends or those outfits selected by other virtual fitting room users who are in similar body shapes to her.
The ranking model may then be based on mathematical definitions of user similarity metric. Let b be the concise feature representation (a vector) of a user. For example b can be a vector of body metrics (height and weight) and tape measurements (bust, waist, hips, etc.), and/or other demographic and social network attributes. The similarity metric m between two users can be defined as the Mahalanobis distance of their body measurements ba and bb:
m(ba,bb)=(ba−bb)TM(ba−bb), (2.10)
where M is a weighting matrix accounting for the weights and the correlation among different dimensions of measurement input. The smaller the m, the more similar the two users. The recommended outfits are then ranked by m in an ascending order.
We can recommend popular outfit combinations containing one or more garments that are identical or very similar to a subset of the garments in the current outfit selected by the user. We may then rank the distances or the depths of the body models by a measurement of the popularity and the similarity between the two outfit combinations.
Mathematically this can be achieved by defining feature representations of the outfit and the similarity metrics, and applying a collaborative filtering. To formulate the problem, we represent a garment by a feature vector g, which may contain information including, but not limited to, garment type, contour, pattern, colour, and other types of features.
The outfit combination may be defined as a set of garments (feature vectors): O={g1, g2, . . . gN}. The dissimilarity metric d(Oa, Ob) of two outfit combinations Oa and Ob may be defined as the symmetric Chamfer distance:
The weighted ranking metric mi for outfit ranking is then defined based on the product of the dissimilarity between the current outfit O′ user selected and each existing outfit Oi published on the social network or stored in the database, and the popularity pi of the outfit Oi, which could be related to the click rate ci for example, as the following equation (2.12) shows:
m
i
=p
i
d(O′,Oi)=log(ci+1)d(O′,Oi) (2.12)
To recommend an outfit to a user, we may rank the all the existing outfits )Oi)i=1X according to their corresponding weighted ranking metrics (mi)i=1M in an ascending order, and dress them onto the body models in the “Crowd” from the near to the far.
We may define a combined ranking metric m which also takes user similarity into account. This may be done by modifying the definition of the popularity pi of the outfit Oi, which is used in the following equation (2.13):
where β is a hyper-parameters adjusting the influence of user similarity, b is the user feature of the current user, and bij is the user feature of the each Metail user profile j that has tried on the outfit Oi. The ranking and recommendation rules will still follow the equation (2.13).
2.4 Other Product Features
Other product features derived from this “Crowd” design may include:
User Interaction Features
The user may interact with the crowd to navigate through it. Examples are:
Section 3: Dynamic Perspective User Interface
3.1 Summary of the User Interface
The dynamic perspective user interface generates a user experience wherein one is given the feeling of being able to move around the sides of the virtual avatar by either moving one's head around the mobile device (eg. phone), or simply turning the mobile device (eg. phone) in one's hand, which is detected with a head-tracker module, or which could be identified by processing the output of other sensors like an accelerometer (see
3.2 Scene Construction
In the dynamic perspective design, the scene itself consists of three images indicating distinct 3D layers: the virtual avatar, the remote vertical background, and the floor plane. This setting is compatible with the application programming interfaces (APIs) of 3D perspective control libraries available on the mobile platform, which may include but are not limited to e.g. Amazon Euclid package.
As a specific example of implementation, the scene can be constructed using the Amazon Euclid package of Android objects, which allow the specification of a 3D depth such that images and other objects move automatically in response to user head movement. The Euclid 3D scene building does not easily allow for much customisation of the movement response, so the 3D geometry of the objects must be chosen carefully to give the desired behaviour. This behaviour may be emulated with other, simpler screen layouts in 2D with carefully designed movement of the images in response to detected head movement. Within the main application screen, the scene is held within a frame to keep it separate from the buttons and other features. The frame crops the contents so that when zoomed in or rotated significantly, edge portions are not visible.
3.2.1 The Virtual Avatar
Since the desired behaviour of the virtual avatar is for it to rotate about the vertical axis passing through the centre of the model, its motion cannot properly be handled by most of the 3D perspective control libraries on the mobile platform, as these would treat it as a planar body, which is a poor approximation when dealing with areas like the face or arms where significant variation in movement would be expected. This may instead be dealt with by placing the virtual avatar image as a static image at zero depth in the 3D scene and using a sequence of pre-rendered images as hereafter detailed in Section 3.3.
3.2.2 Background
Most built-in 3D perspective control libraries on the mobile platform, e.g. Amazon Euclid, treat all images as planar objects at a given depth and orientation. Observation of the movements produced as the user's head moves indicates that a point is translated at constant depth in response to either vertical or horizontal head movement. This is what makes it ineffective for the virtual avatar, as it does not allow for out-of-plane rotation. To achieve the desired effect of a floor and a remote vertical background (e.g. a wall or the sky at the horizon), the distant part of the background must be placed independently of the floor section, with the distant image placed as a vertical plane, and the floor image oriented such that the top of the image is deeper than the bottom of it (that is, rotated about the x-axis, which is the horizontal screen direction). Mathematically, it may be set up such that:
where v=vertical coordinate of the pivot point, as a fraction of the total image height (set to correspond to the position of the feet of the virtual avatar, measured from the top of the image; analysis of a virtual avatar image indicates the value should be around 0.9); other variables may be defined as shown in
The values of h and b are retrieved automatically as the pixel heights of the separated remote background and floor images, which are created by dividing a background image at a manually determined horizon line, as illustrated in
3.3 Modelling the Rotation of the Virtual Avatar
The avatar is shown to rotate by use of a progressive sequence of images depicting the model at different angles. For details about the methods which may be used to generate these parallax images of the virtual avatars from 3D models and 2D models, see Section 3.4.
Given that the parallax images are indexed with a file suffix indicating the rotation angle depicted, the desired image may be selected using the following formula for the stored image angle p:
where:
Taking this value, together with a garment identifier, view number, and image size, an image key is built and the correct image collected from the available resources using said key, for example as described in section 3.5.2.
3.3.1 Generating Stereo Image Pair for 3D Display
Based on Eq. (3.2), we can render a pair of parallax images (p, −p) with the same parallax amount p but of the opposite directions of rotation. This pair of images can be fed into the left-eye channel and the right-eye channel of a 3D display device respectively for the purpose of stereo visualisation. The possible 3D display device includes but is not limited to e.g. Google cardboard, or a display device based on polarised light. An example of a parallax image pair is given in
3.4 Generating Texture Images for the Rotated Virtual Avatar
An example of an end to end process of rendering 2D texture images of an arbitrarily rotated virtual avatar (see Section 3.3) is summarised in
When 3D textured geometry of the whole virtual avatar and 3D garment models dressed on the avatar are all present, generating a render with a rotated virtual avatar can be implemented by applying a camera view rotation of angle φ along the y-axis (the up axis) during the rendering process. The render is straightforwardly in a standard graphics rendering pipeline.
Some components of the virtual avatar may not have underlying 3D geometries. E.g. we may use 2D garment models for outfitting, in which only single 2D texture cut-out of the garment are present in specific viewpoint). Generating a rotated version of 2D garment models requires first approximating the 3D geometry of the 2D garment model based on some root assumptions, a depth calculation (see Section 3.4.1 for details), and finally a corresponding 2D texture movement will be applied to the image in order to emulate a 3D rotation (see Section 3.4.2 for details).
3.4.1. Generate 3D Approximate Garment Geometry from a 2D Texture Cut-Out
During the process of garment digitisation, each garment is photographed in 8 camera views: front, front right, right, back right, back, back left, left, and front left. The neighbouring camera views are approximately spaced by 45 degrees. The input 2D garment images are hence in one of the 8 camera views above. From these images, 2D garment silhouettes can be extracted using interactive tools (e.g. Photoshop, Gimp), or existing automatic image segmentation algorithms (e.g. an algorithm based on graph-cut).
For a 2D torso-based garment model (e.g. sleeveless dresses, sleeves top, or skirts) with a single 2D texture cut-out or silhouette, the 3D geometry model of the garment is approximated by applying the following simplifications:
An example of 3D geometry of a dress created from a single 2D texture cut-out using the method described above is given in
In the implementation, we generate this 3D geometry for each row of the garment image from the top, which corresponds to a given height on the body. In each row, the left and right extrema xleft and xright are estimated from the silhouette. For each of the 8 camera views in the digitisation, the semi-major axis length s for the garment ellipse is then given by:
The depth of the ellipse dellipse (i.e. the perpendicular distance from the camera) at each
pixel in the row is then approximated as the ellipse y-coordinate, yellipse, subtracted from the body origin depth, ybody:
d
ellipse
=y
body
−y
ellipse ,(3.4)
as yellipse>0 for most x and the garment is closer than the body (See
where b is the smoothing factor, the extent to which the transition is gradual or severe, j is the current image row index (0 at top), t is the predefined threshold indicating how far up the body the ellipse should begin taking effect, usually defined by the waist height of the body model.
The final depth used to generate the mesh for the approximate geometry is ensured to be lower than that of the body by at least a constant margin dmargin, thus given as:
d=min(dbody−dmargin,dbody(1−w)+dellipsew). (3.6)
The above approach can be generalised to model complex garment models, e.g. sleeved tops and trousers. In those cases, we may generate the approximate geometry for each part of the garment individually based on the corresponding garment layers and body parts using the equations (3.4)-(3.6) and the example equations shown in
An example of generating 3D approximate geometry of multiple layers for a pair of trousers is given in
Based on the reconstructed approximated 3D geometry we can then model the 3D rotation of a garment by a 2D texture morph solution as described in Section 3.4.2.
3.4.2 Morph a 2D Texture Based on the Approximated 3D Geometry
Having generated a smooth 3D mesh with faces from the point cloud of vertices given by the depth approximations at each pixel in the previous step, a final normalised depth map of the garment may be generated for the required view. This depth map may be used to calculate the extent to which a given point on the garment texture needs to move in the image in order to simulate an out-of-plane rotation about the vertical axis (the y-axis in screen coordinates). The current normalised position p of a texture pixel is set to:
p=(px,py,pz,1), (3.7)
where:
1−j/(w/2), j is the horizontal pixel position, w is the mage pixel width.
py=
1−i/(h/2), i is the vertical pixel position, h is the image pixel height;
px is the normalised depth from the depth map; resultant values are in range [−1, +1].
Using the viewing camera 4×4 projection, view, and world transformation matrices, P, V, and W respectively, where the multiplied combination WVP represents the post-multiplication transformation from the world coordinates to the image coordinates; a rotation matrix, R, is computed for rotation about the z-axis based on the required angle. The new image coordinate position p′ of the corresponding point on the 3D geometry is then given by:
p′=pp−1V−1W−1RWVP. (3.8)
The resultant 2D transformation on the image, normalised by the full image dimensions, is given by:
These 2D transformations are stored for a sampled frequency of pixels across the entire image, creating a 2D texture morph field that maps these normalised movements to the pixels.
The 2D texture morph field only has accurately calculated transformations for the region inside the garment silhouette and so must be extrapolated to give smooth behaviour across the entire image. The extrapolation and alteration of the morph to give this smoothness can be carried out in a number of distinct steps as follows:
1. Limit the morph such that any texture areas that are meant to become overlapping are instead forced to collapse to a single vertical line. Owing to internal interpolation between sample points, this is imperfect, but helps to avoid self-intersection of the texture.
2. Extrapolate the morph horizontally from the garment silhouette edges, using a weighted average of the morph values close to the edge to ensure the value does not jump significantly in these areas.
3. Extrapolate the morph vertically from the now-complete rows, simply copying the top and bottom rows upwards and downwards to the top and bottom of the image.
4. Apply a distributed blur smoothing to the morph, e.g. by using a 5×5 kernel in expression (3.10):
The resultant images produced are the likes of those shown in for example in
For a more complex garment like trousers or sleeved-top, the above texture morph solution will be applied for each individual garment layer (i.e. torso, left/right sleeve, leg/right leg) individually.
To implement the dynamic perspective visualization systems, two different approaches may be applied:
1) The visualization server generates and transmits the full dynamic perspective images of the garments, given a query parallax angle from the client. This involves computing 2D texture morph fields based on the method described above, and then applying the 2D texture morph fields onto the original 2D garment images to generate the dynamic perspective images.
2) The visualization server only computes and transmits image manipulation functions to the client side. As concrete examples, the image manipulation function can be the 2D texture morph fields (of all garment layers) above, or the parameters to reproduce the morph fields. Then, the client will finish generating the dynamic perspective images from the original 2D garment images locally based on returned image manipulation functions. Since the image manipulation functions are usually much more compact than the full images, this design can be more efficient and give better user experience when the bandwidth is low and/or the images are of a high resolution.
3.4.3 3D approximate geometry and texture morph for the 2D head sprites or 2D hairstyle
We can use a similar approach to approximately model the 3D rotation of a 2D head sprite or 2D hairstyle image when the explicit 3D geometry is not present. For this, we use the underlying head and neck base geometry of the user's 3D body shape model as the approximate 3D geometry (see
3.5 Other Features and Related Designs
Note that the term “parallax” is used loosely in that it refers only to the principle by which the rotated images are generated (i.e. image sections at different distances from the viewer move by different amounts). In particular, “parallax” angles indicate that the angle in question is related to the rotation of the virtual avatar in the image.
3.5.1 Settings and Customisation
This section gives a sample user interface for setting the parameters of the application. As shown in
3.5.2 Image Selection
Given the settings as described in Section 3.5.1, a resource identifier is constructed with which to access the required image resources. The image resources can be indexed by garment setting, view setting, and image size setting.
Whenever settings are initialised or altered, a list of available parallax values for those settings is stored based on the accessible image resources. The list is sorted in increasing values of parallax value from large negative values to large positive values. A nearest index search can be implemented given an input parallax value p. Given an integral equivalent of p (rounded to 2 decimal places, then multiplied by 100), the following ordering of criteria are checked:
This closest available integral equivalent ofp is then used as the final value in the name construction used to access the required image resource.
Notes
In the above, examples are given predominantly for female users. However, the skilled person will understand that these examples may also be applied for male users, with appropriate modifications where necessary.
It is to be understood that the above-referenced arrangements are only illustrative of the application for the principles of the present invention. Numerous modifications and alternative arrangements can be devised without departing from the spirit and scope of the present invention. While the present invention has been shown in the drawings and fully described above with particularity and detail in connection with what is presently deemed to be the most practical and preferred example(s) of the invention, it will be apparent to those of ordinary skill in the art that numerous modifications can be made without departing from the principles and concepts of the invention as set forth herein.
| Number | Date | Country | Kind |
|---|---|---|---|
| 1422401.8 | Dec 2014 | GB | national |
| 1502806.1 | Feb 2015 | GB | national |
| 1514450.4 | Aug 2015 | GB | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/GB2015/054042 | 12/16/2015 | WO | 00 |
