The present invention refers to methods and systems for determining body measurements of a person, as well as to methods and systems for providing clothing size recommendations.
Shopping online for clothing enjoys ever increasing popularity. One downside of the online shopping experience is that, although the design of items of clothing can be evaluated to some degree, the item cannot be tried on as it would be possible in an offline store. This leads to several undesired effects; for example it is not uncommon for shoppers that are interested in an item of a certain design to order one and the same item in several different sizings. The intention is to find the one sizing that optimally fits and/or suits the shopper, and to send back the other sizings.
Herein, a sizing should be understood to mean one particular version of the same item of clothing that is different from other versions, i.e. other sizings, only in at least some of its measurements. For example, one and the same sweater may be available in sizings XS (extra small), S (small), M (medium), L (large), XL (extra large) and so on.
An even more common occurrence is that a shopper orders only one sizing of an item but finds out that the ordered sizing does not fit after all, either because the shopper misjudged which sizing would be appropriate, or because for said specific item of clothing a particular sizing is smaller than other items of clothing of the same size. For example, an item of clothing that is considered to be “M” in Japan may be considered to be “S” in Germany or the like.
As a result, there is a lot of mail traffic because of items of clothing being sent back to the vendor by shoppers, often encouraged by national laws that force vendors to pay the transport fees. This mail traffic taxes not only environmental and public resources, but is also a substantial cost factor for vendors. Moreover, also the shoppers' patience is tried and shoppers may refrain from ordering clothing online at all because of the hassle with trying different sizings and having to send many or even all of them back to the vendor.
One way to meliorate these problems would be to compare detailed body measurements of a particular shopper—at least one measurement but the more measurements, the better—to properties of each sizing of each item of clothing, for example as a professional tailor would. However, very few people have their body measurements taken by professional tailors, mostly due to the inconvenience and cost of it. Many people also take their own measurements wrongly or simply attempt to guess the right sizing because it is too much effort to measure oneself, ending up with inaccurate guesses.
Accordingly, a method is desired for conveniently yet accurately determining at least one body measurement of a person.
However, even when body measurements are known, it is still not straightforward to determine the correct sizing of a particular item of clothing for a particular person since it will be a rare occurrence that the measurements of the person fit exactly the measurements of a particular sizing.
In addition, for the shoppers comparing their measurements with numerous sizing charts is too time-consuming when shopping. Moreover, different brands often have different sizing standards and sometimes even clothing items of nominally the same sizings and the same brand but for different items (designs) may differ from each other. Due to the negligibly rare usage of sizing charts web shops often publish a single averaged (across products) sizing chart to avoid the hassle of uploading (let alone generating) one sizing chart for every product in stock. The more averaged the sizing chart is, however, the less useful it will be when used to look up the right size of a product.
Accordingly, a method is desired for conveniently yet accurately providing a clothing size recommendation for a person based on at least one body measurement of the person. The clothing sizing recommendation may be a general size recommendation such as “Sizing M fits this person”, a sizing recommendation for a specific type of clothing such as “For trousers, sizing M fits this person”, an item-based sizing recommendation such as “For this particular item of clothing, sizing M fits this person”, an overview recommendation such as “For this person, the following sizings of the following items of clothing fit: . . . ” and/or the like.
The above objects, as well as further objects that may become explicitly or implicitly apparent in the following, are solved by the independent claims of the present invention.
Thus, according to a first aspect of the present invention, a computer-implemented method for determining at least one body measurement of a person, comprising at least the steps of:
Nowadays, with the widespread availability of cameras of high quality, for example in webcams, laptops, computer monitors, mobile phone, tablet computers and the like, it is very easy and convenient to obtain images of people, and indeed comes natural to many people that are fond of appreciating, buying and experimenting with fashion.
In some variants or embodiments, simply all of the obtained images may be selected as the first input images. In other variants, at least one criterion may be applied on the obtained images to select therefrom the first input images. One criterion may be that only such of the obtained images may be selected as the first input images that fulfill a pre-defined image quality criterion, for example that have a minimum image resolution, a minimum size, a minimum contrast (due to lighting etc.) and/or the like. Additionally, or alternatively, the method may be configured such that every x-th image of the obtained images is selected, preferably every x-th image out of a pre-selection of the obtained images based on the at least one criterion, wherein x is suitably chosen integer number.
The segmentation maps comprise at least one segmentation masks, wherein each segmentation mask contains, or indicates, pixels that are associated with a specific object category. In simple cases, a segmentation map comprises a single segmentation mask which differentiates the person in the image from the background. In other variants, the segmentation maps may comprise several segmentation masks, wherein different segmentation masks identify different body parts. For example, a segmentation map may comprise six segmentation masks: one for each of head, torso, and each extremity.
In the present context, “at least one body measurement” shall be understood to include a particular measurement type of a body of a person in a defined way in a defined location. Typical body measurement types include, in particular for women, shoulder, arm hole, upper arm, bust, upper belly, belly, waist, hips, thighs, inseam, bottom length, and/or knees, and may also include, in particular for men, collar, top of shoulder, chest, rise, inside leg and/or outside leg. Thus, determining more than one body measurement may include determining values for more than one type of body measurement. However, determining more than one body measurement may also include determining more than one measurement for the same body measurement type but using different methods.
In some advantageous embodiments, variants, or refinements of embodiments, the plurality of images of the person is a video of the person or is extracted from a video of the person. For example, the method may be configured such that a certain percentage of frames of the video and/or a certain minimum number of frame or the like is selected as the first input images.
In order to more accurately determine a plurality of body measurements, it is preferred that the video shows the person from at least two, more preferably from at least four, different viewing angles with respect to the longitudinal axis (from feet to crown) of the person, wherein the viewing different angles are most preferably separated from one another by a pre-defined minimum angle difference. For example, when the video shows a person turning around their longitudinal axis by 90 degrees, and the pre-defined minimum angle is 30 degrees, then said video shows the person from four different viewing angles.
It is most preferred if the video shows the person turning at least once completely around the longitudinal axis (i.e. at least 360 degrees), wherein additional complete turns (i.e. 720 degrees, 1080 degrees and so on) may further improve the accuracy using averaging techniques and the like.
In other words, in some advantageous embodiments, variants, or refinements of embodiments, obtaining the plurality of images of the person comprises acquiring the video of the person by filming the person while the person is at least partially turning around their longitudinal axis, and is preferably turning at least once completely around their longitudinal axis. Advantageously, the
The filming may be performed by one or more fixed cameras (for example a camera mounted in a computer screen or a laptop screen) or by a camera handheld by another person which may, for example, use their mobile phone to film the person for which the at least one body measurement is to be determined.
It is preferred that the filming is performed using a dedicated application (“the app”) which may be installed on and run by a mobile device which, apart from acquiring the video, may advantageously acquire additional data that may be saved and, partially or completely, transmitted together with the video data. The app may also be a web application which is accessed by a web browser of the mobile device and which is displayed by a display of the mobile device.
For example, during the video acquisition process (i.e. the filming), the orientation of the mobile device as well as its linear accelerations in 3-dimensional space may be recorded. These may be helpful to determine, and possibly compensate, changes in the viewing angle that are not due to the filmed person moving but instead due to the filming mobile device moving. For example, when a person holding the mobile device running the app suddenly moves a step back, the person in the images appears to become smaller. However, using the orientation and acceleration of the mobile device it may be easily determined that the person has actually stayed the same size.
The above is a rather trivial example; however, it will be understood that involuntary or other movements of the filming mobile device could in principle result in other distortions of the size of the person in the video which are however compensable due to the measurements of orientation and/or acceleration of the filming mobile device. Preferably, the filming (i.e. the acquiring or recording of the video) is performed at frequencies in the range of 1 Hz to 100 Hz. It has been found that such frequencies provide a good balance between the constraints of taking enough image frames in the video so as to show the person from enough different viewing angles on the one hand, and the constraints of the capabilities of the camera as well as the desire to reduce data storage space needs.
In some advantageous embodiments, variants, or refinements of embodiments, the method further comprises selecting at least a sub-set of the first input images to provide second input images. Preferably, all of the first input images are selected as the second input images to be provided. However, in some variants additional filters and/or criteria which have to be fulfilled by the second input images may be applied to the first input images. It is further preferred that the number of second input images is in the range of 3 to 60.
The method may further comprise determining, in order to estimate a shape and/or pose of the person in each of the second input images, a corresponding parameter value θki for each image k of the second input images and for each parameter of a set of pre-defined parameters θi. In particular, this may be performed by fitting a parametric human body model to the person in each of the second input images based on the generated segmentation maps.
The parametric human body model used may e.g. be the SMPL body model (see Loper et al., “SMPL: A Skinned Multi-Person Linear Model”, ACM Trans. Graphics (Proc. SIGGRAPH Asia), October 2015, No. 6, pages 248:1-248:16, volume 34 of 2015), BlendSCAPE (see D. A. Hirshberg et al., “Coregistration: simultaneous alignment and modeling of articulated 3d shape”, Proceedings of the twelfth European conference on computer vision (2012), pp. 242-255) or any other suitable known parametric human body model. It is preferred that a parametric body is used that comprises any set of parameters (θ=(θ1, θ2, . . . , θn) of Rn, 10<n<1000 (preferably 20<n<300), that describe the geometrical shape, pose and other features of the human body.
The method further comprises generating an input tensor based on the determined parameter values θki. The input tensor may be generated such as to comprise all individual values for θki for each parameter θi for each image k of the second input images. Alternatively, a pre-processing may be performed; for example, an averaging metric θavgi for the values for each parameters θi may be determined, and the input tensor may comprise the n averaging metrics θavgi. The averaging metric may be a mean such as a geometric mean or an algebraic mean, a median and/or the like.
The method may then comprise inputting the input tensor into an artificial neural network configured and trained to infer at least one body measurement based on the input tensor, and generating, by the artificial neural network, at least one body measurement based on the input tensor.
In some advantageous embodiments, variants, or refinements of embodiments, the method comprises selecting at least a subset of the first input images to provide third input images. All of the first input images may be selected as the third input images, and/or the third input images may overlap, or be identical with, the second input images.
The method may further comprise determining locations for a number of pre-defined key points on the human body by inputting the third input images into an artificial neural network configured and trained to infer the location for the key points based on the input third input images.
The method may further comprise determining, based on the determined locations for the key points, a subset of the first input images so as to provide fourth input images such that each of the fourth input images corresponds to one view of a set of pre-defined views of the person from different angles, preferably at least four different pre-defined views, each corresponding to a different viewing angle of the person. It is preferred that the at least four pre-defined view comprise a (full) front view, a (full) back view, a left (side) view and a right (side) view. More preferably, more than four pre-defined views are determined to increase accuracy.
The number of pre-defined key points may comprise a number of pairs of symmetric key points (in particular symmetric respective to the sagittal-longitudinal plane of the human body).
Advantageously, in images (or: frames) distances between the two key points of each pair of symmetric key points can be determined, and a pose/orientation of the person can be determined on at least one of these determined lengths. In particular, the full front or full back viewing angle can be determined by the images in which at least one (or a majority, or all) of the determined distances are maximal. Correspondingly, a left or right side view of the person can be determined by the images in which in which at least one (or a majority, or all) of the determined distances are minimal.
The method may further comprise generating, for each of the fourth input images, a silhouette of the person therein, and determining at least one body measurement for the person by inputting an input based on the determined locations for the key points and the generated silhouettes, into a body measurement determining model. Said input may comprise, or consist of, the key points and the generated silhouettes themselves and/or data calculated based thereon.
According to a second aspect of the present invention, a computer-implemented method for providing a clothing size recommendation for a person is provided comprising generating at least one body measurement according to the method according to any embodiment of the first aspect of the present invention. The method may further comprise inputting the at least one body measurement into a clothing size recommendation model configured to generate a clothing size recommendation based on the at least one body measurement, and generating, by the size recommendation model, the clothing size recommendation based on the at least one body measurement.
As has been described in the foregoing, the clothing sizing recommendation may be a general size recommendation such as “Sizing M fits this person”, a sizing recommendation for a specific type of clothing such as “For trousers, sizing M fits this person” or “For jeans, 34W32L fits this person” or “For shirts, size 38 fits this person”, an item-based sizing recommendation such as “For this particular item of clothing, sizing M fits this person”, an overview recommendation such as “For this person, the following sizings of the following items of clothing fit: . . . ” and/or the like.
In some advantageous embodiments, variants, or refinements of embodiments, the method according to the second aspect comprises receiving or retrieving clothing item information indicating at least one property of an item of clothing, and the clothing size recommendation is generated also based on the clothing item information.
For example, the clothing item information may comprise, or consist of, information about at least one measurement of at least one item of clothing (e.g. one that a customer has selected) such as a recommended, or a maximum body measurement of specific type for the item of clothing (e.g. recommended waist size X cm). the clothing item information may also comprise, or consist of, information about a relationship between the person and at least one sizing of the item of clothing.
For example, a user may have order sizing L of one particular item of clothing, may have found out that sizing L is too large and has sent the item back, giving the feedback that sizing L is too large for the person. When generating a clothing size recommendation for that user regarding the same item of clothing, this information may be used (for example to recommend a sizing smaller than L for the person). The clothing item information may comprise, or consist of, many other items or types of information such as an elasticity of the item of clothing at a position corresponding to one of the body measurements (such as the waist) and/or the like.
In some advantageous embodiments, variants, or refinements of embodiments, the clothing item information indicates properties of at least two different sizings of an item of clothing, and wherein the clothing size recommendation includes, for at least one of the at least two different sizings, a recommendation information of fitting or non-fitting of said at least one sizing for the person. Preferably, the clothing item information indicates properties of all available or existing sizings of the item of clothing and the clothing size recommendation includes a recommendation information of fitting or non-fitting of each of the sizings for the person. Optionally, the user may input additional parameters, for example indicating that the user prefers a “loose fit” or a “tight fit” in general or in a particular body measurement region, which can be used in the method for the clothing size recommendation.
In some advantageous embodiments, variants, or refinements of embodiments, the method according to the second aspect further comprises the step of receiving or retrieving return information indicating at least one item of clothing which has been labelled as being of a wrong size or poor fit (i.e. has been given a low fit rating) for a user. This user may be the same as the user that is currently desiring a clothing size recommendation but may also be another user, preferably a plurality of other users.
The reason is that, for example, users with the same body type, or even very similar body measurements, as the current user may have consistently given low fit ratings to a particular sizing so it stands to reason that the same will be true also for the current user. Vice versa, when no such information is available, but for example a lot of users with very different body measurements from the current user's have given high fit ratings to a particular sizing then it stands to reason that the same sizing will very likely not be a good fit for the current user.
“Wrong size” may indicate that the person cannot fit (optionally differentiating for at least one body measurement where the person does not fit), whereas “poor fit” may indicate that the person fits but (for at least one body measurement) desires a different fit, for example because the person desires a looser or tighter fit for the at least one body measurement for medical, anatomical and/or aesthetic reasons.
The return information may further indicate at least one property of said returned at least one item of clothing, most preferably information relating to the fitting of the person to at least one specific body measurement of the item of clothing. Thus, the return information may in particular comprise, or consist of, user feedback of the person for which body measurements have been determined and which has tried the returned at least one item of clothing.
For example, the person may have bought a shirt and then have returned it, indicating the degree or quality of fitting of the person at the shoulders, chest, sleeve length and/or the like. Such information is not only highly useful for providing an improved clothing size recommendation to the person in the future, it may also be used to train at least one of the artificial neural networks used in the method.
Preferably, the clothing size recommendation is generated also based on the return information.
In some advantageous embodiments, variants, or refinements of embodiments, the clothing size recommendation model comprises a learning sub-model, LSM, and a non-learning sub-model, NLSM. A weighted sum of an output of the LSM and of an output of the NLSM may calculated to provide a fitting score vector, and the clothing size recommendation is generated based at least on the provided fitting score vector. The weighting between the LSM and the NLSM may be based on a quality metric of the LSM such that with improving quality metric of the LSM, automatically the weight of the LSM increases. Said quality metric may include an accuracy of the LSM with a validation set, a number of training sets used for training the LSM, a number of training epochs and/or the like.
In some advantageous embodiments, variants, or refinements of embodiments, at least one weighting factor for the weighted sum for a clothing size recommendation regarding a particular item of clothing is variable and is based on a number of datasets (in particular training datasets) available for said particular item of clothing, wherein the datasets include at least one information about at least one body measurement of a user and at least one information (“fitting information”) about how said particular item of clothing fits said user, specifically how at least one measurement of the item of clothing compared to at least one body measurement of the user (e.g. “waist too tight”, “hips to loose”, “inseam perfect fit”).
In some advantageous embodiments, variants, or refinements of embodiments, the NLSM is based at least on a distance metric between:
a) at least one parameter of at least one item of clothing, and
b) at least one of the parameter value θki and/or at least one entry of the input tensor (for example, an averaging metric θavgi of the parameter values θki for one parameter θi a).
This implements the basic idea that an item of clothing fits a person well if the body measurements of the person fit are as close as possible to the corresponding measurements of the item of clothing (i.e. waist measurement value of the person with respect to waist size of an item of clothing and so on). The distance metric may treat negative and positive distances differently, and may in particular penalize a too-small measurement of the item of clothing relatively more than a too-large measurement, since as a compromise often a loose fit for some body measurement which may cause a slight unseemliness is generally more acceptable than a tight fit that may cause discomfort.
In some advantageous embodiments, variants, or refinements of embodiments, the LSM comprises an artificial intelligence entity configured and trained to receive the input tensor as an input and to output a first fit likelihood vector of the same dimensions as a second fit likelihood vector output by the NLSM and/or outputs a first fit score. The fit likelihood vector may comprise, for each of a list of sizings and/or items of clothing, an entry indicating the likelihood that said sizing or item of clothing or combination of sizing and item of clothing is the best fit for the person. The first fit score may indicate a score (e.g. in the range of 0 to 100) of how well a particular (e.g. the best-fitting) sizing/item according to the fit likelihood vector actually fits the person. Alternatively, a first fit score vector may be comprised that indicates for each sizing/item how well it fits the person. Further alternatively, the fit likelihood vector may comprise the first fit score in one or more entries.
According to a third aspect, the present invention provides a computer-implemented method for training an artificial intelligence entity, in particular an artificial neural network, for use in the method according to any embodiment of the first aspect of the present invention and/or for use in the method according to any embodiment of the second aspect of the present invention. In particular, an artificial neural network may be trained by return information provided by users, wherein the return information indicates fit ratings of the users in which they describe how well at least one measurement type of at least one item of clothing fits them.
Information about at least one body measurement of the users may be provided separately, e.g. retrievable via a unique identifier, or may be provided in a signal in which return information is transmitted. Thus, the return information encodes information about users' body measurements and how well they fit in relation to measurements of items of clothing. The return information may be automatically transmitted to a computing device configured for training the artificial neural network from web shops that request said information from users when they intend to return an item of clothing.
According to fourth aspect of the present invention, thus also a computing device is provided which is configured to receive return information and to train an artificial neural network and/or adapt a pre-trained artificial neural network, in particular for use in the method according to any embodiment of the first aspect of the present invention and/or for use in the method according to any embodiment of the second aspect of the present invention.
The computing device may be realized in hardware, such as a circuit or a printed circuit board and/or comprising transistors, logic gates and other circuitry. Additionally, the computing device may be at least partially realized in terms of software. Accordingly, the computing device may comprise a processor (such as at least one CPU and/or at least one GPU) and a memory storing software or a firmware that is executed by the processor to perform the functions of the computing device.
The computing device may also be realized as a cloud computing platform and/or in a distributed way. For example, some computations may be performed by a mobile computing device (or edge device), and some computations may be performed by a cloud computing platform.
The invention further provides, according to a fifth aspect, a system configured to perform the method according to any embodiment of the first aspect of the present invention and/or configured to perform the method according to any embodiment of the second aspect of the present invention. The system may in particular comprise a mobile device (such as a smartphone, a tablet computer and/or the like) and a remote computing device (such as a cloud computing platform or a web server). The remote computing device may comprise a database and/or may be operatively coupled to a database of the system for storing data thereon, for example learning models to be trained, return information, user parameters, unique identifiers and/or the like.
Preferably the mobile device is configured (and used) to obtain the plurality of images of the person, for example by running (or, more preferably, in the case of a web application, displaying) an app that performs the necessary method steps. Further, the remote computing device preferably performs the method steps not performed by the mobile device, in particular the generating of the segmentation maps, the determining of the at least one body measurement, the generating of the clothing size recommendation and so on.
According to a further aspect, the invention thus also provides a server configured to receive the plurality of images obtained from the mobile device and/or the selected first input images, and to perform the remaining steps of any embodiment of the first aspect and/or of the second aspect of the present invention, in particular the generating of the segmentation maps, the determining of the at least one body measurement, the generating of the clothing size recommendation and so on.
According to a sixth aspect, the invention provides a computer program product comprising executable program code configured to, when executed (e.g. by a computing device), perform the method according to an embodiment of the first aspect of the present invention and/or according to an embodiment of the second aspect of the present invention.
According to a seventh aspect, the invention provides a non-transitory, computer-readable data storage medium comprising executable program code configured to, when executed (e.g. by a computing device), perform the method according to an embodiment of the first aspect of the present invention and/or according to an embodiment of the second aspect of the present invention. The data storage medium may be a CD, a DVD, a magnetic hard drive, a solid-state hard drive, a USB stick and/or the like.
The invention will be explained in yet greater detail with reference to exemplary embodiments depicted in the drawings as appended.
The accompanying drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of the specification. The drawings illustrate the embodiments of the present invention and together with the description serve to illustrate the principles of the invention. Other embodiments of the present invention and many of the intended advantages of the present invention will be readily appreciated as they become better understood by reference to the following de-tailed description. Like reference numerals designate corresponding similar parts.
The numbering of method steps is intended to facilitate understanding and should not be construed, unless explicitly stated otherwise, or implicitly clear, to mean that the designated steps have to be performed according to the numbering of their reference signs. In particular, several or even all of the method steps may be performed simultaneously, in an over-lapping way or sequentially.
In a step S10 a plurality of images of the person is obtained, wherein the images in the plurality of images show the person from at least two different viewing angles, more preferably from at least four viewing angles, wherein the viewing angles are advantageously equally-spaced.
Most preferably, the plurality of images is obtained by recording a video of the person. In the following, an exemplary situation will be described in which a smartphone is used to take a video of the person for which the body measurements are to be determined using an app installed on the smartphone. However, it should be understood that numerous are variants, some of which have been described in the foregoing, exist for recording a video of the person.
In the present example, the recording of a video is performed using a standard smartphone RGB camera (front or back-facing).
The smartphone 1 is handed to a person nearby to take the video (variant (A) in
During the video recording process, in a step S13, the orientation of the smartphone 1 as well as the linear accelerations in 3-dimensional space are recorded, preferably at frequencies of between 1 and 100 Hz. A unit normal vector describing the axis parallel to the optical axis of the device's front or back-side camera is referred to herein as
In a step S20, at least a sub-set of the received plurality of images is selected to provide first input images. In the present example, individual frames of the recorded video may be selected as the first input images. For example, every N-th frame of a video may be selected as a first input image, wherein N may be determined as N=fps/C, fps being the frames per second of the video and C being a constant that is preferably in the range of between 30 and 5. Alternatively, N may be fixed to a specific number of frames per second which may also be in the range of between 5 and 30.
In a step S30, segmentation maps 3 for the first input images are generated, wherein the segmentation maps 3 at least distinguish the person 2 and/or differentiate their body parts from a background 4. Any known segmentation method or segmentation algorithm, in particular using at least one artificial neural network, can be used for the generation of the segmentation maps.
In a step S40, location for a number of pre-defined key points on the human body as well as at least one silhouette of the person 2 in at least one of the plurality of obtained images (preferably in the first input images) are determined, specifically as follows:
In a step S41, at least a subset of the first input images are selected to provide third input images; all of the first input images may be selected as the third input images.
In a step S42, the locations for a number of pre-defined key points K on the human body are determined by inputting the third input images into a first artificial neural network configured and trained to infer the location for the key points based on the input third input images.
The first artificial neural network DCNN1 can include multiple network branches in a combination of 2D convolution and deconvolution layers with any non-linearity as activation functions, Batch Normalization (BatchNorm) layers. The first artificial neural network DCNN1 may comprise between 25 and 500 hidden layers trained on training datasets to estimate the pose of the person from images (e.g. video frames).
The model infers a number key points (preferably in the range of from 4 to 100) on the human body in 3-dimensional space which describe the pixel (on the respective third input image) position and orientation (Euler angles, quaternions, dual quaternions etc.) of pre-defined joints such as wrist, elbow, shoulder, knees etc. or center points of organs or other landmarks on the human body such as the eyes, mouth, ears and others (marked as points in
From a subset (or all) of the key points Ki located on the human torso (in the present example e.g. K2, K4, K10 and K11), in a step S43 a unit normal vector
In a step S44, at least one distance L between pairs of symmetric key points on the left and right parts of the subject's torso (e.g. length from left shoulder key point K2 to right shoulder key point K4 projected to the current camera view) is measured and recorded for all or a subset of the frames. This may be performed already during the recording of the video by the app.
Preferably, between 1 and 10 distances L are measured between pairs of symmetric key points. If the pose is estimated for only a subset of the available frames (i.e. when there are fewer third input images than first input images), the measured distances L may be, in an optional step S45, interpolated and smoothed such as to have L for each of the first input images, or even for each frame of the video.
In a step S46, a frontal and back view F+ (front) and F− (back) are selected by taking the images with the maximal Euclidean distance between the selected key point pairs. The two side views (left and right) designated as SL and SR are selected by taking the frames with minimal Euclidean distance either of one of the pairs (e.g. the largest one or the smallest one), or of the mean of the pairs. The app used for obtaining S10 the video may guide the person 2 to move in such a way that the specified view of the person are as clear as possible in the video.
If no clear the minima/maxima can be determined by using L, the unit normal vectors
Apart from said four views, additional views may be determined for improved accuracy and robustness against occlusions and the like. For example, equidistantly between each two adjacent views an additional view may be determined so that in total eight views are provided. In the following, the situation of only four views will be described for simplicity.
Once the images (frames) with the four views (front, back, left, right) are obtained from the video, in an optional step S47 each image is normalized to have the same number of pixels and cropped to the subject's segmentation map to have equal distance from the outermost pixel in every side of the image.
In a step S48, in every third input image, the outline of the segmentation maps (herein designated as a “silhouette”) is determined (or: calculated), preferably by using an edge detection algorithm such as Canny or similar (Canny generally consists of 5 steps: Noise reduction; Gradient calculation; Non-maximum suppression; Double threshold; Edge Tracking by Hysteresis). The silhouette of the segmentation maps includes the pixel coordinates of every pixel in the subject's silhouette.
In a step S49, for every pixel in the subjects silhouette the nearest key point K is identified via 1-Nearest-Neighbor search with a Euclidean distance metric on the pixel coordinates along the width (horizontal, cw) and height (vertical, ch) of the 2-dimensional third input image. The horizontal and vertical distances along with the index of the key point K the silhouette pixel j was assigned to are stored as a silhouette triplet stj=(ikp, dw, dh), wherein ikp indicates the corresponding key point Ki.
In a step S50, a first input tensor {tilde over (X)} is formed by concatenating the key points Ki from the pose estimation together with all silhouette triplets stj for each selected third input image are concatenated, for example as
{tilde over (X)}=(SSM+,SSM−,SSML,SSMR,K+,K−,KL,KR),
wherein SSM+ designates all silhouette triplets stj of the front view F+, SSM− designates all silhouette triplets stj of the back view F−, SSML designates all silhouette triplets stj of the left side view FL, SSMR designates all silhouette triplets stj of the right side view FR, K+ designates all key points Ki of the front view F+, K− designates all key points Ki of the back view F−, KL designates all key points Ki of the left side view FL, and KR designates all key points Ki of the right side view FR, and the “,” in the above formula designates a concatenation.
In a step S60, the first input tensor {tilde over (X)} is input into a body measurement determining model, preferably into a general linear model or an artificial neural network regressor model. The artificial neural network regressor model may comprise hidden layers in the range of 1 to 200 and may be configured and trained to receive the first input tensor {tilde over (X)} and to determine, based thereon, at least one body measurement of the person, preferably all of the pre-defined body measurements (or: body measurement types). In other words, the body measurement determining model will map the first input tensor {tilde over (X)} to a first body measurement vector y1, wherein y1 preferably has a size in the range of from 5 to 50, wherein each of the 5 to 50 entries indicates one body measurement value. In yet other words, the body measurement determining module represents a function f1 with f1({tilde over (X)})=y1.
The body measurements of the first body measurement vector y1 are herein also designated as first type body measurements. The method described in the foregoing is comparatively easy and quick to perform.
In the following, with respect to
Thus,
Steps S10, S20 and S30 may be performed as has been described in the foregoing.
In a step S100, at least a sub-set of the first input images is selected to provide second input images. The selected first input images may be the same as the selected third input images, may overlap with the third input images, or may be completely separate from the third input images. Preferably, a number in the range of 10 to 300 frames are selected from the video as the second input images, and such frames are selected that have the rotating person facing equidistant angles in the ground plane (x-y plane in the coordinate system as defined in the foregoing) or, in other words, in a plane perpendicular to the ground plane, as measured or indicated by
The goal is to take frames that differ in camera viewpoint V as much as possible regardless of the speed of rotation of the person, of video duration and frame rate. Here a viewpoint V constitutes the position and orientation of the camera view's optical axis.
In a step S110, in order to estimate a shape and/or pose of the person in each of the second input images, a corresponding parameter value θki for each image k of the second input images and for each parameter of a set of pre-defined parameters θi by fitting a parametric human body model (such as SMPL, BlendSCAPE or the like) to the person in each of the second input images based on the generated segmentation maps.
Specifically, the parametric human body model is fitted to 2-dimensional joint detections in an image for each frame under consideration of its neighboring k-frames in the sequence to estimate an initial shape and 3-dimensional pose. The 2-dimensional joint detections may be provided for example by any or all of the key points K on the human body determined as has been described in the foregoing. Usable 2-dimensional joint detections are described for example in Bogo et al.: “Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image.”, Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9909 LNCS, 561-578, https://doi.org/10.1007/978-3-319-46454-1_34.
In a simple variant, the influence of clothing on the person 2 while the images are obtained in step S10 is neglected. In some variants, however, the present clothing may be modelled by a noise term. The silhouettes of the segmentation maps 3 aligned in 3-dimensional space based on the determined key points K may be used to form, or as a basis for interpolation of, a 3-dimensional silhouette of the person. In such an approach, the images may be treated, instead of being consecutive images of a rotating person 2, as images of a non-rotating person 2 taken from a camera virtually revolving around the person 2 so that the entirety of the segmentation map silhouettes form, or at least fit in, the 3-dimensional silhouette. The 3-dimensional silhouette may then be used to define boundary conditions for a process reconstructing the exact geometry of the “scanned” person 2 (i.e. of the “subject”) apart from the naked body of the person 2.
Then, depending on the item of clothing or type of clothing for which a clothing size recommendation is sought, the noise term caused by the clothing worn by the person 2 may be neglected or included. For example, when the item of clothing or type of clothing is a jacket which is usually worn over a sweater, a noise term caused by the scanned person 2 wearing a sweater may be included. If, on the other hand, the same person 2 also requests a clothing size recommendation for an undershirt, the noise term may be neglected since undershirts are usually worn on the naked upper body.
Here a parametric body model is a model with any set of parameters θ=(θ1, θ2, . . . , θn) of R″, 10<n<1000 that describe the geometrical shape, pose and other features of the human body. The number of frames k is preferably in the range from 3 to 60 and the parametric body pose of a subsequent frame is initialized with the one from the previous frame to ensure smoothness over time. In the present example, for every parameter entry θi the mean value θavgi across the entire sequence of k-frames (i.e. of all θki) is taken as a definitive parameter value. The resulting 3-dimensional body can be optionally unposed to a canonical A pose or T pose and the geometry can be intermediately stored in any standard format such as <.obj>, <.fbx> or similar.
In a step S120, a second input tensor {circumflex over (X)} is generated. For example, the resulting geometry can be transformed to:
Both versions A) and B) describe all or a sub-set of the geometrical data within the respective format and are referred to as second input tensor {circumflex over (X)} herein. Both W, H or A represent an ordered list of all geometric vertices v and vertex normals vn or parameter space vertices vp, polygonal face elements f, line elements and others included in the format.
In a step S130, the second input tensor {circumflex over (X)} is input into a second artificial neural network DCNN3 which is a deep convolutional neural network configured and trained to learn a mapping function f2 ({circumflex over (X)})=y2, wherein y2 is a second body measurement vector.
In a step S140, the second body measurement vector y2 is generated by the second artificial neural network based on the input tensor {circumflex over (X)}. y2 preferably has a size in the range of from 5 to 100, wherein each of the 5 to 100 entries indicates one body measurement value. The body measurement vectors y1 and y2 may have the same size and their entries may indicate the same body measurement types, respectively.
The second artificial neural network preferably has between 3 and 200 hidden layers and is trained, e.g., on a dataset of between 100 and 1,000,000 unique instances of a human body's geometry. The training examples may consist of a combination from different datasets merged together as one data set with a common input format transformed into the format of the second input tensor {circumflex over (X)} from the following sources:
The 3-dimensional human bodies can have different poses and the target measurements y for each training sample and were extracted from the up to scale scans or manually taken.
As has been described in the foregoing, both the first and the second body measurement vector y1, y2 may be provided, or it may be decided either based on pre-defined rules and/or on the input of a user which body measurement vector is provided in a specific case. It should be understood that the embodiment or embodiments as described with respect to
The first and/or the second body measurement vector y1, y2 may be stored for the person in a databased, e.g. connected to a unique identifier of the person so that the person, as long as their body measurements do not significantly change, in the future may enter only their unique identifier to be provided with their first and/or second body measurement vector y1, y2.
First, a body measurement is generated according to an embodiment of the method according to the first aspect of the present invention, in particular a first and/or second body measurement vector y1, y2 is generated according to the method as described with respect to
In a step S200, the at least one generated body measurement vector y1, y2 is input into a clothing size recommendation model configured to generate a clothing size recommendation based on the at least one body measurement vector y1, y2.
In a step S210, a clothing size recommendation is generated based on the at least one body measurement.
In the presently described example, the clothing size recommendation in plain words uses a weighted sum fr of two terms to indicate to the person to whom the body measurements belong at least one item of clothing and whether and/or how that item would fit the person.
For example, a user of the method (usually the person to whom the body measurements belong) may select a particular item of clothing, and the clothing size recommendation will indicate at least one of the sizings of said item of clothing which will fit the person the best and/or how well said sizing will fit the person. Alternatively or additionally, the user may also select a number of different items of clothing and will be provided with the same information, either with the best-fitting sizing of each item, or with the best-fitting sizing of all of the items and/or with corresponding information how well said sizing or sizings will fit the person.
In the presently used example, the goal is to predict fr=(frsz1, frsz2, frszM), i.e. a fit rating vector fr for all available product or brand sizings 1, 2, . . . M of a particular item of clothing, wherein the comma indicates a concatenation, and wherein each entry may comprise a fit likelihood and a fit score. The plurality of fit likelihood for all of the sizings may be designated as a fit likelihood vector. In some variants, the fit rating vector fr comprises the fit likelihood vector concatenated with the fit score for the sizing most likely to fit (according to the fit likelihood vector).
The fit rating vector fr is preferably calculated as a weighted sum by:
fr=w
NL
fr
NL
+wL fr
L
In other words, the fit rating vector fr is calculated by a weighted sum of two terms, fNL and fL (with corresponding weighting factors wNL and wL), corresponding to a hybrid combination of a learning (L) and a non-learning (NL) approaches, realized by a learning sub-model LSM and a non-learning sub-model NLSM. The weighting factors wNL and wL may be normalized such that they always add up to one, but can also be defined in other ways.
In some variants, in order to calculate best fitting sizes of clothes, per-product or per-brand sizing data is stored in a database. The sizing data containing the relevant measurements can be derived from the sizing charts/tables provided by the clothing manufacturer. Alternatively, the relevant measurements can be directly exported from CAD software used to design clothes.
A sizing chart SC may be structured as follows in Table 1:
The relevant measurements on the first column refer to the different body measurement types, and the remaining columns to minimum and maximum size values for these measurements for the respective sizings (S, M, L, XL).
The product fit modifier PF may be a number in a zero-mean range normalized to the range −1 to +1 defined by the available fits for a brand B. For example if a brand offers skinny fit, slim fit, regular, and comfort fit for the product category PC: pants, then PF∈(skinny fit: −1; slim fit: −0.5; regular: +0.5; comfort fit: +1).
The product category PC can be for example ‘tops’ (shirts, t-shirts, sweaters, etc.), ‘bottoms’ (pants, jeans, tracksuit, shorts, etc.), ‘outwear’ (suit, blazer, coat, parka) and ‘dresses’ and one may differentiate between men, women and children versions for each, wherein the person may input additional information such as their gender, age and/or the like. The relevant measurements and the boundary values in the sizing chart SC can depend on the product category and the sizing data provided by the brand. The particular sizes depend on the brand's definition and can be converted between regional standards like EU, USA, DE, FR, UK, JP etc.
The learning (L) term is provided by the learning sub-model LSM, which may comprise a general linear model (multivariate regression model) or a third artificial neural network (regression network) with between 1 and 10 hidden layers of 3 to 100 hidden units each. The learning sub-model LSM is trained to predict the fit rating for each relevant body measurement type (depending on the product category, for example for pants: waist circumference, hips circumference, crotch length, in seam length, leg opening, etc.) with the user's relevant body measurements (according to the first and/or the second body measurement vector y1, y2), gender, input height, fit preference and product sizing chart according to the mapping function:
f(SC,EBM,UH,UG,UFP,UR)=f(featuressubject,featurestarget,)=f(ui,pj,s)=frL
wherein SC is the target product/brand sizing chart; EBM is the extracted body measurements of the user according to the first and/or the second body measurement vector y1, y2; UFP is the person's (or: user's) fit preference; UH is the person's (or: user's) height; UG is the person's (or: user's) gender; and UR is a user rating. As illustrated by the middle terms of the above equation, the input of the learning sub-model LSM may be divided in a feature vector ui relating to a particular person (user or “subject”) i, and a feature vector pj,s comprising clothing item information, in particular information related to a specific “target”, i.e. target product/brand j and/or sizing s.
Herein, instead of directly using distance metric, a distance metric (e.g., Mean Squared Error (L2) or similar) is used to calculate the loss of the learning sub-model LSM during training via gradient descent-based optimization. In contrast to the non-learning approach described in the following, here an additional input UR (“user rating”) is introduced. This can be seen as a content-based recommender system which predicts the fit rating given the similarity of person's features to the fit ratings for persons of similar body type, as well as the similarity of products they ordered and kept in the past and/or dissimilarity with products they ordered and sent back in the past (encapsulated e.g. in “return information”), as well as and the similarity of target product/brand, where the similarity measure is implicitly learned. In other words, the user rating may encapsulate information about products the same person has previously ordered and kept and/or about items of clothing the same person has previously returned.
The non-learning (NL) approach uses a distance function φ using a distance metric between the measurements present in the sizing chart of the target product/brand (i.e. parameters of at least one item of clothing) and the same body measurements according to the provided first and/or the second body measurement vector y1, y2.
The fit likelihoods (i.e. a number indicating whether a particular sizing is the best-fitting sizing for the person 2) for each sizing of a certain item of clothing and brand may be calculated by building a sizing vector szsizing for each available sizing as
szSizing=(m1,avg, m2,avg, . . . , mk,avg) with the mean reference values for each of K body measurement types the K measurement points and M available sizes of a product.
The distance function φ may then receive parameters and input and may then output a second fit likelihood vector and/or a fit output score frNL, i.e.:
φ(SC,PF,EBM,UH,UG,UFP)=frNL,
wherein SC is the target product/brand sizing chart; PF is the product fit modifier; EBM is the extracted body measurements of the user according to the first and/or the second body measurement vector y1, y2; UFP is the person's (or: user's) fit preference; UH is the person's (or: user's) height; UG is the person's (or: user's) gender. Any or all of these parameters may be removed or replaced with other suitable parameters, and the user requesting the clothing size recommendation may be asked (e.g. by an app, a web user interface, a web application and/or the like) to input any or all of these parameters.
The distance function φ can use any distance metric such as Mean Square Error (MSE) or the like.
Preferably, depending on how much data is available, the weights between the learning (L) and a non-learning (NL) terms are changed such that more weight is put towards NL when labelled data is scarce and vice versa the weight of the L approach is increased as the dataset grows. This can for example be encoded by providing that:
w
NL=1−wL
w
L=max(Di−τ,0),
wherein Di is the number of users that ordered a particular item and provided fitting information for it (preferably in relationship to at least one of their own body measurements, for example “waist was too narrow” together with information that the user has a specific waist body measurement), τ is a threshold constant between 10 and 1,000,000. The labelled dataset that is used for training the third artificial neural network is automatically growing as users who order using the recommendation and try the clothes on, submit fitting information as feedback (e.g. on the recommendations).
As soon as more than T ratings have been submitted by shoppers for a product wL>0 and the learning sub-model LSM starts to contribute towards the total fit rating vector fr.
The fit rating vector fr (indicating for each sizing how well it fits the person 2) can be converted into a fit likelihoods vector fl for all available sizes by applying a softmax function a over the fit rating vector fr:
The item of clothing and sizing most likely to fit the person is then predicted e.g. via the formula argmaxszi(fl)
In
According to
Three options are in principle available for the user 10 to provide their body measurements. As an option A, the user may use the method according to any embodiment of the first aspect of the present invention, in particular according to any embodiment or embodiments of
If the user 10 is browsing the web shop 20 on a device without a suitable camera, e.g. a desktop PC, the user 10 may be provided automatically with a QR code and/or a URL. The user 10 can then either scan the QR code with a mobile device with a camera, or enter the URL in a mobile device with a camera, as a result again a browser tab with the app will be opened.
Once the user 10 has been guided to the app, they follow the instructions provided automatically by the app to complete the scan and record a video of themselves (or another person 2) rotating. The video can either be recorded alone by placing the mobile device on the ground or a friend nearby can take a video of the user 10. When the measurement extraction is completed the user 10 sees a message to return back to the web shop 20 either on desktop PC or on the mobile device 1. From now on, as will be described in the following, next to every product they see on the web shop 20, they will be automatically shown e.g. the recommended sizes (or sizings) for the top two most likely-to-fit product sizes and the respective fit score (fit rating) for the single most likely-to-fit sizing. Using the app, the user 10 may be prompted to also enter user parameters such as user gender UG, user height UG, user fit preference UFP and so on. Any or all of these may be marked by the app as “required” or as “optional”.
The recorded video, optionally together with acquired angles and orientation of the mobile device 1 over time, and further optionally user parameters (height, gender, fit preference) can be sent to a remote computing device (such as a web server or a virtual machine, VM, on the Cloud) which temporarily stores the video locally and stores the rest in a database attached to the browser session id.
The remote computing device may run the algorithm to select the images/frames according to the method described in the foregoing and reconstruct the human body and extract measurements. The resulting measurements are saved in the database an attached to the browser session id. The video and selected frames are preferably deleted afterwards to ensure privacy. For every new browser session id a (preferably human-readable) unique identifier UID) is generated. It is attached to the initial browser session id on the web shop 20 where the scan was initiated from, but can be reused on any other web shop 20 that employs the methods described herein to directly get recommendations without the person 2 having to be scanned anew.
As an alternative option B, the user 10 may choose to provide manual measurements MM.
As a third alternative option C, the user 10 may input (e.g. into the app) a unique identifier UID which has already been previously provided to them and which encapsulates or indicates the previously generated first and/or the second body measurement vector y1, y2 (or: EBM).
Options B and C may be offered to the user 10 within the app, or may be offered to the user 10 as alternatives to the app, for example as separate buttons shown by the browser displaying the website of the web shop.
In any case, the body measurements provided according to the chosen option A, B, or C are provided to a service provider, in particular to a web shop 20. In addition, user parameters as have been described in the foregoing may also be provided to the web shop 20 such as user fit preference UFP, user height UH and/or user gender UG. In case the user 10 already has a unique identifier UID, such user parameters may already be stored as linked to said unique identifier UID so that, choosing option C, the user 10 conveniently does not have to input them again.
Optionally, in addition return information RI may be provided to (and/or may be already stored at) the web shop 20, for example as has been described in the foregoing. Further optionally, clothing item information CII may be provided to the web shop 20, e.g. from a sizing chart SC.
The user 10 is then provided with a clothing size recommendation REC for at least one item of clothing, for example for a type of pants that the user is interested in. The clothing size recommendation REC may be made according to any embodiment of the method according to the second aspect of the present invention, in particular according to any variant as described with respect to
The user 10 then makes an order ORD of a particular sizing as recommended to the user 10. If the user 10 has used method A and has not previously had a unique identifier UID provided then, as a result of the order ORD (as illustrated schematically in
Said user 10 may then, for their next shopping, choose option C of simply entering their unique identifier UID into a prompt (e.g. in a web interface, a web application, or in any other type), as a result of which the corresponding first and/or the second body measurement vector y1, y2.(or: EBM) will be retrieved from the database DB.
In case the user 10 then wants to return, RET, the ordered item of clothing, the user 10 will have to fill in a user interface 30 inquiring about fitting information of the various body measurements of the user 10 as related to the specific purchased item of clothing being returned. As has been described in the foregoing, a schematic user interface 30 is shown in
As is also illustrated in
Moreover, preferably the third artificial neural network used in the learning sub-model LSM is then trained, or updated, based on the information input by the user 10 into the user interface 30 (or based on the entirety of the return information RI).
In other words, the third artificial neural network may be pre-trained and may then be trained by the return information provided by all of the users 10. The method as outlined above ensures that for the first few (depending on the choice of τ) uses of the user 10 of the described system, the non-learning sub-model NLSM will provide suitable results, and that with increasing return information (for the rare cases in which the NLSM was incorrect) the learning sub-model LSM will improve, and weigh in on the clothing size recommendation, more and more. The users 10 will thus help to train the learning sub-model which guarantees increasing satisfaction of the users 10 with the process described herein. Thus, the present invention also teaches a method for training a learning sub-module, LSM, more specifically an artificial neural network or a learning sub-module, LSM, using return information provided by a plurality of users.
The unique identifier UID is preferably a short code from between 3 and 6 digits and letters of lower or uppercase (e.g. 3G7, 412k, OL65A, or A1b2C3) that is attached to the browser session ID of the web shop 20 the user 10 was routed to from to the scanning app used for acquiring the video of the user 10. After the EBM are extracted from the video, preferably all visual information is deleted and only the information needed for the clothing size recommendations (e.g. EBMs, user height UH, user gender UG and user fit preference UFP) corresponding to the unique identifier UI is stored in the database DB. The unique identifier UI is anonymous in the sense that it only connects a browser session ID and a set of letters and digits to a limited set of values, which are not unique to a specific user/person. No names, emails or other information which makes it easy to identify a person can be inferred from the unique identifier UI.
For further improved data protection, optionally, upon submitting a video by the app to a server, or even already during the acquisition of the video, the face of the person in the video can be hidden e.g. via blurring so identification in the time span between uploading and generating measurements is not possible. The unique identifier UI can be re-used on the same web shop 20 after the browser sessions has expired, when shopping from another device or shared with friends and family to get product size recommendations for the person (user 10) it was generated for.
The unique identifier UI can also be used on other supported web shops 20 to get size recommendations for their fashion products. The unique identifier UI does not grant access to view the concrete measurements, but only to the measurements and/or the recommendations associated with the measurements that were extracted from the video and stored. The unique identifier UI can also be used offline in retail stores to always get the most likely to fit size of clothes and try them on. Here the user 10 opens, for example, a web application in the browser by scanning a code (e.g. a visual code such as a QR code or barcode, or another type of wireless code such as RFID) on the clothes tags/labels or directly navigates to the web application and scans the bar code of the product. If the user does not already have a unique identifier UI from before, the scanning process can also be done in a fitting room in the store.
In the foregoing detailed description, various features are grouped together in the examples with the purpose of streamlining the disclosure. It is to be understood that the above description is intended to be illustrative and not restrictive. It is intended to cover all alternatives, modifications and equivalence. Many other examples will be apparent to one skilled in the art upon reviewing the above specification, taking into account the various variations, modifications and options as described or suggested in the foregoing.
In a short summary, the present invention provides at least a method for determining at least one body measurement by obtaining a plurality of images, in particular a video, of a user. The video can be easily acquired by the user or a friend using a mobile device comprising a camera, such as a digital camera, a smartphone, a table computer and/or the like. Optionally, the invention also provides generating a clothing size recommendation based at least on the determined at least one body measurement. The clothing size recommendation may take additional parameters into account. Both for the determining of the body measurements and/or for the generating of the clothing size recommendation, artificial intelligence entities, in particular machine learning entities, more particular artificial neural networks may be used. Preferably, return information comprising ratings of users of whether and how well certain items of clothing fit them in the past is collected and used for the training of an artificial intelligence entity.
Number | Date | Country | Kind |
---|---|---|---|
19177584.0 | May 2019 | EP | regional |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/EP2020/000106 | 5/29/2020 | WO | 00 |