Patent Application
20200275861
This disclosure relates to 3-D digital modeling and, more particularly, to computer interpretation of image data to generate 3-D digitals of orthotics.
People tend to like products that are customized for them more than generic products. Despite interest in customized products, consumers are less inclined toward customization if obtaining personal specifications is bothersome. Physically measuring oneself is bothersome. Using complex equipment to measure oneself either by oneself, or at the office of a related professional is also bothersome. Most people carry smartphones that include digital cameras and a connection to the Internet. 3-D printers and other programmable manufacturing apparatus enable the generation of custom physical wearables from digital models of users.
By using computer vision techniques, two-dimensional (2-D) and/or three-dimensional (3-D) digital models can be constructed for objects found in image data. The digital models subsequently can be used for numerous activities, including generation of 3-D printed objects sized and shaped to match the objects found in the image data. For example, in some embodiments, images of a human body are used to model at least a portion of the human body, and then customized wearables can be printed for the modeled portion of the human body (e.g., footwear, headwear, undergarments, sportswear, etc.). Depending on the subject object (e.g., body part), the image data includes views of that object and key points on that object in various physical states (e.g., physical positions, weight loads, gravity states, temporal periods) in one or more data types (e.g., video frames, 2D/3D static images, inertial measurements, user preference).
In some embodiments, users are directed to use a mobile device, such as a smartphone including a camera, to take photos of some subject object (e.g., their feet). In some embodiments, different and/or multiple modes of the smartphone camera are used. For example, a given implementation may make use of static 2-D images, static 3-D images, video frames where the user's body is in motion, inertial measurements, and specified user preferences. In this case, a “static” image is one that is not associated with a series of frames in a video. In some embodiments, additional apparatus beyond a given mobile device (or smartphone) is used to collect input data of the user's body. Videos of the user's body in motion may include different poses (e.g., clenched/unclenched, different states of bearing weight, etc.) and/or may include cycles of motion (e.g., walking/jogging/running one or more strides, jumping, flexing, rotating a joint, etc.).
Key points on a target body part and/or associated body parts are attached to visual input (image data static/video). Tracking the movement of those body parts between the various images or video frames provides important data to accurately understand the user's body and the type of wearable item that person would want or need. Using machine learned models, AI, and/or heuristics, the shift/motion of the key points in various body states directs a system to generate a model of a wearable for the user. That wearable is biometrically suited for the user.
As an illustrative example, identifying how a foot arch displaces as weight is applied to it can be used to determine the amount and style of arch support a person needs. Knowing the direction and amount of force that motion of a woman's breast puts on the torso during aerobic activity can similarly be used to determine the amount and style of breast support the woman's needs.
The 3-D models generated based on the user body data are sent to a manufacturing apparatus to generate. In some embodiments, the manufacturing apparatus is a 3-D printer. 3-D printing refers to a process of additive manufacturing. In some embodiments, components are machine generated in custom sizes based off the 3-D model (e.g., laser cut cloth or machine sewed) and then assembled by hand or machine.
The mobile device 22 is a device that is capable of capturing and transmitting images over a network, such as the Internet 30. In practice, a number of mobile devices 22 can be used. In some embodiments, the mobile device 22 is a handheld device. Examples of mobile devices 22 include a smart phone (e.g., Apple iPhone, Samsung Galaxy), a confocal microscopy body scanner, an infrared camera, an ultrasound camera, a digital camera, and a tablet computer (e.g., Apple iPad or Dell Venture 107000). The mobile device 22 is a processor enabled device including a camera 34, an inertial measurement unit 35, a network transceiver 36A, a user interface 38A, and digital storage and memory 40A containing client application software 42.
The camera 34 on the mobile device may be a simple digital camera or a more complex 3-D camera, scanning device, InfraRed device, or video capture device. Examples of 3-D cameras include Intel RealSense cameras or Lytro light field cameras. Further examples of complex cameras may include scanners developed by TOM-CAT Solutions, LLC (the TOM-CAT, or iTOM-CAT), adapted versions of infrared cameras, ultrasound cameras, or adapted versions of intra-oral scanners by 3Shape.
The inertial measurement unit 35 is enabled to track movement of the mobile device 22. Movement may include translation and rotation within 6 degrees-of-freedom as well as acceleration. In some embodiments, the motion tracked may be used to generate a path through space. The path through space may be reduced to a single vector having a starting point and an end point. For example, if held in the hand while running, the mobile device 22 will jostle up and down as the runner sways their arms. A significant portion of this motion is negated over the course of several strides
Simple digital cameras (including no sensors beyond 2-D optical) use reference objects of known size to calculate distances within images. Use of a 3-D camera may reduce or eliminate the need for a reference object because 3-D cameras are capable of calculating distances within a given image without any predetermined sizes/distances in the images.
The mobile device also provides a user interface 38A that is used in connection with the client application software 42. The client application software 42 provides the user with the ability to select various 3-D printed wearable products. The selection of products corresponds with camera instructions for images that the user is to capture. Captured images are delivered over the Internet 30 to the processing server 24.
The processer 32B controls the overall operation of the processing server 24. The processing server 24 receives image data from the mobile device 22. Using the image data, server application software 44 performs image processing, machine learning and computer vision operations that populate characteristics of the user. The server application software 44 includes computer vision tools 46 to aid in the performance of computer vision operations. Examples of computer vision tools 46 include OpenCV or SimpleCV, though other suitable examples are known in the art and may be programmed to identify pixel variations in digital images. Pixel variation data is implemented as taught herein to produce desired results.
In some embodiments, a user or administrative user may perform manual checks and/or edits to the results of the computer vision operations. The manual checks are performed on the manual inspection computer 28 or at a terminal that accesses processing server's 24 resources. The processing server 24 includes a number of premade tessellation model kits 48 corresponding to products that the user selects from the client application software 42. Edits may affect both functional and cosmetic details of the wearable—such edits can include looseness/tightness, and high rise/low rise fit. Edits are further stored by the processing server 24 as observations to improve machine learning algorithms. In some embodiments, modeling software 49 is used to generate models of wearables from input body data.
In some embodiments, the tessellation model kits 48 are used as a starting point from which the processing server 24 applies customizations. Tessellation model kits 48 are a collection of data files that can be used to digitally render an object for 3-D printing and to print the object using the 3-D printer 26. Common file types of tessellation model kits 48 include .3mf, .3dm, .3ds, .blend, .bvh, .c4d, .dae, .dds, .dxf, .fbx, .lwo, .lws, .max, .mtl, .obj, .skp, .stl, .tga, or other suitable file types known in the art. The customizations generate a file for use with a 3-D printer. The processing server 24 is in communication with the manufacturing apparatus 26 in order to print out the user's desired 3-D wearable. In some embodiments, tessellation files 48 are generated on the fly from the input provided to the system. The tessellation file 48 is instead generated without premade input through an image processing, computer vision, and machine learning process.
Any of numerous models of manufacturing apparatus 26 may be used by the system 20. Manufacturing apparatus 26 vary in size and type of generated wearable article. In the case where the 3-D wearable is a bra, for example, one may implement a laser cut cloth. Where the 3-D wearable is an insole, or arch support, one may implement a 3-D printer.
Users of the system may take a number of roles. Some users may be administrators, some may be intended wearers of a 3-D printed product, some users may facilitate obtaining input data for the system, and some may be agents working on behalf of any user type previously mentioned.
In step 204, the server application software analyzes the image data to determine distances between known points or objects on the subject's body part. Example distances include heel to big toe, heel to little toe, joint of big toe horizontally across, and distance from either side of the first to fifth metatarsal bones. This process entails using predetermined or calculable distances based on a reference object or calculated distances with knowledge of camera movement to provide a known distance and angle using stereoscopic images or other 3-D imaging technique. In some embodiments, the reference object can be a piece of standard size paper (such as 8.5″×11″), as mentioned above. The application software then uses known distances to calculate unknown distances associated with the user's body part based on the image.
In step 206, the processing server analyzes the image data for body part curvature and/or key points. The key points may exist both on and off the target body part. Key points that exist off the target body part are used as a control or reference point. The computer vision process seeks an expected curve or stress area associated with the body part and with the type of wearable selected. Key points and curves have corresponding data across each of the images. In each image the key point has potentially shifted based on body part movement.
Once the curve or stress area is found, in step 208, points are plotted on the image data (either from the static frames or video frames) in a coordinate graph (see
Notably, the analysis of
Returning to
In step 212, the system identifies changes to each of the corresponding points across the image data. The data regarding the shift of a given key point illustrates where stress/force is being applied to the body. In step 214, the system identifies the stress and support needs of the body part. In some embodiments, the magnitude of the force is calculated based on overall mass and vectors of movement (including distance and speed traveled). Vectors of movement indicate the regions of the body that shift and move and the direction of that movement. The system identifies a support plan based on where those regions are, and whether that portion of the body is intended to move during the specified sequence of motion or change in physical state.
A support plan includes a position of a support feature based on where stresses are being experienced, and a structure for the support feature based on the magnitude of the stress. For example, depending on whether a user's walking/jogging/running gait is one where the user plants with their heel, or their toes, support is positioned differently in an orthotic. In some embodiments, the support plan includes a rigidity factor that may vary across regions of the sole of the foot (“plantar zones”). The rigidity for each plantar zone refers to how rigid an orthotic is across various points of interface with the foot. Further, the speed of their stride will affect the magnitude of the force experienced, and thus a user with faster, heavier strides will include more padding. Additionally, in some embodiments, the manner the wearer plans to use the orthotic influences the support plan. Wearers who are runners may include a different support plan than wearers who stand in place all day. In a bra example, the magnitude of the force may influence varied bra strap configurations, bra strap thicknesses, padding thicknesses and positioning, and underwire configurations and thicknesses.
In some embodiments, the shift of the points is used generate a gait analysis. The manner in which a given person walks tends to determine the manner in which force is applied to various parts of their body. Each aspect that may be extrapolated from the shift of key points is actionable data for identifying support/stress needs of the body. A trained biometric model is applied to the extrapolated body data to determine a style of wearable to generate that addresses these stress/support needs.
In each image of a given physical condition there are key points 60. The key points 60 correspond to one another across each physical condition. As pictured, key points 60A, 60B, and 60C each correspond to one another. Each of the corresponding key points 60A-C are located at the same location on the foot and have shifted based on changes in the physical condition of the foot.
Prior to biomechanical analysis, the system identifies what body part is subject of the analysis. Based on the target body part, a different analysis, trained machine model, and/or knowledge base is applied to the input data. For example, various types of physical conditions across the data may be used for various orthotics. Examples of physical conditions include supporting different weight loads (as shown in
In step 502, the system performs a computer vision analysis on the image data 54-58. The computer vision analysis identifies anatomical measurements of the body part, as well as identify location of corresponding key points.
In step 504, the system performs a biomechanical evaluation of the various images. The biomechanical analysis may vary based on body part. The biomechanical analysis includes tracking the shift of the key points across the different physical conditions. As part of the biomechanical evaluation, the system may generate a model of the body part. Various embodiments of the body part model may exist as a 3-D point cloud, a set of 2-D coordinates, a set of vectors illustrating movement/shift of key points, a set of force measurements, and/or data describing the body (e.g., an estimated mass).
Based on the body part analyzed, a different biomechanical knowledge base is applied. Based on body type, a different anthropometric database is applied. For example, a user who has fallen arches in their feet uses an anthropometric database/trained model for users with fallen arches. Based on the shift of key points and the applicable anthropometric database/trained model, the system identifies a particular orthotic design (e.g., a starting tessellation kit to work from) that the user needs. The system adjusts the orthotic design for the user's specific measurements.
In step 506, the model of the wearable orthotic is transmitted towards a manufacturing apparatus such as a 3-D printer or a garment generator (e.g., procedural sewing device or automatic clothing laser cutter).
In step 602, the user interface of the mobile device instructs the user how to collect the video data. In some circumstances a partner (“secondary user”) may be necessary to obtain the relevant target body part. Depending on the body part, or sequence of motion for capture, the instructions vary.
A number of sequences of motion are depicted in
Returning to
In another example, where the target body part is a hand (see
Techniques may be employed by the interface instructions to improve consistency. For example, auditory instructions about the current content of the viewfinder aids users who cannot see the viewfinder and do not have the aid of a secondary user (e.g., “take a small step to your left to center yourself in frame”). In another example, instructions direct the primary user to position themselves relative to a reference point that does not move despite repositioning of the camera (e.g., a mark on the ground, etc.). In some embodiments, the reference point may not be an intentionally chosen reference point. That is, the software onboard the camera may identify, via machine vision, a marking (e.g., such as a given point on a patterned floor, or a knot in a hardwood floor) and provide auditory instructions for the primary user to stand relative to that reference point without identifying the reference point to the user in the instructions.
In step 604, the processing server receives video data as collected from the mobile device. Once received, in step 606 and 608, the processing server performs machine vision operations on the acquired video data to determine size and curvature specifications for the user's applicable body part in different physical states (throughout the sequence of motion). In step 606, distances are mapped differently based on the manner in which the video data was gathered. In some embodiments, the video includes reference objects of known sizes. The reference object enables various other lengths to be derived.
In some embodiments, stereoscopic viewpoints can be used to identify distances/sizes. Numerous methods exist to obtain stereoscopic viewpoints. In some embodiments, some cameras include multiple lenses and naturally capture image data where derived depth is included in image meta data. In some single lens embodiments, where a secondary user operates the camera, the UI of the camera is instructed to shift the camera (as tracked by the IMU) prior to initiation of the sequence of movement. Frames captured during the initial shift of the camera enable the derivation of distances captured in later frames.
In some embodiments, a single user repositions the camera themselves and cannot guarantee consistent position of their body between the multiple stereoscopic viewpoints. In these embodiments, the camera may instead identify a reference points that is off the body of the primary user (e.g., static markings on the floor, heavy furniture, etc.). Reference points with a high certainty (determined via machine learning/machine vision) of static positioning are usable to determine a reference size in the video frames. Using the reference size, a remainder of distances and sizes included in the video data (including the target body part) may be derived.
In step 608, the processing server analyzes the video data for body part curvature and/or key points. The key points may exist both on and off the target body part (e.g., where a breast is the target body part, a key point may be on the user's breast and on the user's sternum). Key points that exist off the target body part are used as a control or reference point. The machine vision process seeks an expected curve or stress area associated with the body part and with the type of wearable selected. Key points and curves have corresponding data across a number of frames of the video data. In each frame, the key point has potentially shifted based on body part movement.
Once the curve or stress area is found, in step 610, points are plotted in 3-D space (either from video frames or static images) in a coordinate space. In some embodiments, the plotted points can be normalized for motion (e.g., correct for translational movement). Taken together, the collection of points model the transition the body part takes through the sequence of motion.
In step 610, the system identifies changes to each of the corresponding points across the video data. The data regarding the shift of a given key point illustrates where stress/force is being applied to the body, and how much force is present. Step 610 can be performed for any body part. However, as an example, a process for determining force/stress experienced by various portions of a breast during a sequence of motion is discussed below.
Tracking the above points through the sequence of motion enables a determination of where stress is applied to the body and provides an approximation of the magnitude of that stress. For example, because force=(mass)(acceleration), force can be determined based on a derived value for breast mass and the acceleration of a given key point on the breast. Mass can be approximated using volume and density.
Volume can be calculated using derived dimensions (see
The system calculates acceleration of various key points 86, 88, 90, 92 based on translational movement during the video data using the known length of the video data. Distance over time is velocity, and the derivative of velocity is acceleration. Thus, using the derived values for breast mass and acceleration at a given key point, the system computes the force experienced at that key point.
Calculating a nipple movement index (a combination of the displacement of the nipple and the acceleration of the nipple) based on the key points is a further statistic that may be used to evaluate a desirable support feature in a wearable.
Returning to
Other secondary sources of data include static image data (see
In
Each data type has advantages and disadvantages. Video data may not be as precise as static image data but does provide insight into physical transitions the body goes through while performing a sequence of movement. However, the disparate data sources cannot be inherently compared to one another.
In step 900, input body part image data is provided to the system. The input data may be provided in various ways (e.g., through direct upload from smartphone applications, web uploads, API uploads, partner application uploads, etc.). Initial input data describes a sequence of motion; examples may include uncategorized video frames, or a history of acceleration received by an IMU. The video frames/IMU data includes a known sequence of motion of a known target body part. “Uncategorized” refers to unknown physical conditions and weight loading states from frame to frame. Within a given sequence of motion, there are extremities (e.g., greatest weight loading/least weight loading, least to most extension, etc.). Identifying the frames where the body part reaches extremities enables the system to evaluate various sources of input with respect to one another. Data sources that are comparable construct better models than data that is evaluated in isolation. Static images (see
In step 902, the system prepares the sequence of motion data for categorization. In steps 904 and 906, the system detects the points during a sequence of motion where extremities are reached. This is performed both through computer vision and machine learning. For example, computer vision may analyze frames stepwise to identify where full extension of a body part is reached, whereas a trained machine learning model has a comparative background (the model training) for what a body part of the type being evaluated looks like when at a given extremity. Prior observations and models (e.g., a hidden Markov model) influence the machine learning operation.
In step 908, the system checks whether frames embodying the extremities are identified. Where system certainty is low, the method performs a feedback loop (810). In some embodiments, the user interface will additionally signal the user and the user may initiate the method again form the beginning. Where frames are identified as having extremities, the method proceeds to step 912.
In step 912, the system aligns disparate data sources to one another for comparison. Static images that are already labeled as extremity points are matched to those frames that are identified as extremities in steps 904 and 906. Static frames that include intermediate physical conditions (e.g., partial weight loading) are aligned with frames between the extremity frames.
In step 914, the system builds a model of the body part. In step 916, the system considers the user's preferences on worn orthotics. In some circumstances, the user's preferences are reflected in their body model. That is, their preferences are consistent with what is recommended based on their anatomy. Where the preferences are consistent, in step 918, a 3-D model of the orthotic is generated according to model recommendation and the rest of the orthotic printing process continues separately.
Where the user's preferences are not validated by the body model, in step 920, the system determines whether to override the user's preferences. Overriding a user's preferences is based on a degree of deviation implementing the user's preferences in the orthotic would cause to an orthotic built based purely on a recommendation using the body model of step 914. Where the degree of deviation is below a threshold, the method proceeds to step 922, and generates an orthotic model that is influenced by the user's preferences.
Where the degree of deviation is above a threshold, the user may be queried regarding their preferences. Where the user is insistent on their preferences, the method similarly proceeds to step 922. Where the threshold is exceeded, and the user does not insist upon implementation of their preference, the method proceeds to step 918, and generates an orthotic according to model recommendation. The rest of the orthotic printing process continues separately.
In step 924, the system adds the transmitted images to the total observations. In step 926, the system enables users, or administrators to do an audit review.
After steps 924-926, the data is added to a database. The process of
Although the invention is described herein with reference to the preferred embodiment, one skilled in the art will readily appreciate that other applications may be substituted for those set forth herein without departing from the spirit and scope of the present invention. Accordingly, the invention should only be limited by the claims included below.
