Patent Application
20210165561
Design of objects is often facilitated by tools which allow user input to create a model of the desired object. For example computer-aided design tools allow a user to create a three-dimensional object model and display the object model in two dimensions (e.g., plan view) or three dimensions (perspective view). The user may create edges or surfaces of the desired object model and change the features. Creating the object model in such a tool may precede and facilitate production of the three-dimensional object.
For a more complete understanding of various examples, reference is now made to the following description taken in connection with the accompanying drawings in which:
As noted above, tools for designing of an object allow a user to generate an object model. Creation of a model using such tools typically calls for a level of expertise from the user. Further, creation of an accurate model can be time consuming and inefficient.
Various examples described herein relate to generation of a shape or a model of an object based on a sketch provided by a user. In various examples, a three-dimensional model of an object may be provided to a user based on a 2- or 3-dimensional sketch. Example systems are provided with a user interface that allows a user to input a sketch, such as on a 2D plane or a 3D virtual reality input, for example. The input sketch may be used to match a model of an object in a object model reservoir, or database. In various examples, the system includes a generator which uses an artificial intelligence (AI) agent to generate models which are not in the reservoir and may add the additional models generated to the reservoir. In one example, the generator uses an input, such as the matched model for the sketch input by the user, and converts the input into a latent vector. The generator processes the vector and outputs a binary 3D matrix which can represent different objects. A discriminator may be provided to filter out unrealistic models. In some examples, the system may iteratively select a matching object after the addition of newly generated objects to the reservoir. In some examples, the discriminator is activated during a training phase and is de-activated during operation with user input.
Referring now to the Figures,
The example system 100 is further provided with an object model reservoir 120. The object model reservoir 120 may be a database or other store of electronic models of various objects. The reservoir 120 may include any practical number of models, and the models may be stored in categories of objects. For example, the models may be stored in separate libraries corresponding to categories such as airplanes, automobiles, buildings, etc. In some examples, the object model reservoir 120 includes models of three-dimensional objects stored therein. For example, the reservoir 120 may include voxel representations of various three-dimensional objects.
In the example system 100 illustrated in
The example system 100 of
The additional models generated by the generator 140 may then be added to the object model reservoir 120. Additionally, in some examples, certain models added in a previous iteration may be removed from the object model reservoir 120. For example, the lowest scoring models or models with a score below a secondary threshold may be deleted. In some examples, the sample matching portion 130 may perform further matching of the sketch input from the sketch interface 110 with models in the object model reservoir 120, including the additional models generated by the generator 140. Further, the sample matching portion 130 may update the matching based on updated sketch input. For example, a user may first sketch an airplane including the fuselage and wings only. The sample matching portion 130 may perform a comparison and identify a best match from the object model reservoir 120. When the user adds jet engines or propellers to the sketch, the sample matching portion 130 may update the match.
Referring now to
The example system 200 of
The latent space vector representation 242 of the AI agent 240 is provided to generate a vector representation of latent space around the object model match output by the sample matching portion 230. In various examples, the latent space vector may be generated from a voxel representation of the object model match. In this regard, various latent space vectors may be sampled around the input latent space vector which the generator 244 used to generate the matched object model. For example, the sample matching portion 230 may provide a 64×64×64 binary voxel representation of the matched object.
In one example, the latent vector is determined as:
z−Alpha*z+(1−Alpha)*t,
where Alpha is an interpolation rate, z is the current latent vector, and t is one of the anchor latent vector for a category of object (e.g., aircraft). In one example, Alpha is set at 0.8. In other examples a different value of Alpha between 0 and 1 may be selected.
The generator 244 may use the latent space vector to generate additional models. In one example, the generator 244 takes random N-dimensional vectors and turns them into 3D volumetric objects. For example, generator may use convolution layers to generate additional 64×64×64 binary matrices from the latent space vector. Each 64×64×64 matrix may represent an additional candidate model.
In one example, the AI agent may be trained in an offline mode using a generative adversarial neural network (GAN). In this example, the generator 244 and the discriminator 246 act as a balance to each other. The discriminator 246 may be provided to eliminate selected 3-D representations as unrealistic. For example, certain candidate models generated by the generator 244 may be difficult or impossible to exist. As an example, the generator 244 may generate a candidate model that has a component (e.g., a wing) detached from the main body (e.g., aircraft fuselage). The discriminator 246 can recognize and eliminate such candidates before they are added to the object model reservoir 220.
In one example, the elimination of candidate models by the discriminator 246 may be based on a confidence value generated by the discriminator. The discriminator uses the matrix (e.g., the 64×64×64 voxel representation) output by the generator 244 and outputs a real number between 0 and 1 which may be used as the confidence value. A threshold value may be selected to determine whether the voxel representation is to be added to the object model reservoir 220 or is to be eliminated.
In various examples, the adversarial relationship between the generator 244 and the discriminator 246 may be exploited during a training phase, or an offline mode. The training may continue until the discriminator 246 is unable to distinguish the object models generated by the generator 244 from various reference objects. During an online mode, such as during receipt of an input sketch from a user, the discriminator 246 may be de-activated, allowing the generator to generate object models at an increased rate which may be appropriate for interactive operation with the user. Accordingly,
Referring now to
The example method 300 further includes identifying a matching object model for the sketch input from a reservoir of object models (block 320). In this regard, features may be extracted from the sketch input and compared with features of various models in the reservoir of object models. A best match or multiple candidate matches may be provided as a result of the matching.
The example method 300 further includes generating additional models of objects (block 330). In various examples, the generation of additional models may be based on the matching object model identified in block 320. As described above, the generation of additional models may be facilitated by an AI agent.
Referring now to
The example method 400 further includes identifying a matching object model for the sketch input from a reservoir of object models (block 420). As described above, features may be extracted from the sketch input and compared with features of various models in the reservoir of object models, and a best match or multiple candidate matches may be provided as a result of the matching.
The example method 400 further includes generating additional models of objects using a latent space vector representation (block 430). As described above with reference to the example system 200 of
The additional models generated using the latent space vector representation, after elimination of the unrealistic models, are then added to the reservoir (block 440). The process may then return to block 420 and iteratively repeat the steps. In this regard, additional models may be generated and added to the reservoir in an offline mode. For example, the process may continue even after the user has been provided with a best match to continue to generate additional models and add them to the reservoir.
Referring now to
The example instructions include receive sketch input instructions 521. In this regard, a sketch input may be received from a user through a sketch interface. As noted above, the sketch interface may be a two-dimensional input or a three-dimensional input.
The example instructions further include identify matching object model instructions 522. As described above, features may be extracted from the sketch input provided by the user. The extracted features may be compared with features of object models stored in a reservoir, and a best match may be identified.
The example instructions further include generate additional models instructions 523. As noted above, based on the match identified, additional models may be generated using, for example, an AI agent. In some examples, the additional models may be added to the reservoir of models.
Thus, various examples described above can allow a user to provide a sketch of a desired object to obtain a model (e.g., a voxel representation) of an object. Users with little or no expertise can generate such models since only a sketch input is used.
Software implementations of various examples can be accomplished with standard programming techniques with rule-based logic and other logic to accomplish various database searching steps or processes, correlation steps or processes, comparison steps or processes and decision steps or processes.
The foregoing description of various examples has been presented for purposes of illustration and description. The foregoing description is not intended to be exhaustive or limiting to the examples disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of various examples. The examples discussed herein were chosen and described in order to explain the principles and the nature of various examples of the present disclosure and its practical application to enable one skilled in the art to utilize the present disclosure in various examples and with various modifications as are suited to the particular use contemplated. The features of the examples described herein may be combined in all possible combinations of methods, apparatus, modules, systems, and computer program products.
It is also noted herein that while the above describes examples, these descriptions should not be viewed in a limiting sense. Rather, there are several variations and modifications which may be made without departing from the scope as defined in the appended claims.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2018/036840 | 6/11/2018 | WO | 00 |
