Patent Application
20240135513
Recent years have seen significant advancement in hardware and software platforms for performing computer vision and image editing tasks. Indeed, systems provide a variety of image-related tasks, such as object identification, classification, segmentation, composition, style transfer, image inpainting, etc.
One or more embodiments described herein provide benefits and/or solve one or more problems in the art with systems, methods, and non-transitory computer-readable media that implement artificial intelligence models to facilitate flexible and efficient scene-based image editing. To illustrate, in one or more embodiments, a system utilizes one or more machine learning models to learn/identify characteristics of a digital image, anticipate potential edits to the digital image, and/or generate supplementary components that are usable in various edits. Accordingly, the system gains an understanding of the two-dimensional digital image as if it were a real scene, having distinct semantic areas reflecting real-world (e.g., three-dimensional) conditions. Further, the system enables the two-dimensional digital image to be edited so that the changes automatically and consistently reflect the corresponding real-world conditions without relying on additional user input. In one or more embodiments, the system performs scene-based image editing by performing a variety of infill modifications, human inpainting, reposing of subjects portrayed within digital images, and facial expression transfer and animation. Moreover, for performing the aforementioned infill modifications, human inpainting, reposing, and facial expression transfer and animation, the system utilizes various generative models and various instances of artificial intelligence to generate modified digital images or animations. Thus, the system facilitates flexible and intuitive editing of digital images while efficiently reducing the user interactions typically required to make such edits.
Additional features and advantages of one or more embodiments of the present disclosure are outlined in the description which follows, and in part will be obvious from the description, or may be learned by the practice of such example embodiments.
This disclosure will describe one or more embodiments of the invention with additional specificity and detail by referencing the accompanying figures. The following paragraphs briefly describe those figures, in which:
One or more embodiments described herein include a scene-based image editing system that implements scene-based image editing techniques using intelligent image understanding. Indeed, in one or more embodiments, the scene-based image editing system utilizes one or more machine learning models to process a digital image in anticipation of user interactions for modifying the digital image. For example, in some implementations, the scene-based image editing system performs operations that build a knowledge set for the digital image and/or automatically initiate workflows for certain modifications before receiving user input for those modifications. Based on the pre-processing, the scene-based image editing system facilitates user interactions with the digital image as if it were a real scene reflecting real-world conditions. For instance, the scene-based image editing system enables user interactions that target pre-processed semantic areas (e.g., objects that have been identified and/or masked via pre-processing) as distinct components for editing rather than target the individual underlying pixels. Further, the scene-based image editing system automatically modifies the digital image to consistently reflect the corresponding real-world conditions.
As indicated above, in one or more embodiments, the scene-based image editing system utilizes machine learning to process a digital image in anticipation of future modifications. In particular, in some cases, the scene-based image editing system employs one or more machine learning models to perform preparatory operations that will facilitate subsequent modification. In some embodiments, the scene-based image editing system performs the pre-processing automatically in response to receiving the digital image. For instance, in some implementations, the scene-based image editing system gathers data and/or initiates a workflow for editing the digital image before receiving user input for such edits. Thus, the scene-based image editing system allows user interactions to directly indicate intended edits to the digital image rather than the various preparatory steps often utilized for making those edits.
As an example, in one or more embodiments, the scene-based image editing system pre-processes a digital image to facilitate object-aware modifications. In particular, in some embodiments, the scene-based image editing system pre-processes a digital image in anticipation of user input for manipulating one or more semantic areas of a digital image, such as user input for moving or deleting one or more objects within the digital image.
To illustrate, in some instances, the scene-based image editing system utilizes a segmentation neural network to generate, for each object portrayed in a digital image, an object mask. In some cases, the scene-based image editing system utilizes a hole-filing model to generate, for each object (e.g., for each corresponding object mask), a content fill (e.g., an inpainting segment). In some implementations, the scene-based image editing system generates a completed background for the digital image by pre-filling object holes with the corresponding content fill. Accordingly, in one or more embodiments, the scene-based image editing system pre-processes the digital image in preparation for an object-aware modification, such as a move operation or a delete operation, by pre-generating object masks and/or content fills before receiving user input for such a modification.
Thus, upon receiving one or more user inputs targeting an object of the digital image for an object-aware modification (e.g., a move operation or a delete operation), the scene-based image editing system leverages the corresponding pre-generated object mask and/or content fill to complete the modification. For instance, in some cases, the scene-based image editing system detects, via a graphical user interface displaying the digital image, a user interaction with an object portrayed therein (e.g., a user selection of the object). In response to the user interaction, the scene-based image editing system surfaces the corresponding object mask that was previously generated. The scene-based image editing system further detects, via the graphical user interface, a second user interaction with the object (e.g., with the surfaced object mask) for moving or deleting the object. Accordingly, the moves or deletes the object, revealing the content fill previously positioned behind the object.
Additionally, in one or more embodiments, the scene-based image editing system pre-processes a digital image to generate a semantic scene graph for the digital image. In particular, in some embodiments, the scene-based image editing system generates a semantic scene graph to map out various characteristics of the digital image. For instance, in some cases, the scene-based image editing system generates a semantic scene graph that describes the objects portrayed in the digital image, the relationships or object attributes of those objects, and/or various other characteristics determined to be useable for subsequent modification of the digital image.
In some cases, the scene-based image editing system utilizes one or more machine learning models to determine the characteristics of the digital image to be included in the semantic scene graph. Further, in some instances, the scene-based image editing system generates the semantic scene graph utilizing one or more predetermined or pre-generated template graphs. For instance, in some embodiments, the scene-based image editing system utilizes an image analysis graph, a real-world class description graph, and/or a behavioral policy graph in generating the semantic scene.
Thus, in some cases, the scene-based image editing system uses the semantic scene graph generated for a digital image to facilitate modification of the digital image. For instance, in some embodiments, upon determining that an object has been selected for modification, the scene-based image editing system retrieves characteristics of the object from the semantic scene graph to facilitate the modification. To illustrate, in some implementations, the scene-based image editing system executes or suggests one or more additional modifications to the digital image based on the characteristics from the semantic scene graph.
As one example, in some embodiments, upon determining that an object has been selected for modification, the scene-based image editing system provides one or more object attributes of the object for display via the graphical user interface displaying the object. For instance, in some cases, the scene-based image editing system retrieves a set of object attributes for the object (e.g., size, shape, or color) from the corresponding semantic scene graph and presents the set of object attributes for display in association with the object.
In some cases, the scene-based image editing system further facilitates user interactivity with the displayed set of object attributes for modifying one or more of the object attributes. For instance, in some embodiments, the scene-based image editing system enables user interactions that change the text of the displayed set of object attributes or select from a provided set of object attribute alternatives. Based on the user interactions, the scene-based image editing system modifies the digital image by modifying the one or more object attributes in accordance with the user interactions.
As another example, in some implementations, the scene-based image editing system utilizes a semantic scene graph to implement relationship-aware object modifications. To illustrate, in some cases, the scene-based image editing system detects a user interaction selecting an object portrayed in a digital image for modification. The scene-based image editing system references the semantic scene graph previously generated for the digital image to identify a relationship between that object and one or more other objects portrayed in the digital image. Based on the identified relationships, the scene-based image editing system also targets the one or more related objects for the modification.
For instance, in some cases, the scene-based image editing system automatically adds the one or more related objects to the user selection. In some instances, the scene-based image editing system provides a suggestion that the one or more related objects be included in the user selection and adds the one or more related objects based on an acceptance of the suggestion. Thus, in some embodiments, the scene-based image editing system modifies the one or more related objects as it modifies the user-selected object.
In one or more embodiments, in addition to pre-processing a digital image to identify objects portrayed as well as their relationships and/or object attributes, the scene-based image editing system further pre-processes a digital image to aid in the removal of distracting objects. For example, in some cases, the scene-based image editing system utilizes a distractor detection neural network to classify one or more objects portrayed in a digital image as subjects of the digital image and/or classify one or more other objects portrayed in the digital image as distracting objects. In some embodiments, the scene-based image editing system provides a visual indication of the distracting objects within a display of the digital image, suggesting that these objects be removed to present a more aesthetic and cohesive visual result.
Further, in some cases, the scene-based image editing system detects the shadows of distracting objects (or other selected objects) for removal along with the distracting objects. In particular, in some cases, the scene-based image editing system utilizes a shadow detection neural network to identify shadows portrayed in the digital image and associate those shadows with their corresponding objects. Accordingly, upon removal of a distracting object from a digital image, the scene-based image editing system further removes the associated shadow automatically.
Moreover, in some embodiments, the scene-based image editing system generates modified digital images portraying human subjects. For example, in some implementations the scene-based image editing system 106 receives an indication to expand a digital image frame or remove a portion of a digital image portraying a human. For instance, in some cases, the scene-based image editing system utilizes generative machine learning models to generate infill modifications for the digital image. Further, the infill modifications aid in completing a digital image based on indicated modifications (e.g., uncropping/expansion and/or removal of a portion of a digital image). In other instances, the scene-based image editing system also generates modified digital images by inpainting indicated portions of a digital image. In particular, the scene-based image editing system utilizes a structure guidance map and machine learning models to perform partial human inpainting.
Furthermore, in additional embodiments, the scene-based image editing system reposes subjects within a digital image to generate a modified digital image. In particular, the scene-based image editing system utilizes generative models to performs human reposing for human(s) portrayed in a source digital image according to a target pose. In addition to reposing, the scene-based image editing system also generates modified digital images or animations of static digital images by utilizing facial expression transfer and facial expression animation. In particular, the scene-based image editing system transfers the facial expression(s) from a target digital image to a source digital image and for animation, the scene-based image editing system animates a static image with a target digital video.
The scene-based image editing system provides advantages over conventional systems. Indeed, conventional image editing systems suffer from several technological shortcomings that result in inflexible and inefficient operation. To illustrate, conventional systems are typically inflexible in that they rigidly perform edits on a digital image on the pixel level. In particular, conventional systems often perform a particular edit by targeting pixels individually for the edit. Accordingly, such systems often rigidly require user interactions for editing a digital image to interact with individual pixels to indicate the areas for the edit. Additionally, many conventional systems (e.g., due to their pixel-based editing) require users to have a significant amount of deep, specialized knowledge in how to interact with digital images, as well as the user interface of the system itself, to select the desired pixels and execute the appropriate workflow to edit those pixels.
Additionally, conventional image editing systems often fail to operate efficiently. For example, conventional systems typically require a significant amount of user interaction to modify a digital image. Indeed, in addition to user interactions for selecting individual pixels, conventional systems typically require a user to interact with multiple menus, sub-menus, and/or windows to perform the edit. For instance, many edits may require multiple editing steps using multiple different tools. Accordingly, many conventional systems require multiple interactions to select the proper tool at a given editing step, set the desired parameters for the tool, and utilize the tool to execute the editing step.
The scene-based image editing system operates with improved flexibility when compared to conventional systems. In particular, the scene-based image editing system implements techniques that facilitate flexible scene-based editing. For instance, by pre-processing a digital image via machine learning, the scene-based image editing system allows a digital image to be edited as if it were a real scene, in which various elements of the scene are known and are able to be interacted with intuitively on the semantic level to perform an edit while continuously reflecting real-world conditions. Indeed, where pixels are the targeted units under many conventional systems and objects are generally treated as groups of pixels, the scene-based image editing system allows user interactions to treat whole semantic areas (e.g., objects) as distinct units. Further, where conventional systems often require deep, specialized knowledge of the tools and workflows needed to perform edits, the scene-based editing system offers a more intuitive editing experience that enables a user to focus on the end goal of the edit.
Further, the scene-based image editing system operates with improved efficiency when compared to conventional systems. In particular, the scene-based image editing system implements a graphical user interface that reduces the user interactions required for editing. Indeed, by pre-processing a digital image in anticipation of edits, the scene-based image editing system reduces the user interactions that are required to perform an edit. Specifically, the scene-based image editing system performs many of the operations required for an edit without relying on user instructions to perform those operations. Thus, in many cases, the scene-based image editing system reduces the user interactions typically required under conventional systems to select pixels to target for editing and to navigate menus, sub-menus, or other windows to select a tool, select its corresponding parameters, and apply the tool to perform the edit. By implementing a graphical user interface that reduces and simplifies user interactions needed for editing a digital image, the scene-based image editing system offers improved user experiences on computing devices—such as tablets or smart phone devices—having relatively limited screen space.
Additional detail regarding the scene-based image editing system will now be provided with reference to the figures. For example,
Although the system 100 of
The server(s) 102, the network 108, and the client devices 110a-110n are communicatively coupled with each other either directly or indirectly (e.g., through the network 108 discussed in greater detail below in relation to
As mentioned above, the system 100 includes the server(s) 102. In one or more embodiments, the server(s) 102 generates, stores, receives, and/or transmits data including digital images and modified digital images. In one or more embodiments, the server(s) 102 comprises a data server. In some implementations, the server(s) 102 comprises a communication server or a web-hosting server.
In one or more embodiments, the image editing system 104 provides functionality by which a client device (e.g., a user of one of the client devices 110a-110n) generates, edits, manages, and/or stores digital images. For example, in some instances, a client device sends a digital image to the image editing system 104 hosted on the server(s) 102 via the network 108. The image editing system 104 then provides options that the client device may use to edit the digital image, store the digital image, and subsequently search for, access, and view the digital image. For instance, in some cases, the image editing system 104 provides one or more options that the client device may use to modify objects within a digital image.
In one or more embodiments, the client devices 110a-110n include computing devices that access, view, modify, store, and/or provide, for display, digital images. For example, the client devices 110a-110n include smartphones, tablets, desktop computers, laptop computers, head-mounted-display devices, or other electronic devices. The client devices 110a-110n include one or more applications (e.g., the client application 112) that can access, view, modify, store, and/or provide, for display, digital images. For example, in one or more embodiments, the client application 112 includes a software application installed on the client devices 110a-110n. Additionally, or alternatively, the client application 112 includes a web browser or other application that accesses a software application hosted on the server(s) 102 (and supported by the image editing system 104).
To provide an example implementation, in some embodiments, the scene-based image editing system 106 on the server(s) 102 supports the scene-based image editing system 106 on the client device 110n. For instance, in some cases, the scene-based image editing system 106 on the server(s) 102 learns parameters for a neural network(s) 114 for analyzing and/or modifying digital images. The scene-based image editing system 106 then, via the server(s) 102, provides the neural network(s) 114 to the client device 110n. In other words, the client device 110n obtains (e.g., downloads) the neural network(s) 114 with the learned parameters from the server(s) 102. Once downloaded, the scene-based image editing system 106 on the client device 110n utilizes the neural network(s) 114 to analyze and/or modify digital images independent from the server(s) 102.
In alternative implementations, the scene-based image editing system 106 includes a web hosting application that allows the client device 110n to interact with content and services hosted on the server(s) 102. To illustrate, in one or more implementations, the client device 110n accesses a software application supported by the server(s) 102. In response, the scene-based image editing system 106 on the server(s) 102 modifies digital images. The server(s) 102 then provides the modified digital images to the client device 110n for display.
Indeed, the scene-based image editing system 106 is able to be implemented in whole, or in part, by the individual elements of the system 100. Indeed, although
As mentioned, in one or more embodiments, the scene-based image editing system 106 manages a two-dimensional digital image as a real scene reflecting real-world conditions. In particular, the scene-based image editing system 106 implements a graphical use interface that facilitates the modification of a digital image as a real scene.
As shown in
As illustrated in
As shown in
Further, the digital image 206 includes a plurality of individual pixels that collectively portray various semantic areas. For instance, the digital image 206 portrays a plurality of objects, such as the objects 208a-208c. While the pixels of each object are contributing to the portrayal of a cohesive visual unit, they are not typically treated as such. Indeed, a pixel of a digital image is typically inherently treated as an individual unit with its own values (e.g., color values) that are modifiable separately from the values of other pixels. Accordingly, conventional systems typically require user interactions to target pixels individually for modification when making changes to a digital image.
As illustrated in
To illustrate, as shown in
In one or more embodiments, the scene-based image editing system 106 pre-processes the digital image 206 by learning characteristics of the digital image 206. For instance, in some cases, the scene-based image editing system 106 segments the digital image 206, identifies objects, classifies objects, determines relationships and/or attributes of objects, determines lighting characteristics, and/or determines depth/perspective characteristics. In some embodiments, the scene-based image editing system 106 pre-processes the digital image 206 by generating content for use in modifying the digital image 206. For example, in some implementations, the scene-based image editing system 106 generates an object mask for each portrayed object and/or generates a content fill for filling in the background behind each portrayed object. Background refers to what is behind an object in an image. Thus, when a first object is positioned in front of a second object, the second object forms at least part of the background for the first object. Alternatively, the background comprises the furthest element in the image (often a semantic area like the sky, ground, water, etc.). The background for an object, in or more embodiments, comprises multiple object/semantic areas. For example, the background for an object can comprise part of another object and part of the furthest element in the image. The various pre-processing operations and their use in modifying a digital image will be discussed in more detail below with reference to the subsequent figures.
As shown in
As further shown in
As illustrated, upon deleting the object 208c from the digital image 206, the scene-based image editing system 106 automatically reveals background pixels that have been positioned in place of the object 208c. Indeed, as mentioned, in some embodiments, the scene-based image editing system 106 pre-processes the digital image 206 by generating a content fill for each portrayed foreground object. Thus, as indicated by
Thus, the scene-based image editing system 106 operates with improved flexibility when compared to many conventional systems. In particular, the scene-based image editing system 106 implements flexible scene-based editing techniques in which digital images are modified as real scenes that maintain real-world conditions (e.g., physics, environment, or object relationships). Indeed, in the example shown in
Further, the scene-based image editing system 106 operates with improved efficiency. Indeed, by segmenting the digital image 206 and generating the content fill 210 in anticipation of a modification that would remove the object 208c from its position in the digital image 206, the scene-based image editing system 106 reduces the user interactions that are typically required to perform those same operations under conventional systems. Thus, the scene-based image editing system 106 enables the same modifications to a digital image with less user interactions when compared to these conventional systems.
As just discussed, in one or more embodiments, the scene-based image editing system 106 implements object-aware image editing on digital images. In particular, the scene-based image editing system 106 implements object-aware modifications that target objects as cohesive units that are interactable and can be modified.
Indeed, many conventional image editing systems are inflexible and inefficient with respect to interacting with objects portrayed in a digital image. For instance, as previously mentioned, conventional systems are often rigid in that they require user interactions to target pixels individually rather than the objects that those pixels portray. Thus, such systems often require a rigid, meticulous process of selecting pixels for modification. Further, as object identification occurs via user selection, these systems typically fail to anticipate and prepare for potential edits made to those objects.
Further, many conventional image editing systems require a significant amount of user interactions to modify objects portrayed in a digital image. Indeed, in addition to the pixel-selection process for identifying objects in a digital image—which can require a series of user interactions on its own—conventional systems may require workflows of significant length in which a user interacts with multiple menus, sub-menus, tool, and/or windows to perform the edit. Often, performing an edit on an object requires multiple preparatory steps before the desired edit is able to be executed, requiring additional user interactions.
The scene-based image editing system 106 provides advantages over these systems. For instance, the scene-based image editing system 106 offers improved flexibility via object-aware image editing. In particular, the scene-based image editing system 106 enables object-level—rather than pixel-level or layer level—interactions, facilitating user interactions that target portrayed objects directly as cohesive units instead of their constituent pixels individually.
Further, the scene-based image editing system 106 improves the efficiency of interacting with objects portrayed in a digital image. Indeed, previously mentioned, and as will be discussed further below, the scene-based image editing system 106 implements pre-processing operations for identifying and/or segmenting for portrayed objects in anticipation of modifications to those objects. Indeed, in many instances, the scene-based image editing system 106 performs these pre-processing operations without receiving user interactions for those modifications. Thus, the scene-based image editing system 106 reduces the user interactions that are required to execute a given edit on a portrayed object.
In some embodiments, the scene-based image editing system 106 implements object-aware image editing by generating an object mask for each object/semantic area portrayed in a digital image. In particular, in some cases, the scene-based image editing system 106 utilizes a machine learning model, such as a segmentation neural network, to generate the object mask(s).
In one or more embodiments, an object mask includes a map of a digital image that has an indication for each pixel of whether the pixel corresponds to part of an object (or other semantic area) or not. In some implementations, the indication includes a binary indication (e.g., a “1” for pixels belonging to the object and a “0” for pixels not belonging to the object). In alternative implementations, the indication includes a probability (e.g., a number between 1 and 0) that indicates the likelihood that a pixel belongs to an object. In such implementations, the closer the value is to 1, the more likely the pixel belongs to an object and vice versa.
In one or more embodiments, a machine learning model includes a computer representation that is tunable (e.g., trained) based on inputs to approximate unknown functions used for generating the corresponding outputs. In particular, in some embodiments, a machine learning model includes a computer-implemented model that utilizes algorithms to learn from, and make predictions on, known data by analyzing the known data to learn to generate outputs that reflect patterns and attributes of the known data. For instance, in some instances, a machine learning model includes, but is not limited to a neural network (e.g., a convolutional neural network, recurrent neural network or other deep learning network), a decision tree (e.g., a gradient boosted decision tree), association rule learning, inductive logic programming, support vector learning, Bayesian network, regression-based model (e.g., censored regression), principal component analysis, or a combination thereof.
In one or more embodiments, a neural network includes a model of interconnected artificial neurons (e.g., organized in layers) that communicate and learn to approximate complex functions and generate outputs based on a plurality of inputs provided to the model. In some instances, a neural network includes one or more machine learning algorithms. Further, in some cases, a neural network includes an algorithm (or set of algorithms) that implements deep learning techniques that utilize a set of algorithms to model high-level abstractions in data. To illustrate, in some embodiments, a neural network includes a convolutional neural network, a recurrent neural network (e.g., a long short-term memory neural network), a generative adversarial neural network, a graph neural network, or a multi-layer perceptron. In some embodiments, a neural network includes a combination of neural networks or neural network components.
In one or more embodiments, a segmentation neural network includes a computer-implemented neural network that generates object masks for objects portrayed in digital images. In particular, in some embodiments, a segmentation neural network includes a computer-implemented neural network that detects objects within digital images and generates object masks for the objects. Indeed, in some implementations, a segmentation neural network includes a neural network pipeline that analyzes a digital image, identifies one or more objects portrayed in the digital image, and generates an object mask for the one or more objects. In some cases, however, a segmentation neural network focuses on a subset of tasks for generating an object mask.
As mentioned,
Although
Similarly, in one or more implementations, the scene-based image editing system 106 utilizes, as the segmentation neural network (or as an alternative to a segmentation neural network), one of the machine learning models or neural networks described in Ning Xu et al., “Deep GrabCut for Object Selection,” published Jul. 14, 2017; or U.S. Patent Application Publication No. 2019/0130229, entitled “Deep Salient Content Neural Networks for Efficient Digital Object Segmentation,” filed on Oct. 31, 2017; or U.S. patent application Ser. No. 16/035,410, entitled “Automatic Trimap Generation and Image Segmentation,” filed on Jul. 13, 2018; or U.S. Pat. No. 10,192,129, entitled “Utilizing Interactive Deep Learning To Select Objects In Digital Visual Media,” filed Nov. 18, 2015, each of which are incorporated herein by reference in their entirety.
In one or more implementations the segmentation neural network is a panoptic segmentation neural network. In other words, the segmentation neural network creates object mask for individual instances of a given object type. Furthermore, the segmentation neural network, in one or more implementations, generates object masks for semantic regions (e.g., water, sky, sand, dirt, etc.) in addition to countable things. Indeed, in one or more implementations, the scene-based image editing system 106 utilizes, as the segmentation neural network (or as an alternative to a segmentation neural network), one of the machine learning models or neural networks described in U.S. patent application Ser. No. 17/495,618, entitled “PANOPTIC SEGMENTATION REFINEMENT NETWORK,” filed on Oct. 2, 2021; or U.S. patent application Ser. No. 17/454,740, entitled “MULTI-SOURCE PANOPTIC FEATURE PYRAMID NETWORK,” filed on Nov. 12, 2021, each of which are incorporated herein by reference in their entirety.
Returning now to
As just mentioned, the detection-masking neural network 300 utilizes both the object detection machine learning model 308 and the object segmentation machine learning model 310. In one or more implementations, the object detection machine learning model 308 includes both the encoder 302 and the detection heads 304 shown in
As just mentioned, in one or more embodiments, the scene-based image editing system 106 utilizes the object detection machine learning model 308 to detect and identify objects within the digital image 316 (e.g., a target or a source digital image).
As shown in
In particular, the encoder 302, in one or more implementations, comprises convolutional layers that generate a feature vector in the form of a feature map. To detect objects within the digital image 316, the object detection machine learning model 308 processes the feature map utilizing a convolutional layer in the form of a small network that is slid across small windows of the feature map. The object detection machine learning model 308 further maps each sliding window to a lower-dimensional feature. In one or more embodiments, the object detection machine learning model 308 processes this feature using two separate detection heads that are fully connected layers. In some embodiments, the first head comprises a box-regression layer that generates the detected object and an object-classification layer that generates the object label.
As shown by
As mentioned, the object detection machine learning model 308 detects the objects within the digital image 316. In some embodiments, and as illustrated in
As illustrated in
Upon detecting the objects in the digital image 316, the detection-masking neural network 300 generates object masks for the detected objects. Generally, instead of utilizing coarse bounding boxes during object localization, the detection-masking neural network 300 generates segmentations masks that better define the boundaries of the object. The following paragraphs provide additional detail with respect to generating object masks for detected objects in accordance with one or more embodiments. In particular,
As illustrated in
In one or more implementations, prior to generating an object mask of a detected object, scene-based image editing system 106 receives user input 312 to determine objects for which to generate object masks. For example, the scene-based image editing system 106 receives input from a user indicating a selection of one of the detected objects. To illustrate, in the implementation shown, the scene-based image editing system 106 receives user input 312 of the user selecting bounding boxes 321 and 323. In alternative implementations, the scene-based image editing system 106 generates objects masks for each object automatically (e.g., without a user request indicating an object to select).
As mentioned, the scene-based image editing system 106 processes the bounding boxes of the detected objects in the digital image 316 utilizing the object segmentation machine learning model 310. In some embodiments, the bounding box comprises the output from the object detection machine learning model 308. For example, as illustrated in
In some embodiments, the scene-based image editing system 106 utilizes the object segmentation machine learning model 310 to generate the object masks for the aforementioned detected objects within the bounding boxes. For example, the object segmentation machine learning model 310 corresponds to one or more deep neural networks or models that select an object based on bounding box parameters corresponding to the object within the digital image 316. In particular, the object segmentation machine learning model 310 generates the object mask 324 and the object mask 326 for the detected man and bird, respectively.
In some embodiments, the scene-based image editing system 106 selects the object segmentation machine learning model 310 based on the object labels of the object identified by the object detection machine learning model 308. Generally, based on identifying one or more classes of objects associated with the input bounding boxes, the scene-based image editing system 106 selects an object segmentation machine learning model tuned to generate object masks for objects of the identified one or more classes. To illustrate, in some embodiments, based on determining that the class of one or more of the identified objects comprises a human or person, the scene-based image editing system 106 utilizes a special human object mask neural network to generate an object mask, such as the object mask 324 shown in
As further illustrated in
In some embodiments, the scene-based image editing system 106 also detects the objects shown in the digital image 316 via the collective network, i.e., the detection-masking neural network 300, in the same manner outlined above. For example, in some cases, the scene-based image editing system 106, via the detection-masking neural network 300 detects the woman, the man, and the bird within the digital image 316. In particular, the scene-based image editing system 106, via the detection heads 304, utilizes the feature pyramids and feature maps to identify objects within the digital image 316 and generates object masks via the masking head 306.
Furthermore, in one or more implementations, although
In one or more embodiments, the scene-based image editing system 106 implements object-aware image editing by generating a content fill for each object portrayed in a digital image (e.g., for each object mask corresponding to portrayed objects) utilizing a hole-filing model. In particular, in some cases, the scene-based image editing system 106 utilizes a machine learning model, such as a content-aware hole-filling machine learning model to generate the content fill(s) for each foreground object.
In one or more embodiments, a content fill includes a set of pixels generated to replace another set of pixels of a digital image. Indeed, in some embodiments, a content fill includes a set of replacement pixels for replacing another set of pixels. For instance, in some embodiments, a content fill includes a set of pixels generated to fill a hole (e.g., a content void) that remains after (or if) a set of pixels (e.g., a set of pixels portraying an object) has been removed from or moved within a digital image. In some cases, a content fill corresponds to a background of a digital image. To illustrate, in some implementations, a content fill includes a set of pixels generated to blend in with a portion of a background proximate to an object that could be moved/removed. In some cases, a content fill includes an inpainting segment, such as an inpainting segment generated from other pixels (e.g., other background pixels) within the digital image. In some cases, a content fill includes other content (e.g., arbitrarily selected content or content selected by a user) to fill in a hole or replace another set of pixels.
In one or more embodiments, a content-aware hole-filling machine learning model includes a computer-implemented machine learning model that generates content fill. In particular, in some embodiments, a content-aware hole-filling machine learning model includes a computer-implemented machine learning model that generates content fills for replacement regions in a digital image. For instance, in some cases, the scene-based image editing system 106 determines that an object has been moved within or removed from a digital image and utilizes a content-aware hole-filling machine learning model to generate a content fill for the hole that has been exposed as a result of the move/removal in response. As will be discussed in more detail, however, in some implementations, the scene-based image editing system 106 anticipates movement or removal of an object and utilizes a content-aware hole-filling machine learning model to pre-generate a content fill for that object. In some cases, a content-aware hole-filling machine learning model includes a neural network, such as an inpainting neural network (e.g., a neural network that generates a content fill—more specifically, an inpainting segment—using other pixels of the digital image). In other words, the scene-based image editing system 106 utilizes a content-aware hole-filling machine learning model in various implementations to provide content at a location of a digital image that does not initially portray such content (e.g., due to the location being occupied by another semantic area, such as an object).
Indeed, in one or more embodiments, the replacement region 404 includes an area corresponding to an object (and a hole that would be present if the object were moved or deleted). In some embodiments, the scene-based image editing system 106 identifies the replacement region 404 based on user selection of pixels (e.g., pixels portraying an object) to move, remove, cover, or replace from a digital image. To illustrate, in some cases, a client device selects an object portrayed in a digital image. Accordingly, the scene-based image editing system 106 deletes or removes the object and generates replacement pixels. In some case, the scene-based image editing system 106 identifies the replacement region 404 by generating an object mask via a segmentation neural network. For instance, the scene-based image editing system 106 utilizes a segmentation neural network (e.g., the detection-masking neural network 300 discussed above with reference to
As shown, the scene-based image editing system 106 utilizes the cascaded modulation inpainting neural network 420 to generate replacement pixels for the replacement region 404. In one or more embodiments, the cascaded modulation inpainting neural network 420 includes a generative adversarial neural network for generating replacement pixels. In some embodiments, a generative adversarial neural network (or “GAN”) includes a neural network that is tuned or trained via an adversarial process to generate an output digital image (e.g., from an input digital image). In some cases, a generative adversarial neural network includes multiple constituent neural networks such as an encoder neural network and one or more decoder/generator neural networks. For example, an encoder neural network extracts latent code from a noise vector or from a digital image. A generator neural network (or a combination of generator neural networks) generates a modified digital image by combining extracted latent code (e.g., from the encoder neural network). During training, a discriminator neural network, in competition with the generator neural network, analyzes a generated digital image to generate an authenticity prediction by determining whether the generated digital image is real (e.g., from a set of stored digital images) or fake (e.g., not from the set of stored digital images). The discriminator neural network also causes the scene-based image editing system 106 to modify parameters of the encoder neural network and/or the one or more generator neural networks to eventually generate digital images that fool the discriminator neural network into indicating that a generated digital image is a real digital image.
Along these lines, a generative adversarial neural network refers to a neural network having a specific architecture or a specific purpose such as a generative inpainting neural network. For example, a generative inpainting neural network includes a generative adversarial neural network that inpaints or fills pixels of a digital image with a content fill (or generates a content fill in anticipation of inpainting or filling in pixels of the digital image). In some cases, a generative inpainting neural network inpaints a digital image by filling hole regions (indicated by object masks). Indeed, as mentioned above, in some embodiments an object mask defines a replacement region using a segmentation or a mask indicating, overlaying, covering, or outlining pixels to be removed or replaced within a digital image.
Accordingly, in some embodiments, the cascaded modulation inpainting neural network 420 includes a generative inpainting neural network that utilizes a decoder having one or more cascaded modulation decoder layers. Indeed, as illustrated in
As shown, the scene-based image editing system 106 utilizes the cascaded modulation inpainting neural network 420 (and the cascaded modulation decoder layers 410, 412, 414, 416) to generate the inpainted digital image 408. Specifically, the cascaded modulation inpainting neural network 420 generates the inpainted digital image 408 by generating a content fill for the replacement region 404. As illustrated, the replacement region 404 is now filled with a content fill having replacement pixels that portray a photorealistic scene in place of the replacement region 404.
As mentioned above, the scene-based image editing system 106 utilizes a cascaded modulation inpainting neural network that includes cascaded modulation decoder layers to generate inpainted digital images.
As illustrated, the cascaded modulation inpainting neural network 502 includes an encoder 504 and a decoder 506. In particular, the encoder 504 includes a plurality of convolutional layers 508a-508n at different scales/resolutions. In some cases, the scene-based image editing system 106 feeds the digital image input 510 (e.g., an encoding of the digital image) into the first convolutional layer 508a to generate an encoded feature vector at a higher scale (e.g., lower resolution). The second convolutional layer 508b processes the encoded feature vector at the higher scale (lower resolution) and generates an additional encoded feature vector (at yet another higher scale/lower resolution). The cascaded modulation inpainting neural network 502 iteratively generates these encoded feature vectors until reaching the final/highest scale convolutional layer 508n and generating a final encoded feature vector representation of the digital image.
As illustrated, in one or more embodiments, the cascaded modulation inpainting neural network 502 generates a global feature code from the final encoded feature vector of the encoder 504. A global feature code includes a feature representation of the digital image from a global (e.g., high-level, high-scale, low-resolution) perspective. In particular, a global feature code includes a representation of the digital image that reflects an encoded feature vector at the highest scale/lowest resolution (or a different encoded feature vector that satisfies a threshold scale/resolution).
As illustrated, in one or more embodiments, the cascaded modulation inpainting neural network 502 applies a neural network layer (e.g., a fully connected layer) to the final encoded feature vector to generate a style code 512 (e.g., a style vector). In addition, the cascaded modulation inpainting neural network 502 generates the global feature code by combining the style code 512 with a random style code 514. In particular, the cascaded modulation inpainting neural network 502 generates the random style code 514 by utilizing a neural network layer (e.g., a multi-layer perceptron) to process an input noise vector. The neural network layer maps the input noise vector to a random style code 514. The cascaded modulation inpainting neural network 502 combines (e.g., concatenates, adds, or multiplies) the random style code 514 with the style code 512 to generate the global feature code 516. Although
As mentioned above, in some embodiments, the cascaded modulation inpainting neural network 502 generates an image encoding utilizing the encoder 504. An image encoding refers to an encoded representation of the digital image. Thus, in some cases, an image encoding includes one or more encoding feature vectors, a style code, and/or a global feature code.
In one or more embodiments, the cascaded modulation inpainting neural network 502 utilizes a plurality of Fourier convolutional encoder layer to generate an image encoding (e.g., the encoded feature vectors, the style code 512, and/or the global feature code 516). For example, a Fourier convolutional encoder layer (or a fast Fourier convolution) comprises a convolutional layer that includes non-local receptive fields and cross-scale fusion within a convolutional unit. In particular, a fast Fourier convolution can include three kinds of computations in a single operation unit: a local branch that conducts small-kernel convolution, a semi-global branch that processes spectrally stacked image patches, and a global branch that manipulates image-level spectrum. These three branches complementarily address different scales. In addition, in some instances, a fast Fourier convolution includes a multi-branch aggregation process for cross-scale fusion. For example, in one or more embodiments, the cascaded modulation inpainting neural network 502 utilizes a fast Fourier convolutional layer as described by Lu Chi, Borui Jiang, and Yadong Mu in Fast Fourier convolution, Advances in Neural Information Processing Systems, 33 (2020), which is incorporated by reference herein in its entirety.
Specifically, in one or more embodiments, the cascaded modulation inpainting neural network 502 utilizes Fourier convolutional encoder layers for each of the encoder convolutional layers 508a-508n. Thus, the cascaded modulation inpainting neural network 502 utilizes different Fourier convolutional encoder layers having different scales/resolutions to generate encoded feature vectors with improved, non-local receptive field.
Operation of the encoder 504 can also be described in terms of variables or equations to demonstrate functionality of the cascaded modulation inpainting neural network 502. For instance, as mentioned, the cascaded modulation inpainting neural network 502 is an encoder-decoder network with proposed cascaded modulation blocks at its decoding stage for image inpainting. Specifically, the cascaded modulation inpainting neural network 502 starts with an encoder E that takes the partial image and the mask as inputs to produce multi-scale feature maps from input resolution to resolution 4×4:
F
e
(1)
, . . . ,F
e
(L)
=E(x⊙(1−m),m),
where Fe(i) are the generated feature at scale 1≤i≤L (and L is the highest scale or resolution). The encoder is implemented by a set of stride-2 convolutions with residual connection.
After generating the highest scale feature Fe(L), a fully connected layer followed by a l2 normalization products a global style code s=fc(Fe(L))∥fc(Fe(L))∥2 to represent the input globally. In parallel to the encoder, an MLP-based mapping network produces a random style code w from a normalized random Gaussian noise z, simulating the stochasticity of the generation process. Moreover, the scene-based image editing system 106 joins w with s to produce the final global code g=[s; w] for decoding. As mentioned, in some embodiments, the scene-based image editing system 106 utilizes the final global code as an image encoding for the digital image.
As mentioned above, in some implementations, full convolutional models suffer from slow growth of effective receptive field, especially at the early stage of the network. Accordingly, utilizing strided convolution within the encoder can generate invalid features inside the hole region, making the feature correction at decoding stage more challenging. Fast Fourier convolution (FFC) can assist early layers to achieve receptive field that covers an entire image. Conventional systems, however, have only utilized FFC at a bottleneck layer, which is computationally demanding. Moreover, the shallow bottleneck layer cannot capture global semantic features effectively. Accordingly, in one or more implementations the scene-based image editing system 106 replaces the convolutional block in the encoder with FFC for the encoder layers. FFC enables the encoder to propagate features at early stage and thus address the issue of generating invalid features inside the hole, which helps improve the results.
As further shown in
Moreover, each of the cascaded modulation layers include multiple modulation blocks. For example, with regard to
As illustrated, the cascaded modulation layers 520a-520n are cascaded in that the global modulation block feeds into the spatial modulation block. Specifically, the cascaded modulation inpainting neural network 502 performs the spatial modulation at the spatial modulation block based on features generated at the global modulation block. To illustrate, in one or more embodiments the cascaded modulation inpainting neural network 502 utilizes the global modulation block to generate an intermediate feature. The cascaded modulation inpainting neural network 502 further utilizes a convolutional layer (e.g., a 2-layer convolutional affine parameter network) to convert the intermediate feature to a spatial tensor. The cascaded modulation inpainting neural network 502 utilizes the spatial tensor to modulate the input features analyzed by the spatial modulation block.
For example,
For example, a modulation block (or modulation operation) includes a computer-implemented process for modulating (e.g., scaling or shifting) an input signal according to one or more conditions. To illustrate, modulation block includes amplifying certain features while counteracting/normalizing these amplifications to preserve operation within a generative model. Thus, for example, a modulation block (or modulation operation) includes a modulation layer, a convolutional layer, and a normalization layer in some cases. The modulation layer scales each input feature of the convolution, and the normalization removes the effect of scaling from the statistics of the convolution's output feature maps.
Indeed, because a modulation layer modifies feature statistics, a modulation block (or modulation operation) often includes one or more approaches for addressing these statistical changes. For example, in some instances, a modulation block (or modulation operation) includes a computer-implemented process that utilizes batch normalization or instance normalization to normalize a feature. In some embodiments, the modulation is achieved by scaling and shifting the normalized activation according to affine parameters predicted from input conditions. Similarly, some modulation procedures replace feature normalization with a demodulation process. Thus, in one or more embodiments, a modulation block (or modulation operation) includes a modulation layer, convolutional layer, and a demodulation layer. For example, in one or more embodiments, a modulation block (or modulation operation) includes the modulation approaches described by Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila in Analyzing and improving the image quality of StyleGAN, Proc. CVPR (2020) (hereinafter StyleGan2), which is incorporated by reference herein in its entirety. In some instances, a modulation block includes one or more modulation operations.
Moreover, in one or more embodiments, a global modulation block (or global modulation operation) includes a modulation block (or modulation operation) that modulates an input signal in a spatially-invariant manner. For example, in some embodiments, a global modulation block (or global modulation operation) performs a modulation according to global features of a digital image (e.g., that do not vary spatially across coordinates of a feature map or image). Thus, for example, a global modulation block includes a modulation block that modulates an input signal according to an image encoding (e.g., global feature code) generated by an encoder. In some implementations, a global modulation block includes multiple global modulation operations.
In one or more embodiments, a spatial modulation block (or spatial modulation operation) includes a modulation block (or modulation operation) that modulates an input signal in a spatially-varying manner (e.g., according to a spatially-varying feature map). In particular, in some embodiments, a spatial modulation block (or spatial modulation operation) utilizes a spatial tensor, to modulate an input signal in a spatially-varying manner. Thus, in one or more embodiments a global modulation block applies a global modulation where affine parameters are uniform across spatial coordinates, and a spatial modulation block applies a spatially-varying affine transformation that varies across spatial coordinates. In some embodiments, a spatial modulation block includes both a spatial modulation operation in combination with another modulation operation (e.g., a global modulation operation and a spatial modulation operation).
For instance, in some embodiments, a spatial modulation operation includes spatially-adaptive modulation as described by Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu in Semantic image synthesis with spatially-adaptive normalization, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019), which is incorporated by reference herein in its entirety (hereinafter Taesung). In some embodiments, the spatial modulation operation utilizes a spatial modulation operation with a different architecture than Taesung, including a modulation-convolution-demodulation pipeline.
Thus, with regard to
As shown, the first global modulation operation 604 includes a modulation layer 604a, an upsampling layer 604b, a convolutional layer 604c, and a normalization layer 604d. In particular, the scene-based image editing system 106 utilizes the modulation layer 604a to perform a global modulation of the global feature map 612 based on a global feature code 614 (e.g., the global feature code 516). Specifically, the scene-based image editing system 106 applies a neural network layer (i.e., a fully connected layer) to the global feature code 614 to generate a global feature vector 616. The scene-based image editing system 106 then modulates the global feature map 612 utilizing the global feature vector 616.
In addition, the scene-based image editing system 106 applies the upsampling layer 604b (e.g., to modify the resolution scale). Further, the scene-based image editing system 106 applies the convolutional layer 604c. In addition, the scene-based image editing system 106 applies the normalization layer 604d to complete the first global modulation operation 604. As shown, the first global modulation operation 604 generates a global intermediate feature 618. In particular, in one or more embodiments, the scene-based image editing system 106 generates the global intermediate feature 618 by combining (e.g., concatenating) the output of the first global modulation operation 604 with an encoded feature vector 620 (e.g., from a convolutional layer of the encoder having a matching scale/resolution).
As illustrated, the scene-based image editing system 106 also utilizes a second global modulation operation 606. In particular, the scene-based image editing system 106 applies the second global modulation operation 606 to the global intermediate feature 618 to generate a new global feature map 622. Specifically, the scene-based image editing system 106 applies a global modulation layer 606a to the global intermediate feature 618 (e.g., conditioned on the global feature vector 616). Moreover, the scene-based image editing system 106 applies a convolutional layer 606b and a normalization layer 606c to generate the new global feature map 622. As shown, in some embodiments, the scene-based image editing system 106 applies a spatial bias in generating the new global feature map 622.
Furthermore, as shown in
As shown, the scene-based image editing system 106 utilizes the global modulation operation 608 to generate a local intermediate feature 626 from the local feature map 624. Specifically, the scene-based image editing system 106 applies a modulation layer 608a, an upsampling layer 608b, a convolutional layer 608c, and a normalization layer 608d. Moreover, in some embodiments, the scene-based image editing system 106 applies spatial bias and broadcast noise to the output of the global modulation operation 608 to generate the local intermediate feature 626.
As illustrated in
As shown, the scene-based image editing system 106 also applies a convolutional layer 610b to the modulated tensor. In particular, the convolutional layer 610b generates a convolved feature representation from the modulated tensor. In addition, the scene-based image editing system 106 applies a normalization layer 610c to convolved feature representation to generate the new local feature map 628.
Although illustrated as a normalization layer 610c, in one or more embodiments, the scene-based image editing system 106 applies a demodulation layer. For example, the scene-based image editing system 106 applies a modulation-convolution-demodulation pipeline (e.g., general normalization rather than instance normalization). In some cases, this approach avoids potential artifacts (e.g., water droplet artifacts) caused by instance normalization. Indeed, a demodulation/normalization layer includes a layer that scales each output feature map by a uniform demodulation/normalization value (e.g., by a uniform standard deviation instead of instance normalization that utilizes data-dependent constant normalization based on the contents of the feature maps).
As shown in
In one or more embodiments, upon generating the new global feature map 622 and the new local feature map 628, the scene-based image editing system 106 proceeds to the next cascaded modulation layer in the decoder. For example, the scene-based image editing system 106 utilizes the new global feature map 622 and the new local feature map 628 as input features to an additional cascaded modulation layer at a different scale/resolution. The scene-based image editing system 106 further utilizes the additional cascaded modulation layer to generate additional feature maps (e.g., utilizing an additional global modulation block and an additional spatial modulation block). In some cases, the scene-based image editing system 106 iteratively processes feature maps utilizing cascaded modulation layers until coming to a final scale/resolution to generate an inpainted digital image.
Although
As mentioned, the decoder can also be described in terms of variables and equations to illustrate operation of the cascaded modulation inpainting neural network. For example, as discussed, the decoder stacks a sequence of cascaded modulation blocks to upsample the input feature map Fe(L). Each cascaded modulation block takes the global code g as input to modulate the feature according to the global representation of the partial image. Moreover, in some cases, the scene-based image editing system 106 provides mechanisms to correct local error after predicting the global structure.
In particular, in some embodiments, the scene-based image editing system 106 utilizes a cascaded modulation block to address the challenge of generating coherent features both globally and locally. At a high level, the scene-based image editing system 106 follows the following approach: i) decomposition of global and local features to separate local details from the global structure, ii) a cascade of global and spatial modulation that predicts local details from global structures. In one or more implementations, the scene-based image editing system 106 utilizes spatial modulations generated from the global code for better predictions (e.g., and discards instance normalization to make the design compatible with StyleGAN2).
Specifically, the cascaded modulation takes the global and local feature Fg(i) and Fl(i) from previous scale and the global code g as input and produces the new global and local features Fg(i+1) and Fl(i+1) at next scale/resolution. To produce the new global code Fg(i+1) from Fg(i), the scene-based image editing system 106 utilizes a global code modulation stage that includes a modulation-convolution-demodulation procedure, which generates an upsampled feature X.
Due to the limited expressive power of the global vector g on representing 2-d visual details, and the inconsistent features inside and outside the hole, the global modulation may generate distorted features inconsistent with the context. To compensate, in some cases, the scene-based image editing system 106 utilizes a spatial modulation that generates more accurate features. Specifically, the spatial modulation takes X as the spatial code and g as the global code to modulate the input local feature Fl(i) in a spatially adaptive fashion.
Moreover, the scene-based image editing system 106 utilizes a unique spatial modulation-demodulation mechanism to avoid potential “water droplet” artifacts caused by instance normalization in conventional systems. As shown, the spatial modulation follows a modulation-convolution-demodulation pipeline.
In particular, for spatial modulation, the scene-based image editing system 106 generates a spatial tensor A0=APN(Y) from feature X by a 2-layer convolutional affine parameter network (APN). Meanwhile, the scene-based image editing system 106 generates a global vector α=fc(g) from global gode g with a fully connected layer (fc) to capture global context. The scene-based image editing system 106 generates a final spatial tensor A=A0+α as the broadcast summation of A0 and α for scaling intermediate feature Y of the block with element-wise product ⊙:
Moreover, for convolution, the modulated tensor
Ŷ=
For spatially-aware demodulation, the scene-based image editing system 106 applies a demodularization step to compute the normalized output {tilde over (Y)}. Specifically, the scene-based image editing system 106 assumes that the input features Y are independent random variables with unit variance and after the modulation, the expected variance of the output is not changed, i.e., y∈{tilde over (Y)}[Var(y)]=1. Accordingly, this gives the demodulation computation:
{tilde over (Y)}=Ŷ⊙D,
where D=1/√{square root over (K2⊙a∈A[a2])} is the demodulation coefficient. In some cases, the scene-based image editing system 106 implements the foregoing equation with standard tensor operations.
In one or more implementations, the scene-based image editing system 106 also adds spatial bias and broadcast noise. For example, the scene-based image editing system 106 adds the normalized feature {tilde over (Y)} to a shifting tensor B=APN(X) produced by another affine parameter network (APN) from feature X along with the broadcast noise n to product the new local feature Fl(i+1):
F
l
(i+1)
={tilde over (Y)}+B+n
Thus, in one or more embodiments, to generate a content fill having replacement pixels for a digital image having a replacement region, the scene-based image editing system 106 utilizes an encoder of a content-aware hole-filling machine learning model (e.g., a cascaded modulation inpainting neural network) to generate an encoded feature map from the digital image. The scene-based image editing system 106 further utilizes a decoder of the content-aware hole-filling machine learning model to generate the content fill for the replacement region. In particular, in some embodiments, the scene-based image editing system 106 utilizes a local feature map and a global feature map from one or more decoder layers of the content-aware hole-filling machine learning model in generating the content fill for the replacement region of the digital image.
As discussed above with reference to
In one or more embodiments, an object-aware modification includes an editing operation that targets an identified object in a digital image. In particular, in some embodiments, an object-aware modification includes an editing operation that targets an object that has been previously segmented. For instance, as discussed, the scene-based image editing system 106 generates a mask for an object portrayed in a digital image before receiving user input for modifying the object in some implementations. Accordingly, upon user selection of the object (e.g., a user selection of at least some of the pixels portraying the object), the scene-based image editing system 106 determines to target modifications to the entire object rather than requiring that the user specifically designate each pixel to be edited. Thus, in some cases, an object-aware modification includes a modification that targets an object by managing all the pixels portraying the object as part of a cohesive unit rather than individual elements. For instance, in some implementations an object-aware modification includes, but is not limited to, a move operation or a delete operation.
As shown in
In one or more embodiments, the scene-based image editing system 106 utilizes the segmentation neural network 702 and the content-aware hole-filling machine learning model 704 to analyze the digital image 706 in anticipation of receiving user input for modifications of the digital image 706. Indeed, in some instances, the scene-based image editing system 106 analyzes the digital image 706 before receiving user input for such modifications. For instance, in some embodiments, the scene-based image editing system 106 analyzes the digital image 706 automatically in response to receiving or otherwise accessing the digital image 706. In some implementations, the scene-based image editing system 106 analyzes the digital image in response to a general user input to initiate pre-processing in anticipation of subsequent modification.
As shown in
As further shown in
As shown in
Indeed,
In other implementations, the scene-based image editing system 106 utilizes the object masks 710 as indicators of replacement regions in the digital image 706. In particular, the scene-based image editing system 106 utilizes the object masks 710 as indicators of potential replacement regions that may result from receiving user input to modify the digital image 706 via moving/removing one or more of the objects 708a-708d. Accordingly, the scene-based image editing system 106 utilizes the content fills 712 to replace pixels indicated by the object masks 710.
Though
Further, in some implementations, the scene-based image editing system 106 generates multiple filled-in backgrounds (e.g., semi-completed backgrounds) for a digital image. For instance, in some cases, where a digital image portrays a plurality of objects, the scene-based image editing system 106 generates a filled-in background for each object from the plurality of objects. To illustrate, the scene-based image editing system 106 generates a filled-in background for an object by generating a content fill for that object while treating the other objects of the digital image as part of the background. Thus, in some instances, the content fill includes portions of other objects positioned behind the object within the digital image.
Thus, in one or more embodiments, the scene-based image editing system 106 generates a combined image 718 as indicated in
Further, though
In one or more embodiments, the scene-based image editing system 106 utilizes the combined image 718 (e.g., the digital image 706, the object masks 710, and the content fills 712) to facilitate various object-aware modifications with respect to the digital image 706. In particular, the scene-based image editing system 106 utilizes the combined image 718 to implement an efficient graphical user interface that facilitates flexible object-aware modifications.
Indeed, as shown in
It should be noted that the graphical user interface 802 of
As further shown in
As shown in
The scene-based image editing system 106 detects the user interaction for selecting the object 808d via various operations in various embodiments. For instance, in some cases, the scene-based image editing system 106 detects the selection via a single tap (or click) on the object 808d. In some implementations, the scene-based image editing system 106 detects the selection of the object 808d via a double tap (or double click) or a press and hold operation. Thus, in some instances, the scene-based image editing system 106 utilizes the second click or the hold operation to confirm the user selection of the object 808d.
In some cases, the scene-based image editing system 106 utilizes various interactions to differentiate between a single object select or a multi-object select. For instance, in some cases, the scene-based image editing system 106 determines that a single tap is for selecting a single object and a double tap is for selecting multiple objects. To illustrate, in some cases, upon receiving a first tap on an object, the scene-based image editing system 106 selects the object. Further, upon receiving a second tap on the object, the scene-based image editing system 106 selects one or more additional objects. For instance, in some implementations, the scene-based image editing system 106 selects one or more additional object having the same or a similar classification (e.g., selecting other people portrayed in an image when the first tap interacted with a person in the image). In one or more embodiments, the scene-based image editing system 106 recognizes the second tap as an interaction for selecting multiple objects if the second tap is received within a threshold time period after receiving the first tap.
In some embodiments, the scene-based image editing system 106 recognizes other user interactions for selecting multiple objects within a digital image. For instance, in some implementations, the scene-based image editing system 106 receives a dragging motion across the display of a digital image and selects all object captured within the range of the dragging motion. To illustrate, in some cases, the scene-based image editing system 106 draws a box that grows with the dragging motion and selects all objects that falls within the box. In some cases, the scene-based image editing system 106 draws a line that follows the path of the dragging motion and selects all objects intercepted by the line.
In some implementations, the scene-based image editing system 106 further allows for user interactions to select distinct portions of an object. To illustrate, in some cases, upon receiving a first tap on an object, the scene-based image editing system 106 selects the object. Further, upon receiving a second tap on the object, the scene-based image editing system 106 selects a particular portion of the object (e.g., a limb or torso of a person or a component of a vehicle). In some cases, the scene-based image editing system 106 selects the portion of the object touched by the second tap. In some cases, the scene-based image editing system 106 enters into a “sub object” mode upon receiving the second tap and utilizes additional user interactions for selecting particular portions of the object.
Returning to
In some cases, the scene-based image editing system 106 utilizes the visual indication 812 to indicate, via the graphical user interface 802, that the selection of the object 808d has been registered. In some implementations, the scene-based image editing system 106 utilizes the visual indication 812 to represent the pre-generated object mask that corresponds to the object 808d. Indeed, in one or more embodiments, in response to detecting the user interaction with the object 808d, the scene-based image editing system 106 surfaces the corresponding object mask. For instance, in some cases, the scene-based image editing system 106 surfaces the object mask in preparation for a modification to the object 808d and/or to indicate that the object mask has already been generated and is available for use. In one or more embodiments, rather than using the visual indication 812 to represent the surfacing of the object mask, the scene-based image editing system 106 displays the object mask itself via the graphical user interface 802.
Additionally, as the scene-based image editing system 106 generated the object mask for the object 808d prior to receiving the user input to select the object 808d, the scene-based image editing system 106 surfaces the visual indication 812 without latency or delay associated with conventional systems. In other words, the scene-based image editing system 106 surfaces the visual indication 812 without any delay associated with generating an object mask.
As further illustrated, based on detecting the user interaction for selecting the object 808d, the scene-based image editing system 106 provides an option menu 814 for display via the graphical user interface 802. The option menu 814 shown in
Thus, in one or more embodiments, the scene-based image editing system 106 provides modification options for display via the graphical user interface 802 based on the context of a user interaction. Indeed, as just discussed, the scene-based image editing system 106 provides an option menu that provides options for interacting with (e.g., modifying) a selected object. In doing so, the scene-based image editing system 106 minimizes the screen clutter that is typical under many conventional systems by withholding options or menus for display until it is determined that those options or menus would be useful in the current context in which the user is interacting with the digital image. Thus, the graphical user interface 802 used by the scene-based image editing system 106 allows for more flexible implementation on computing devices with relatively limited screen space, such as smart phones or tablet devices.
As shown in
As indicated in
Additionally, as the scene-based image editing system 106 generated the content fill 820 for the object 808d prior to receiving the user input to move the object 808d, the scene-based image editing system 106 exposes or surfaces the content fill 820 without latency or delay associated with conventional systems. In other words, the scene-based image editing system 106 exposes the content fill 820 incrementally as the object 808d is moved across the digital image 806 without any delay associated with generating content.
As further shown in
As further shown in
Additionally, as shown in
While
The scene-based image editing system 106 provides more flexibility for editing digital images when compared to conventional systems. In particular, the scene-based image editing system 106 facilitates object-aware modifications that enable interactions with objects rather than requiring targeting the underlying pixels. Indeed, based on a selection of some pixels that contribute to the portrayal of an object, the scene-based image editing system 106 flexibly determines that the whole object has been selected. This is in contrast to conventional systems that require a user to select an option from a menu indicating an intention to selection an object, provide a second user input indicating the object to select (e.g., a bounding box about the object or drawing of another rough boundary about the object), and another user input to generate the object mask. The scene-based image editing system 106 instead provides for selection of an object with a single user input (a tap on the object).
Further, upon user interactions for implementing a modification after the prior selection, the scene-based image editing system 106 applies the modification to the entire object rather than the particular set of pixels that were selected. Thus, the scene-based image editing system 106 manages objects within digital images as objects of a real scene that are interactive and can be handled as cohesive units. Further, as discussed, the scene-based image editing system 106 offers improved flexibility with respect to deployment on smaller devices by flexibly and dynamically managing the amount of content that is displayed on a graphical user interface in addition to a digital image.
Additionally, the scene-based image editing system 106 offers improved efficiency when compared to many conventional systems. Indeed, as previously discussed, conventional systems typically require execution of a workflow consisting of a sequence of user interactions to perform a modification. Where a modification is meant to target a particular object, many of these systems require several user interactions just to indicate that the object is the subject of the subsequent modification (e.g., user interactions for identifying the object and separating the object from the rest of the image) as well as user interactions for closing the loop on executed modifications (e.g., filling in the holes remaining after removing objects). The scene-based image editing system 106, however, reduces the user interactions typically required for a modification by pre-processing a digital image before receiving user input for such a modification. Indeed, by generating object masks and content fills automatically, the scene-based image editing system 106 eliminates the need for user interactions to perform these steps.
In one or more embodiments, the scene-based image editing system 106 performs further processing of a digital image in anticipation of modifying the digital image. For instance, as previously mentioned, the scene-based image editing system 106 generates a semantic scene graph from a digital image in some implementations. Thus, in some cases, upon receiving one or more user interactions for modifying the digital image, the scene-based image editing system 106 utilizes the semantic scene graph to execute the modifications. Indeed, in many instances, the scene-based image editing system 106 generates a semantic scene graph for use in modifying a digital image before receiving user input for such modifications.
Indeed, many conventional systems are inflexible in that they typically wait upon user interactions before determining characteristics of a digital image. For instance, such conventional systems often wait upon a user interaction that indicates a characteristic to be determined and then performs the corresponding analysis in response to receiving the user interaction. Accordingly, these systems fail to have useful characteristics readily available for use. For example, upon receiving a user interaction for modifying a digital image, conventional systems typically must perform an analysis of the digital image to determine characteristics to change after the user interaction has been received.
Further, as previously discussed, such operation results in inefficient operation as image edits often require workflows of user interactions, many of which are used in determining characteristics to be used in execution of the modification. Thus, conventional systems often require a significant number of user interactions to determine the characteristics needed for an edit.
The scene-based image editing system 106 provides advantages by generating a semantic scene graph for a digital image in anticipation of modifications to the digital image. Indeed, by generating the semantic scene graph, the scene-based image editing system 106 improves flexibility over conventional systems as it makes characteristics of a digital image readily available for use in the image editing process. Further, the scene-based image editing system 106 provides improved efficiency by reducing the user interactions required in determining these characteristics. In other words, the scene-based image editing system 106 eliminates the user interactions often required under conventional systems for the preparatory steps of editing a digital image. Thus, the scene-based image editing system 106 enables user interactions to focus on the image edits more directly themselves.
Additionally, by generating a semantic scene graph for a digital image, the scene-based image editing system 106 intelligently generates/obtains information the allows an image to be edited like a real-world scene. For example, the scene-based image editing system 106 generates a scene graph that indicates objects, object attributes, object relationships, etc. that allows the scene-based image editing system 106 to enable object/scene-based image editing.
In one or more embodiments, a semantic scene graph includes a graph representation of a digital image. In particular, in some embodiments, a semantic scene graph includes a graph that maps out characteristics of a digital image and their associated characteristic attributes. For instance, in some implementations, a semantic scene graph includes a node graph having nodes that represent characteristics of the digital image and values associated with the node representing characteristic attributes of those characteristics. Further, in some cases, the edges between the nodes represent the relationships between the characteristics.
As mentioned, in one or more implementations, the scene-based image editing system 106 utilizes one or more predetermined or pre-generated template graphs in generating a semantic scene graph for a digital image. For instance, in some cases, the scene-based image editing system 106 utilizes an image analysis graph in generating a semantic scene graph.
In one or more embodiments, an image analysis graph includes a template graph for structing a semantic scene graph. In particular, in some embodiments, an image analysis graph includes a template graph used by the scene-based image editing system 106 to organize the information included in a semantic scene graph. For instance, in some implementations, an image analysis graph includes a template graph that indicates how to organize the nodes of the semantic scene graph representing characteristics of a digital image. In some instances, an image analysis graph additionally or alternatively indicates the information to be represented within a semantic scene graph. For instance, in some cases, an image analysis graph indicates the characteristics, relationships, and characteristic attributes of a digital image to be represented within a semantic scene graph.
Indeed, as shown in
As further shown in
Additionally, as shown in
As further shown in
It should be noted that the characteristic categories and characteristic attributes represented in
In some embodiments, the scene-based image editing system 106 utilizes a real-world class description graph in generating a semantic scene graph for a digital image.
In one or more embodiments, a real-world class description graph includes a template graph that describes scene components (e.g., semantic areas) that may be portrayed in a digital image. In particular, in some embodiments, a real-world class description graph includes a template graph used by the scene-based image editing system 106 to provide contextual information to a semantic scene graph regarding scene components—such as objects—potentially portrayed in a digital image. For instance, in some implementations, a real-world class description graph provides a hierarchy of object classifications and/or an anatomy (e.g., object components) of certain objects that may be portrayed in a digital image. In some instances, a real-world class description graph further includes object attributes associated with the objects represented therein. For instance, in some cases, a real-world class description graph provides object attributes assigned to a given object, such as shape, color, material from which the object is made, weight of the object, weight the object can support, and/or various other attributes determined to be useful in subsequently modifying a digital image. Indeed, as will be discussed, in some cases, the scene-based image editing system 106 utilizes a semantic scene graph for a digital image to suggest certain edits or suggest avoiding certain edits to maintain consistency of the digital image with respect to the contextual information contained in the real-world class description graph from which the semantic scene graph was built.
As shown in
In one or more embodiments, each node cluster corresponds to a separate scene component (e.g., semantic area) class that may be portrayed in a digital image. Indeed, as shown in
As shown in
As an example, the node cluster 1108a includes a node 1104a representing a side table class and a node 1104b representing a table class. Further, as shown in
The degree to which a node cluster represents a hierarchy of class descriptions varies in various embodiments. In other words, the length/height of the represented hierarchy varies in various embodiments. For instance, in some implementations, the node cluster 1108a further includes a node representing a furniture class, indicating that a side table is classifiable as a piece of furniture. In some cases, the node cluster 1108a also includes a node representing an inanimate object lass, indicating that a side table is classifiable as such. Further, in some implementations, the node cluster 1108a includes a node representing an entity class, indicating that a side table is classifiable as an entity. Indeed, in some implementations, the hierarchies of class descriptions represented within the real-world class description graph 1102 include a class description/label—such as an entity class—at such a high level of generality that it is commonly applicable to all objects represented within the real-world class description graph 1102.
As further shown in
As shown in
Similarly, the node cluster 1108a includes object attributes 1110a-1110d associated with the node 1104a for the side table class and an additional object attributes 1112a-1112g associated with the node 1104b for the table class. Thus, the node cluster 1108a indicates that the object attributes 1110a-1110d are specific to the side table class while the additional object attributes 1112a-1112g are more generally associated with the table class (e.g., associated with all object classes that fall within the table class). In one or more embodiments, the object attributes 1110a-1110d and/or the additional object attributes 1112a-1112g are attributes that have been arbitrarily assigned to their respective object class (e.g., via user input or system defaults). For instance, in some cases, the scene-based image editing system 106 determines that all side tables can support one hundred pounds as suggested by
It should be noted that there is some overlap between object attributes included in a real-world class description graph and characteristic attributes included in an image analysis graph in some embodiments. Indeed, in many implementations, object attributes are characteristic attributes that are specific towards objects (rather than attributes for the setting or scene of a digital image). Further, it should be noted that the object attributes are merely exemplary and do not necessarily reflect the object attributes that are to be associated with an object class. Indeed, in some embodiments, the object attributes that are shown and their association with particular object classes are configurable to accommodate different needs in editing a digital image.
In some cases, a node cluster corresponds to one particular class of objects and presents a hierarchy of class descriptions and/or object components for that one particular class. For instance, in some implementations, the node cluster 1108a only corresponds to the side table class and presents a hierarchy of class descriptions and/or object components that are relevant to side tables. Thus, in some cases, upon identifying a side table within a digital image, the scene-based image editing system 106 refers to the node cluster 1108a for the side table class when generating a semantic scene graph but refers to a separate node cluster upon identifying another subclass of table within the digital image. In some cases, this separate node cluster includes several similarities (e.g., similar nodes and edges) with the node cluster 1108a as the other type of table would be included in a subclass of the table class and include one or more table legs.
In some implementations, however, a node cluster corresponds to a plurality of different but related object classes and presents a common hierarchy of class descriptions and/or object components for those object classes. For instance, in some embodiments, the node cluster 1108a includes an additional node representing a dining table class that is connected to the node 1104b representing the table class via an edge indicating that dining tables are also a subclass of tables. Indeed, in some cases, the node cluster 1108a includes nodes representing various subclasses of a table class. Thus, in some instances, upon identifying a table from a digital image, the scene-based image editing system 106 refers to the node cluster 1108a when generating a semantic scene graph for the digital image regardless of the subclass to which the table belongs.
As will be described, in some implementations, utilizing a common node cluster for multiple related subclasses facilitates object interactivity within a digital image. For instance, as noted,
In one or more embodiments, the scene-based image editing system 106 utilizes a behavioral policy graph in generating a semantic scene graph for a digital image.
In one or more embodiments, a behavioral policy graph includes a template graph that describes the behavior of an object portrayed in a digital image based on the context in which the object is portrayed. In particular, in some embodiments, a behavioral policy graph includes a template graph that assigns behaviors to objects portrayed in a digital image based on a semantic understanding of the objects and/or their relationships to other objects portrayed in the digital image. Indeed, in one or more embodiments, a behavioral policy includes various relationships among various types of objects and designates behaviors for those relationships. In some cases, the scene-based image editing system 106 includes a behavioral policy graph as part of a semantic scene graph. In some implementations, as will be discussed further below, a behavioral policy is separate from the semantic scene graph but provides plug-in behaviors based on the semantic understanding and relationships of objects represented in the semantic scene graph.
As shown in
As further shown, the behavioral policy graph 1202 further includes a plurality of classification indicators 1208a-1208e associated with the relationship indicators 1204a-1204e. In one or more embodiments, the classification indicators 1208a-1208e indicate an object class to which the assigned behavior applies. Indeed, in one or more embodiments, the classification indicators 1208a-1208e reference the object class of the corresponding relationship object. As shown by
The level of generality or specificity of a designated object class referenced by a classification indicator within its corresponding hierarchy of object classification varies in various embodiments. For instance, in some embodiments, a classification indicator references a lowest classification level (e.g., the most specific classification applicable) so that there are no subclasses, and the corresponding behavior applies only to those objects having that particular object lowest classification level. On the other hand, in some implementations, a classification indicator references a highest classification level (e.g., the most generic classification applicable) or some other level above the lowest classification level so that the corresponding behavior applies to objects associated with one or more of the multiple classification levels that exist within that designated classification level.
To provide an illustration of how the behavioral policy graph 1202 indicates assigned behavior, the relationship indicator 1204a indicates a “is supported by” relationship between an object (e.g., the relationship subject) and another object (e.g., the relationship object). The behavior indicator 1206a indicates a “moves with” behavior that is associated with the “is supported by” relationship, and the classification indicator 1208a indicates that this particular behavior applies to objects within some designated object class. Accordingly, in one or more embodiments, the behavioral policy graph 1202 shows that an object that falls within the designated object class and has a “is supported by” relationship with another object will exhibit the “moves with” behavior. In other words, if a first object of the designated object class is portrayed in a digital image being supported by a second object, and the digital image is modified to move that second object, then the scene-based image editing system 106 will automatically move the first object with the second object as part of the modification in accordance with the behavioral policy graph 1202. In some cases, rather than moving the first object automatically, the scene-based image editing system 106 provides a suggestion to move the first object for display within the graphical user interface in use to modify the digital image.
As shown by
As shown in
As further shown in
Additionally, as shown in
In one or more implementations, the scene-based image editing system 106 utilizes a depth estimation neural network to estimate a depth of an object in a digital image and stores the determined depth in the semantic scene graph 1412. For example, the scene-based image editing system 106 utilizes a depth estimation neural network as described in U.S. application Ser. No. 17/186,436, filed Feb. 26, 2021, titled “GENERATING DEPTH IMAGES UTILIZING A MACHINE-LEARNING MODEL BUILT FROM MIXED DIGITAL IMAGE SOURCES AND MULTIPLE LOSS FUNCTION SETS,” which is herein incorporated by reference in its entirety. Alternatively, the scene-based image editing system 106 utilizes a depth refinement neural network as described in U.S. application Ser. No. 17/658,873, filed Apr. 12, 2022, titled “UTILIZING MACHINE LEARNING MODELS TO GENERATE REFINED DEPTH MAPS WITH SEGMENTATION MASK GUIDANCE,” which is herein incorporated by reference in its entirety. The scene-based image editing system 106 then accesses the depth information (e.g., average depth for an object) for an object from the semantic scene graph 1412 when editing an object to perform a realistic scene edit. For example, when moving an object within an image, the scene-based image editing system 106 then accesses the depth information for objects in the digital image from the semantic scene graph 1412 to ensure that the object being moved is not placed in front an object with less depth.
In one or more implementations, the scene-based image editing system 106 utilizes a depth estimation neural network to estimate lighting parameters for an object or scene in a digital image and stores the determined lighting parameters in the semantic scene graph 1412. For example, the scene-based image editing system 106 utilizes a source-specific-lighting-estimation-neural network as described in U.S. application Ser. No. 16/558,975, filed Sep. 3, 2019, titled “DYNAMICALLY ESTIMATING LIGHT-SOURCE-SPECIFIC PARAMETERS FOR DIGITAL IMAGES USING A NEURAL NETWORK,” which is herein incorporated by reference in its entirety. The scene-based image editing system 106 then accesses the lighting parameters for an object or scene from the semantic scene graph 1412 when editing an object to perform a realistic scene edit. For example, when moving an object within an image or inserting a new object in a digital image, the scene-based image editing system 106 accesses the lighting parameters for from the semantic scene graph 1412 to ensure that the object being moved/placed within the digital image has realistic lighting.
In one or more implementations, the scene-based image editing system 106 utilizes a depth estimation neural network to estimate lighting parameters for an object or scene in a digital image and stores the determined lighting parameters in the semantic scene graph 1412. For example, the scene-based image editing system 106 utilizes a source-specific-lighting-estimation-neural network as described in U.S. application Ser. No. 16/558,975, filed Sep. 3, 2019, titled “DYNAMICALLY ESTIMATING LIGHT-SOURCE-SPECIFIC PARAMETERS FOR DIGITAL IMAGES USING A NEURAL NETWORK,” which is herein incorporated by reference in its entirety. The scene-based image editing system 106 then accesses the lighting parameters for an object or scene from the semantic scene graph 1412 when editing an object to perform a realistic scene edit. For example, when moving an object within an image or inserting a new object in a digital image, the scene-based image editing system 106 accesses the lighting parameters for from the semantic scene graph 1412 to ensure that the object being moved/placed within the digital image has realistic lighting.
As further shown in
As previously indicated, in one or more embodiments, the image analysis graph 1406, the real-world class description graph 1408, and/or the behavioral policy graph 1410 are predetermined or pre-generated. In other words, the scene-based image editing system 106 pre-generates, structures, or otherwise determines the content and organization of each graph before implementation. For instance, in some cases, the scene-based image editing system 106 generates the image analysis graph 1406, the real-world class description graph 1408, and/or the behavioral policy graph 1410 based on user input.
Further, in one or more embodiments, the image analysis graph 1406, the real-world class description graph 1408, and/or the behavioral policy graph 1410 are configurable. Indeed, the graphs can be re-configured, re-organized, and/or have data represented therein added or removed based on preferences or the needs of editing a digital image. For instance, in some cases, the behaviors assigned by the behavioral policy graph 1410 work in some image editing contexts but not others. Thus, when editing an image in another image editing context, the scene-based image editing system 106 implements the one or more neural networks 1404 and the image analysis graph 1406 but implements a different behavioral policy graph (e.g., one that was configured to satisfy preferences for that image editing context). Accordingly, in some embodiments, the scene-based image editing system 106 modifies the image analysis graph 1406, the real-world class description graph 1408, and/or the behavioral policy graph 1410 to accommodate different image editing contexts.
For example, in one or more implementations, the scene-based image editing system 106 determines a context for selecting a behavioral policy graph by identifying a type of user. In particular, the scene-based image editing system 106 generates a plurality of behavioral policy graphs for various types of users. For instance, the scene-based image editing system 106 generates a first behavioral policy graph for novice or new users. The first behavioral policy graph, in one or more implementations, includes a greater number of behavior policies than a second behavioral policy graph. In particular, for newer users, the scene-based image editing system 106 utilizes a first behavioral policy graph that provides greater automation of actions and provides less control to the user. On the other hand, the scene-based image editing system 106 utilizes a second behavioral policy graph for advanced users with less behavior policies than the first behavioral policy graph. In this manner, the scene-based image editing system 106 provides the advanced user with greater control over the relationship-based actions (automatic moving/deleting/editing) of objects based on relationships. In other words, by utilizing the second behavioral policy graph for advanced users, the scene-based image editing system 106 performs less automatic editing of related objects.
In one or more implementations the scene-based image editing system 106 determines a context for selecting a behavioral policy graph based on visual content of a digital image (e.g., types of objects portrayed in the digital image), the editing application being utilized, etc. Thus, the scene-based image editing system 106, in one or more implementations, selects/utilizes a behavioral policy graph based on image content, a type of user, an editing application being utilizes, or another context.
Moreover, in some embodiments, the scene-based image editing system 106 utilizes the graphs in analyzing a plurality of digital images. Indeed, in some cases, the image analysis graph 1406, the real-world class description graph 1408, and/or the behavioral policy graph 1410 do not specifically target a particular digital image. Thus, in many cases, these graphs are universal and re-used by the scene-based image editing system 106 for multiple instances of digital image analysis.
In some cases, the scene-based image editing system 106 further implements one or more mappings to map between the outputs of the one or more neural networks 1404 and the data scheme of the image analysis graph 1406, the real-world class description graph 1408, and/or the behavioral policy graph 1410. As one example, the scene-based image editing system 106 utilizes various segmentation neural networks to identify and classify objects in various embodiments. Thus, depending on the segmentation neural network used, the resulting classification of a given object can be different (e.g., different wording or a different level of abstraction). Thus, in some cases, the scene-based image editing system 106 utilizes a mapping that maps the particular outputs of the segmentation neural network to the object classes represented in the real-world class description graph 1408, allowing the real-world class description graph 1408 to be used in conjunction with multiple neural networks.
As shown in
In one or more embodiments, the scene-based image editing system 106 generates object proposals and subgraph proposals for the input image 1500 in response to the request. For instance, in some embodiments, the scene-based image editing system 106 utilizes an object proposal network 1520 to extract a set of object proposals for the input image 1500. To illustrate, in some cases, the scene-based image editing system 106 extracts a set of object proposals for humans detected within the input image 1500, objects that the human(s) are wearing, objects near the human(s), buildings, plants, animals, background objects or scenery (including the sky or objects in the sky), etc.
In one or more embodiments, the object proposal network 1520 comprises the detection-masking neural network 300 (specifically, the object detection machine learning model 308) discussed above with reference to
[o0, . . . ,0N-1]=fRPN(I)
where I is the input image, fRPN(⋅) represents the RPN network, and oi is the i-th object proposal.
In some implementations, in connection with determining the object proposals, the scene-based image editing system 106 also determines coordinates of each object proposal relative to the dimensions of the input image 1500. Specifically, in some instances, the locations of the object proposals are based on bounding boxes that contain the visible portion(s) of objects within a digital image. To illustrate, for oi, the coordinates of the corresponding bounding box are represented by ri=[xi, yi, wi, hi], with (xi, yi) being the coordinates of the top left corner and wi and hi being the width and the height of the bounding box, respectively. Thus, the scene-based image editing system 106 determines the relative location of each significant object or entity in the input image 1500 and stores the location data with the set of object proposals.
As mentioned, in some implementations, the scene-based image editing system 106 also determines subgraph proposals for the object proposals. In one or more embodiments, the subgraph proposals indicate relations involving specific object proposals in the input image 1500. As can be appreciated, any two different objects (oi, oj) in a digital image can correspond to two possible relationships in opposite directions. As an example, a first object can be “on top of” a second object, and the second object can be “underneath” the first object. Because each pair of objects has two possible relations, the total number of possible relations for N object proposals is N(N−1). Accordingly, more object proposals result in a larger scene graph than fewer object proposals, while increasing computational cost and deteriorating inference speed of object detection in systems that attempt to determine all the possible relations in both directions for every object proposal for an input image.
Subgraph proposals reduce the number of potential relations that the scene-based image editing system 106 analyze. Specifically, as mentioned previously, a subgraph proposal represents a relationship involving two or more specific object proposals. Accordingly, in some instances, the scene-based image editing system 106 determines the subgraph proposals for the input image 1500 to reduce the number of potential relations by clustering, rather than maintaining the N(N−1) number of possible relations. In one or more embodiments, the scene-based image editing system 106 uses the clustering and subgraph proposal generation process described in Y. Li, W. Ouyang, B. Zhou, Y. Cui, J. Shi, and X. Wang, Factorizable net: An efficient subgraph based framework for scene graph generation, ECCV, Jun. 29, 2018, the entire contents of which are hereby incorporated by reference.
As an example, for a pair of object proposals, the scene-based image editing system 106 determines a subgraph based on confidence scores associated with the object proposals. To illustrate, the scene-based image editing system 106 generates each object proposal with a confidence score indicating the confidence that the object proposal is the right match for the corresponding region of the input image. The scene-based image editing system 106 further determines the subgraph proposal for a pair of object proposals based on a combined confidence score that is the product of the confidence scores of the two object proposals. The scene-based image editing system 106 further constructs the subgraph proposal as the union box of the object proposals with the combined confidence score.
In some cases, the scene-based image editing system 106 also suppresses the subgraph proposals to represent a candidate relation as two objects and one subgraph. Specifically, in some embodiments, the scene-based image editing system 106 utilizes non-maximum-suppression to represent the candidate relations as oi, oj, ski
, where i≠j and ski is the k-th subgraph of all the subgraphs associated with oi, the subgraphs for oi including oj and potentially other object proposals. After suppressing the subgraph proposals, the scene-based image editing system 106 represents each object and subgraph as a feature vector, oi∈
D and a feature map ski∈
D×K
After determining object proposals and subgraph proposals for objects in the input image, the scene-based image editing system 106 retrieves and embeds relationships from an external knowledgebase 1522. In one or more embodiments, an external knowledgebase includes a dataset of semantic relationships involving objects. In particular, in some embodiments, an external knowledgebase includes a semantic network including descriptions of relationships between objects based on background knowledge and contextual knowledge (also referred to herein as “commonsense relationships”). In some implementations, an external knowledgebase includes a database on one or more servers that includes relationship knowledge from one or more sources including expert-created resources, crowdsourced resources, web-based sources, dictionaries, or other sources that include information about object relationships.
Additionally, in one or more embodiments an embedding includes a representation of relationships involving objects as a vector. For instance, in some cases, a relationship embedding includes a vector representation of a triplet (i.e., an object label, one or more relationships, and an object entity) using extracted relationships from an external knowledgebase.
Indeed, in one or more embodiments, the scene-based image editing system 106 communicates with the external knowledgebase 1522 to obtain useful object-relationship information for improving the object and subgraph proposals. Further, in one or more embodiments, the scene-based image editing system 106 refines the object proposals and subgraph proposals (represented by the box 1524) using embedded relationships, as described in more detail below.
In some embodiments, in preparation for retrieving the relationships from the external knowledgebase 1522, the scene-based image editing system 106 performs a process of inter-refinement on the object and subgraph proposals (e.g., in preparation for refining features of the object and subgraph proposals). Specifically, the scene-based image editing system 106 uses the knowledge that each object oi is connected to a set of subgraphs Si, and each subgraph sk is associated with a set of objects Ok to refine the object vector (resp. the subgraphs) by attending the associated subgraph feature maps (resp. the associated object vectors). For instance, in some cases, the inter-refinement process is represented as:
where αks→o (resp. αio→s) is the output of a softmax layer indicating the weight for passing ski (resp. oik) to oi (resp. to sk), and fs→o and fo→s are non-linear mapping functions. In one or more embodiments, due to different dimensions of oi and sk, the scene-based image editing system 106 applies pooling or spatial location-based attention for s→o or o→s refinement.
In some embodiments, once the inter-refinement is complete, the scene-based image editing system 106 predicts an object label from the initially refined object feature vector ōi and matches the object label with the corresponding semantic entities in the external knowledgebase 1522. In particular, the scene-based image editing system 106 accesses the external knowledgebase 1522 to obtain the most common relationships corresponding to the object label. The scene-based image editing system 106 further selects a predetermined number of the most common relationships from the external knowledgebase 1522 and uses the retrieved relationships to refine the features of the corresponding object proposal/feature vector.
In one or more embodiments, after refining the object proposals and subgraph proposals using the embedded relationships, the scene-based image editing system 106 predicts object labels 1502 and predicate labels from the refined proposals. Specifically, the scene-based image editing system 106 predicts the labels based on the refined object/subgraph features. For instance, in some cases, the scene-based image editing system 106 predicts each object label directly with the refined features of a corresponding feature vector. Additionally, the scene-based image editing system 106 predicts a predicate label (e.g., a relationship label) based on subject and object feature vectors in connection with their corresponding subgraph feature map due to subgraph features being associated with several object proposal pairs. In one or more embodiments, the inference process for predicting the labels is shown as:
P
i,j˜softmax(frel([õi⊗
V
i˜softmax(fnode(õi))
where frel(⋅) and fnode(⋅) denote the mapping layers for predicate and object recognition, respectively, and ⊗ represents a convolution operation. Furthermore, õi represents a refined feature vector based on the extracted relationships from the external knowledgebase.
In one or more embodiments, the scene-based image editing system 106 further generates a semantic scene graph 1504 using the predicted labels. In particular, the scene-based image editing system 106 uses the object labels 1502 and predicate labels from the refined features to create a graph representation of the semantic information of the input image 1500. In one or more embodiments, the scene-based image editing system 106 generates the scene graph as =
Vi, pi,j, Vj
, i≠j, where
is the scene graph.
Thus, the scene-based image editing system 106 utilizes relative location of the objects and their labels in connection with an external knowledgebase 1522 to determine relationships between objects. The scene-based image editing system 106 utilizes the determined relationships when generating a behavioral policy graph 1410. As an example, the scene-based image editing system 106 determines that a hand and a cell phone have an overlapping location within the digital image. Based on the relative locations and depth information, the scene-based image editing system 106 determines that a person (associated with the hand) has a relationship of “holding” the cell phone. As another example, the scene-based image editing system 106 determines that a person and a shirt have an overlapping location and overlapping depth within a digital image. Based on the relative locations and relative depth information, the scene-based image editing system 106 determines that the person has a relationship of “wearing” the shirt. On other hand, the scene-based image editing system 106 determines that a person and a shirt have an overlapping location and but the shirt has a greater average depth than an average depth of the person within a digital image. Based on the relative locations and relative depth information, the scene-based image editing system 106 determines that the person has a relationship of “in front of” with the shirt.
By generating a semantic scene graph for a digital image, the scene-based image editing system 106 provides improved flexibility and efficiency. Indeed, as mentioned above, the scene-based image editing system 106 generates a semantic scene graph to provide improved flexibility as characteristics used in modifying a digital image are readily available at the time user interactions are received to execute a modification. Accordingly, the scene-based image editing system 106 reduces the user interactions typically needed under conventional systems to determine those characteristics (or generate needed content, such as bounding boxes or object masks) in preparation for executing a modification. Thus, the scene-based image editing system 106 provides a more efficient graphical user interface that requires less user interactions to modify a digital image.
Additionally, by generating a semantic scene graph for a digital image, the scene-based image editing system 106 provides an ability to edit a two-dimensional image like a real-world scene. For example, based on a generated semantic scene graph for an image generated utilizing various neural networks, the scene-based image editing system 106 determines objects, their attributes (position, depth, material, color, weight, size, label, etc.). The scene-based image editing system 106 utilizes the information of the semantic scene graph to edit an image intelligently as if the image were a real-world scene.
Indeed, in one or more embodiments, the scene-based image editing system 106 utilizes a semantic scene graph generated for a digital image to facilitate modification to the digital image. For instance, in one or more embodiments, the scene-based image editing system 106 facilitates modification of one or more object attributes of an object portrayed in a digital image utilizing the corresponding semantic scene graph.
Many conventional systems are inflexible in that they often require difficult, tedious workflows to target modifications to a particular object attribute of an object portrayed in a digital image. Indeed, modifying an object attribute often requires manual manipulation of the object attribute under such systems. For example, modifying a shape of an object portrayed in a digital image often requires several user interactions to manually restructure the boundaries of an object (often at the pixel level), and modifying a size often requires tedious interactions with resizing tools to adjust the size and ensure proportionality. Thus, in addition to inflexibility, many conventional systems suffer from inefficiency in that the processes required by these systems to execute such a targeted modification typically involve a significant number of user interactions.
The scene-based image editing system 106 provides advantages over conventional systems by operating with improved flexibility and efficiency. Indeed, by presenting a graphical user interface element through which user interactions are able to target object attributes of an object, the scene-based image editing system 106 offers more flexibility in the interactivity of objects portrayed in digital images. In particular, via the graphical user interface element, the scene-based image editing system 106 provides flexible selection and modification of object attributes. Accordingly, the scene-based image editing system 106 further provides improved efficiency by reducing the user interactions required to modify an object attribute. Indeed, as will be discussed below, the scene-based image editing system 106 enables user interactions to interact with a description of an object attribute in order to modify that object attribute, avoiding the difficult, tedious workflows of user interactions required under many conventional systems.
As suggested, in one or more embodiments, the scene-based image editing system 106 facilitates modifying object attributes of objects portrayed in a digital image by determining the object attributes of those objects. In particular, in some cases, the scene-based image editing system 106 utilizes a machine learning model, such as an attribute classification neural network, to determine the object attributes.
In one or more embodiments, an attribute classification neural network includes a computer-implemented neural network that identifies object attributes of objects portrayed in a digital image. In particular, in some embodiments, an attribute classification neural network includes a computer-implemented neural network that analyzes objects portrayed in a digital image, identifies the object attributes of the objects, and provides labels for the corresponding object attributes in response. It should be understood that, in many cases, an attribute classification neural network more broadly identifies and classifies attributes for semantic areas portrayed in a digital image. Indeed, in some implementations, an attribute classification neural network determines attributes for semantic areas portrayed in a digital image aside from objects (e.g., the foreground or background).
As shown in
Furthermore, as shown in
Additionally, as illustrated in
Moreover, as shown
As further shown in
In some instances, the scene-based image editing system 106 utilizes probabilities (e.g., a probability score, floating point probability) output by the classifier neural network 1624 for the particular attributes to determine whether the attributes are classified as positive, negative, and/or unknown attribute labels for the object portrayed in the digital image 1602 (e.g., the chair). For example, the scene-based image editing system 106 identifies an attribute as a positive attribute when a probability output for the particular attribute satisfies a positive attribute threshold (e.g., a positive probability, a probability that is over 0.5). Moreover, the scene-based image editing system 106 identifies an attribute as a negative attribute when a probability output for the particular attribute satisfies a negative attribute threshold (e.g., a negative probability, a probability that is below −0.5). Furthermore, in some cases, the scene-based image editing system 106 identifies an attribute as an unknown attribute when the probability output for the particular attribute does not satisfy either the positive attribute threshold or the negative attribute threshold.
In some cases, a feature map includes a height, width, and dimension locations (H×W×D) which have D-dimensional feature vectors at each of the H×W image locations. Furthermore, in some embodiments, a feature vector includes a set of values representing characteristics and/or features of content (or an object) within a digital image. Indeed, in some embodiments, a feature vector includes a set of values corresponding to latent and/or patent attributes related to a digital image. For example, in some instances, a feature vector is a multi-dimensional dataset that represents features depicted within a digital image. In one or more embodiments, a feature vector includes a set of numeric metrics learned by a machine learning algorithm.
As shown in
In particular, in one or more embodiments, the scene-based image editing system 106 utilizes a convolutional neural network as an embedding neural network. For example, the scene-based image editing system 106 generates a D-dimensional image feature map fimg(I)∈H×W×D with a spatial size H×W extracted from a convolutional neural network-based embedding neural network. In some instance, the scene-based image editing system 106 utilizes an output of the penultimate layer of ResNet-50 as the image feature map fimg(I).
As shown in
By utilizing both low-level feature maps and high-level feature maps, the scene-based image editing system 106 accurately predicts attributes across the wide range of semantic levels. For instance, the scene-based image editing system 106 utilizes low-level feature maps to accurately predict attributes such as, but not limited to, colors (e.g., red, blue, multicolored), patterns (e.g., striped, dotted, striped), geometry (e.g., shape, size, posture), texture (e.g., rough, smooth, jagged), or material (e.g., wooden, metallic, glossy, matte) of a portrayed object. Meanwhile, in one or more embodiments, the scene-based image editing system 106 utilizes high-level feature maps to accurately predict attributes such as, but not limited to, object states (e.g., broken, dry, messy, full, old) or actions (e.g., running, sitting, flying) of a portrayed object.
Furthermore, as illustrated in
In one or more embodiments, the scene-based image editing system 106 generates the image-object feature map 1714 to provide an extra signal to the multi-attribute contrastive classification neural network to learn the relevant object for which it is predicting attributes (e.g., while also encoding the features for the object). In particular, in some embodiments, the scene-based image editing system 106 incorporates the object-label embedding vector 1712 (as an input in a feature composition module fcomp to generate the image-object feature map 1714) to improve the classification results of the multi-attribute contrastive classification neural network by having the multi-attribute contrastive classification neural network learn to avoid unfeasible object-attribute combinations (e.g., a parked dog, a talking table, a barking couch). Indeed, in some embodiments, the scene-based image editing system 106 also utilizes the object-label embedding vector 1712 (as an input in the feature composition module fcomp) to have the multi-attribute contrastive classification neural network learn to associate certain object-attribute pairs together (e.g., a ball is always round). In many instances, by guiding the multi-attribute contrastive classification neural network on what object it is predicting attributes for enables the multi-attribute contrastive classification neural network to focus on particular visual aspects of the object. This, in turn, improves the quality of extracted attributes for the portrayed object.
In one or more embodiments, the scene-based image editing system 106 utilizes a feature composition module (e.g., fcomp) to generate the image-object feature map 1714. In particular, the scene-based image editing system 106 implements the feature composition module (e.g., fcomp) with a gating mechanism in accordance with the following:
f
comp(fimg(I),Φo)=fimg(I)⊙fgate(Φo)
and
f
comp(Φo)=σ(Wg2·ReLU(Wg1Φo+bg1)+bg2)
In the first function above, the scene-based image editing system 106 utilizes a channel-wise product (⊙) of the high-level attribute feature map fimg(I) and a filter fgate of the object-label embedding vector Φo∈d to generate an image-object feature map fcomp(fimg(I), Φo)∈
D.
In addition, in the second function above, the scene-based image editing system 106 utilizes a sigmoid function σ in the fgate(Φo))∈D that is broadcasted to match the feature map spatial dimension as a 2-layer multilayer perceptron (MLP). Indeed, in one or more embodiments, the scene-based image editing system 106 utilizes fgate as a filter that selects attribute features that are relevant to the object of interest (e.g., as indicated by the object-label embedding vector Φo). In many instances, the scene-based image editing system 106 also utilizes fgate to suppress incompatible object-attribute pairs (e.g., talking table). In some embodiments, the scene-based image editing system 106 can identify object-image labels for each object portrayed within a digital image and output attributes for each portrayed object by utilizing the identified object-image labels with the multi-attribute contrastive classification neural network.
Furthermore, as shown in
In some instances, digital images may include multiple objects (and/or a background). Accordingly, in one or more embodiments, the scene-based image editing system 106 utilizes a localizer neural network to learn an improved feature aggregation that suppresses non-relevant-object regions (e.g., regions not reflected in a segmentation prediction of the target object to isolate the target object). For example, in reference to the digital image 1702, the scene-based image editing system 106 utilizes the localizer neural network 1716 to localize an object region such that the multi-attribute contrastive classification neural network predicts attributes for the correct object (e.g., the portrayed chair) rather than other irrelevant objects (e.g., the portrayed horse). To do this, in some embodiments, the scene-based image editing system 106 utilizes a localizer neural network that utilizes supervised learning with object segmentation masks (e.g., ground truth relevant-object masks) from a dataset of labeled images (e.g., ground truth images as described below).
To illustrate, in some instances, the scene-based image editing system 106 utilizes 2-stacked convolutional layers frel (e.g., with a kernel size of 1) followed by a spatial softmax to generate a localized object attention feature vector G (e.g., a localized object region) from an image-object feature map X∈H×W×D in accordance with the following:
For example, the localized object attention feature vector G includes a single plane of data that is H×W (e.g., a feature map having a single dimension). In some instances, the localized object attention feature vector G includes a feature map (e.g., a localized object attention feature map) that includes one or more feature vector dimensions.
Then, in one or more embodiments, the scene-based image editing system 106 utilizes the localized object attention feature vector Gh,w and the image-object feature map Xh,w to generate the localized image-object feature vector Zrel in accordance with the following:
In some instances, in the above function, the scene-based image editing system 106 pools H×W D-dimensional feature vectors Xh,w (from the image-object feature map) in D using weights from the localized object attention feature vector Gh,w into a single D-dimensional feature vector Zrel.
In one or more embodiments, in reference to
Furthermore, as shown in
In one or more embodiments, the scene-based image editing system 106 utilizes the multi-attention feature vector Zatt to accurately predict attributes of a portrayed object within a digital image by providing focus to different parts of the portrayed object and/or regions surrounding the portrayed object (e.g., attending to features at different spatial locations). To illustrate, in some instances, the scene-based image editing system 106 utilizes the multi-attention feature vector Zatt to extract attributes such as “barefooted” or “bald-headed” by focusing on different parts of a person (i.e., an object) that is portrayed in a digital image. Likewise, in some embodiments, the scene-based image editing system 106 utilizes the multi-attention feature vector Zatt to distinguish between different activity attributes (e.g., jumping vs crouching) that may rely on information from surrounding context of the portrayed object.
In certain instances, the scene-based image editing system 106 generates an attention map per attribute portrayed for an object within a digital image. For example, the scene-based image editing system 106 utilizes an image-object feature map with one or more attention layers to generate an attention map from the image-object feature map for each known attribute. Then, the scene-based image editing system 106 utilizes the attention maps with a projection layer to generate the multi-attention feature vector Zatt. In one or more embodiments, the scene-based image editing system 106 generates various numbers of attention maps for various attributes portrayed for an object within a digital image (e.g., the system can generate an attention map for each attribute or a different number of attention maps than the number of attributes).
Furthermore, in one or more embodiments, the scene-based image editing system 106 utilizes a hybrid shared multi-attention approach that allows for attention hops while generating the attention maps from the image-object feature map. For example, the scene-based image editing system 106 extracts M attention maps {A(m)}m=1M from an image-object feature map X utilizing a convolutional layer fatt(mm) (e.g., attention layers) in accordance with the following function:
In some cases, the scene-based image editing system 106 utilizes a convolutional layer fatt(m) that has a similar architecture to the 2-stacked convolutional layers frel from function (3) above. By utilizing the approach outlined in second function above, the scene-based image editing system 106 utilizes a diverse set of attention maps that correspond to a diverse range of attributes.
Subsequently, in one or more embodiments, the scene-based image editing system 106 utilizes the M attention maps (e.g., Ah,w(m)) to aggregate M attention feature vectors ({r(m)}m=1M) from the image-object feature map X in accordance with the following function:
Moreover, in reference to
Z
att
(m)
=f
proj
(m)(r(m)),zatt(m)∈D
Then, in one or more embodiments, the scene-based image editing system 106 generates the multi-attention feature vector Zatt by concatenating the individual attention feature vectors zatt(m) in accordance with the following function:
Z
att=concat([zatt(1), . . . ,zatt(M)])
In some embodiments, the scene-based image editing system 106 utilizes a divergence loss with the multi-attention neural network in the M attention hops approach. In particular, the scene-based image editing system 106 utilizes a divergence loss that encourages attention maps to focus on different (or unique) regions of a digital image (from the image-object feature map). In some cases, the scene-based image editing system 106 utilizes a divergence loss that promotes diversity between attention features by minimizing a cosine similarity (e.g., 2-norm) between attention weight vectors (e.g., E) of attention features. For instance, the scene-based image editing system 106 determines a divergence loss
div in accordance with the following function:
In one or more embodiments, the scene-based image editing system 106 utilizes the divergence loss div to learn parameters of the multi-attention neural network 1722 and/or the multi-attribute contrastive classification neural network (as a whole).
Furthermore, as shown in
By generating and utilizing the localized low-level attribute feature vector Zlow, in one or more embodiments, the scene-based image editing system 106 improves the accuracy of low-level features (e.g., colors, materials) that are extracted for an object portrayed in a digital image. In particular, in one or more embodiments, the scene-based image editing system 106 pools low-level features (as represented by a low-level attribute feature map from a low-level embedding layer) from a localized object attention feature vector (e.g., from a localizer neural network). Indeed, in one or more embodiments, by pooling low-level features from the localized object attention feature vector utilizing a low-level feature map, the scene-based image editing system 106 constructs a localized low-level attribute feature vector Zlow.
As further shown in
In one or more embodiments, the scene-based image editing system 106 utilizes a classifier neural network that is a 2-layer MLP. In some cases, the scene-based image editing system 106 utilizes a classifier neural network that includes various amounts of hidden units and output logic values followed by sigmoid. In some embodiments, the classifier neural network is trained by the scene-based image editing system 106 to generate both positive and negative attribute labels. Although one or more embodiments described herein utilize a 2-layer MLP, in some instances, the scene-based image editing system 106 utilizes a linear layer (e.g., within the classifier neural network, for the fgate, and for the image-object feature map).
Furthermore, in one or more embodiments, the scene-based image editing system 106 utilizes various combinations of the localized image-object feature vector Zrel, the multi-attention feature vector Zatt, and the localized low-level attribute feature vector Zlow with the classifier neural network to extract attributes for an object portrayed in a digital image. For example, in certain instances, the scene-based image editing system 106 provides the localized image-object feature vector Zrel and the multi-attention feature vector Zatt to extract attributes for the portrayed object. In some instances, as shown in
In one or more embodiments, the scene-based image editing system 106 utilizes the classifier neural network 1732 to generate prediction scores corresponding to attribute labels as outputs. For, example, the classifier neural network 1732 can generate a prediction score for one or more attribute labels (e.g., a score of 0.04 for blue, a score of 0.9 for red, a score of 0.4 for orange). Then, in some instances, the scene-based image editing system 106 utilizes attribute labels that correspond to prediction scores that satisfy a threshold prediction score. Indeed, in one or more embodiments, the scene-based image editing system 106 selects various attribute labels (both positive and negative) by utilizing output prediction scores for attributes from a classifier neural network.
Although one or more embodiments herein illustrate the scene-based image editing system 106 utilizing a particular embedding neural network, localizer neural network, multi-attention neural network, and classifier neural network, the scene-based image editing system 106 can utilize various types of neural networks for these components (e.g., CNN, FCN). In addition, although one or more embodiments herein describe the scene-based image editing system 106 combining various feature maps (and/or feature vectors) utilizing matrix multiplication, the scene-based image editing system 106, in some embodiments, utilizes various approaches to combine feature maps (and/or feature vectors) such as, but not limited to, concatenation, multiplication, addition, and/or aggregation. For example, in some implementations, the scene-based image editing system 106 combines a localized object attention feature vector and an image-object feature map to generate the localized image-object feature vector by concatenating the localized object attention feature vector and the image-object feature map.
Thus, in some cases, the scene-based image editing system 106 utilizes an attribute classification neural network (e.g., a multi-attribute contrastive classification neural network) to determine objects attributes of objects portrayed in a digital image or otherwise determined attributes of portrayed semantic areas. In some cases, the scene-based image editing system 106 adds object attributes or other attributes determined for a digital image to a semantic scene graph for the digital image. In other words, the scene-based image editing system 106 utilizes the attribute classification neural network in generating semantic scene graphs for digital images. In some implementations, however, the scene-based image editing system 106 stores the determined object attributes or other attributes in a separate storage location.
Further, in one or more embodiments, the scene-based image editing system 106 facilitates modifying object attributes of objects portrayed in a digital image by modifying one or more object attributes in response to user input. In particular, in some cases, the scene-based image editing system 106 utilizes a machine learning model, such as an attribute modification neural network to modify object attributes.
In one or more embodiments, an attribute modification neural network includes a computer-implemented neural network that modifies specified object attributes of an object (or specified attributes of other specified semantic areas). In particular, in some embodiments, an attribute modification neural network includes a computer-implemented neural network that receives user input targeting an object attribute and indicating a change to the object attribute and modifies the object attribute in accordance with the indicated change. In some cases, an attribute modification neural network includes a generative network.
As shown in
As illustrated by
In one or more embodiments, the object modification neural network 1806 performs text-guided visual feature manipulation to ground the modification input 1804a-1804b on the visual feature maps 1810 and manipulate the corresponding regions of the visual feature maps 1810 with the provided textual features. For instance, as shown in
As further shown in
Further, as shown, the object modification neural network 1806 utilizes a decoder 1826 to generate the modified object 1828. In particular, the decoder 1826 generates the modified object 1828 from the edge 1824 extracted from the object 1802 and the manipulated visual feature maps 1820 generated from the object 1802 and the modification input 1804a-1804b.
In one or more embodiments, the scene-based image editing system 106 trains the object modification neural network 1806 to handle open-vocabulary instructions and open-domain digital images. For instance, in some cases, the scene-based image editing system 106 trains the object modification neural network 1806 utilizing a large-scale image-caption dataset to learn a universal visual-semantic embedding space. In some cases, the scene-based image editing system 106 utilizes convolutional neural networks and/or long short-term memory networks as the encoders of the object modification neural network 1806 to transform digital images and text input into the visual and textual features.
The following provides a more detailed description of the text-guided visual feature manipulation. As previously mentioned, in one or more embodiments, the scene-based image editing system 106 utilizes the joint embedding space 1816 to manipulate the visual feature maps 1810 with the text instructions of the modification input 1804a-1804b via vector arithmetic operations. When manipulating certain objects or object attributes, the object modification neural network 1806 aims to modify only specific regions while keeping other regions unchanged. Accordingly, the object modification neural network 1806 conducts vector arithmetic operations between the visual feature maps 1810 represented as V∈1024×7×7 and the textual features 1814a-1814b (e.g., represented as textual feature vectors).
For instance, in some cases, the object modification neural network 1806 identifies the regions in the visual feature maps 1810 to manipulate (i.e., grounds the modification input 1804a-1804b) on the spatial feature map. In some cases, the object modification neural network 1806 provides a soft grounding for textual queries via a weighted summation of the visual feature maps 1810. In some cases, the object modification neural network 1806 uses the textual features 1814a-1814b (represented as t∈1024×1) as weights to compute the weighted summation of the visual feature maps 1810 g=t1V. Using this approach, the object modification neural network 1806 provides a soft grounding map g∈
7×7 which roughly localizes corresponding regions in the visual feature maps 1810 related to the text instructions.
In one or more embodiments, the object modification neural network 1806 utilizes the grounding map as location-adaptive coefficients to control the manipulation strength at different locations. In some cases, the object modification neural network 1806 utilizes a coefficient α to control the global manipulation strength, which enables continuous transitions between source images and the manipulated ones. In one or more embodiments, the scene-based image editing system 106 denotes the visual feature vector at spatial location (i, j) (where i, j∈{0, 1, . . . 6}) in the visual feature map V∈1024×7×7 as vi,j∈
1024.
The scene-based image editing system 106 utilizes the object modification neural network 1806 to perform various types of manipulations via the vector arithmetic operations weighted by the soft grounding map and the coefficient α. For instance, in some cases, the scene-based image editing system 106 utilizes the object modification neural network 1806 to change an object attribute or a global attribute. The object modification neural network 1806 denotes the textual feature embeddings of the source concept (e.g., “black triangle”) and the target concept (e.g., “white triangle”) as t1 and t2, respectively. The object modification neural network 1806 performs the manipulation of image feature vector v at location (i, j) as follows:
v
m
i,j
=v
i,j
−α
v
i,j
,t
1
t
1
+α
v
i,j
,t
1
t
2,
where i,j∈{0, 1, . . . 6} and vmi,j is the manipulated visual feature vector at location (i,j) of the 7×7 feature map.
In one or more embodiments, the object modification neural network 1806 removes the source features t1 and adds the target features t2 to each visual feature vector vi,j. Additionally, vi,j, t1
represents the value of the soft grounding map at location (i,j), calculated as the dot product of the image feature vector and the source textual features. In other words, the value represents the projection of the visual embedding vi,j onto the direction of the textual embedding t1. In some cases, object modification neural network 1806 utilizes the value as a location-adaptive manipulation strength to control which regions in the image should be edited. Further, the object modification neural network 1806 utilizes the coefficient α as a hyper-parameter that controls the image-level manipulation strength. By smoothly increasing α, the object modification neural network 1806 achieves smooth transitions from source to target attributes.
In some implementations, the scene-based image editing system 106 utilizes the object modification neural network 1806 to remove a concept (e.g., an object attribute, an object, or other visual elements) from a digital image (e.g., removing an accessory from a person). In some instances, the object modification neural network 1806 denotes the semantic embedding of the concept to be removed as t. Accordingly, the object modification neural network 1806 performs the removing operation as follows:
v
m
i,j
=v
i,j
−α
v
i,j
,t
t
Further, in some embodiments, the scene-based image editing system 106 utilizes the object modification neural network 1806 to modify the degree to which an object attribute (or other attribute of a semantic area) appears (e.g., making a red apple less red or increasing the brightness of a digital image). In some cases, the object modification neural network 1806 controls the strength of an attribute via the hyper-parameter α. By smoothly adjusting α, the object modification neural network 1806 gradually strengthens or weakens the degree to which an attribute appears as follows:
v
m
i,j
=v
i,j
±α
v
i,j
,t
t
After deriving the manipulated feature map Vm∈1024×7×7 the object modification neural network 1806 utilizes the decoder 1826 (an image decoder) to generate a manipulated image (e.g., the modified object 1828). In one or more embodiments, the scene-based image editing system 106 trains the object modification neural network 1806 as described by F. Faghri et al., Vse++: Improving visual-semantic Embeddings with Hard Negatives, arXiv:1707.05612, 2017, which is incorporated herein by reference in its entirety. In some cases, the decoder 1826 takes 1024×7×7 features maps as input and is composed of seven ResNet blocks with upsampling layers in between, which generates 256×256 images. Also, in some instances, the scene-based image editing system 106 utilizes a discriminator that includes a multi-scale patch-based discriminator. In some implementations, the scene-based image editing system 106 trains the decoder 1826 with GAN loss, perceptual loss, and discriminator feature matching loss. Further, in some embodiments, the fixed edge extractor 1822 includes a bi-directional cascade network.
Indeed, as shown in
As further shown in
In one or more embodiments, the scene-based image editing system 106 retrieves the object attributes for the object attribute indicators 1912a-1912c from a semantic scene graph generated for the digital image 1906. Indeed, in some implementations, the scene-based image editing system 106 generates a semantic scene graph for the digital image 1906 (e.g., before detecting the user interaction with the object 1908). In some cases, the scene-based image editing system 106 determines the object attributes for the object 1908 utilizing an attribute classification neural network and includes the determined object attributes within the semantic scene graph. In some implementations, the scene-based image editing system 106 retrieves the object attributes from a separate storage location.
As shown in
As shown in
In this case, the user interactions with the graphical user interface 1902 provide instructions to change a material of the object 1908 from a first material (e.g., wood) to a second material (e.g., metal). Thus, upon receiving the textual user input regarding the second material, the scene-based image editing system 106 modifies the digital image 1906 by modifying the object attribute of the object 1908 to reflect the user-provided second material.
In one or more embodiments, the scene-based image editing system 106 utilizes an attribute modification neural network to change the object attribute of the object 1908. In particular, as described above with reference to
As shown in
As shown in
In one or more embodiments, the scene-based image editing system 106 utilizes a textual representation of the alternative object attribute in modifying the object 2008. For instance, as discussed above, the scene-based image editing system 106 provides the textual representation as textual input to an attribute modification neural network and utilizes the attribute modification neural network to output a modified digital image in which the object 2008 reflects the targeted change in its object attribute.
As shown in
As shown in
In particular, in one or more embodiments, the scene-based image editing system 106 utilizes an attribute modification neural network to modify the digital image 2106 in accordance with the user interaction. Indeed, as described above with reference to
By facilitating image modifications that target particular object attributes as described above, the scene-based image editing system 106 provides improved flexibility and efficiency when compared to conventional systems. Indeed, the scene-based image editing system 106 provides a flexible, intuitive approach that visually displays descriptions of an object's attributes and allows user input that interacts with those descriptions to change the attributes. Thus, rather than requiring tedious, manual manipulation of an object attribute as is typical under many conventional systems, the scene-based image editing system 106 allows user interactions to target object attributes at a high level of abstraction (e.g., without having to interact at the pixel level). Further, as scene-based image editing system 106 enables modifications to object attributes via relatively few user interactions with provided visual elements, the scene-based image editing system 106 implements a graphical user interface that provides improved efficiency.
As previously mentioned, in one or more embodiments, the scene-based image editing system 106 further uses a semantic scene graph generated for a digital image to implement relationship-aware object modifications. In particular, the scene-based image editing system 106 utilizes the semantic scene graph to inform the modification behaviors of objects portrayed in a digital image based on their relationships with one or more other objects in the digital image.
Indeed, many conventional systems are inflexible in that they require different objects to be interacted with separately for modification. This is often the case even where the different objects are to be modified similarly (e.g., similarly resized or moved). For instance, conventional systems often require separate workflows to be executed via user interactions to modify separate objects or, at least, to perform the preparatory steps for the modification (e.g., outlining the objects and/or separating the objects from the rest of the image). Further, conventional systems typically fail to accommodate relationships between objects in a digital image when executing a modification. Indeed, these systems may modify a first object within a digital image but fail to execute a modification on a second object in accordance with a relationship between the two objects. Accordingly, the resulting modified image can appear unnatural or aesthetically confusing as it does not properly reflect the relationship between the two objects.
Accordingly, conventional systems are also often inefficient in that they require a significant number of user interactions to modify separate objects portrayed in a digital image. Indeed, as mentioned, conventional systems often require separate workflows to be performed via user interactions to execute many of the steps needed in modifying separate objects. Thus, many of the user interactions are redundant in that a user interaction is received, processed, and responded to multiple times for the separate objects. Further, when modifying an object having a relationship with another object, conventional systems require additional user interactions to modify the other object in accordance with that relationship. Thus, these systems unnecessarily duplicate the interactions used (e.g., interactions for moving an object then moving a related object) to perform separate modifications on related objects even where the relationship is suggestive as to the modification to be performed.
The scene-based image editing system 106 provides more flexibility and efficiency over conventional systems by implementing relationship-aware object modifications. Indeed, as will be discussed, the scene-based image editing system 106 provides a flexible, simplified process for selecting related objects for modification. Accordingly, the scene-based image editing system 106 flexibly allows user interactions to select and modify multiple objects portrayed in a digital image via a single workflow. Further, the scene-based image editing system 106 facilitates the intuitive modification of related objects so that the resulting modified image continues to reflect that relationship. As such, digital images modified by the scene-based image editing system 106 provide a more natural appearance when compared to conventional systems.
Further, by implementing a simplified process for selecting and modifying related objects, the scene-based image editing system 106 improves efficiency. In particular, the scene-based image editing system 106 implements a graphical user interface that reduces the user interactions required for selecting and modifying multiple, related objects. Indeed, as will be discussed, the scene-based image editing system 106 processes a relatively small number of user interactions with one object to anticipate, suggest, and/or execute modifications to other objects thus eliminating the need for additional user interactions for those modifications.
For instance,
In one or more embodiments, the scene-based image editing system 106 references the semantic scene graph previously generated for the digital image 2206 to identify the relationship between the objects 2208a-2208b. Indeed, as previously discussed, in some cases, the scene-based image editing system 106 includes relationships among the objects of a digital image in the semantic scene graph generated for the digital image. For instance, in one or more embodiments, the scene-based image editing system 106 utilizes a machine learning model, such as one of the models (e.g., the clustering and subgraph proposal generation model) discussed above with reference to
Indeed,
As further shown, the semantic scene graph component 2210 includes behavior indicators 2216a-2216b associated with the relationship indicator 2214b. The behavior indicators 2216a-2216b assign a behavior to the object 2208b based on its relationship with the object 2208a. For instance, the behavior indicator 2216a indicates that, because the object 2208b is held by the object 2208a, the object 2208b moves with the object 2208a. In other words, the behavior indicator 2216a instructs the scene-based image editing system 106 to move the object 2208b (or at least suggest that the object 2208b be moved) when moving the object 2208a. In one or more embodiments, the scene-based image editing system 106 includes the behavior indicators 2216a-2216b within the semantic scene graph based on the behavioral policy graph used in generating the semantic scene graph. Indeed, in some cases, the behaviors assigned to a “held by” relationship (or other relationships) vary based on the behavioral policy graph used. Thus, in one or more embodiments, the scene-based image editing system 106 refers to a previously generated semantic scene graph to identify relationships between objects and the behaviors assigned based on those relationships.
It should be noted that the semantic scene graph component 2210 indicates that the behaviors of the behavior indicators 2216a-2216b are assigned to the object 2208b but not the object 2208a. Indeed, in one or more, the scene-based image editing system 106 assigns behavior to an object based on its role in the relationship. For instance, while it may be appropriate to move a held object when the holding object is moved, the scene-based image editing system 106 determines that the holding object does not have to move when the held object is moved in some embodiments. Accordingly, in some implementations, the scene-based image editing system 106 assigns different behaviors to different objects in the same relationship.
As shown in
As illustrated by
Because the objects 2208a-2208b have a relationship, the scene-based image editing system 106 adds the object 2208b to the selection. As shown in
In one or more embodiments, the scene-based image editing system 106 surfaces object masks for the object 2208a and the object 2208b based on their inclusion within the selection. Indeed, the scene-based image editing system 106 surfaces pre-generated object masks for the objects 2208a-2208b in anticipation of a modification to the objects 2208a-2208b. In some cases, the scene-based image editing system 106 retrieves the pre-generated object masks from the semantic scene graph for the digital image 2206 or retrieves a storage location for the pre-generated object masks. In either case, the object masks are readily available at the time the objects 2208a-2208b are included in the selection and before modification input has been received.
As further shown in
Indeed, in one or more embodiments, in addition to determining the relationship between the objects 2208a-2208b, the scene-based image editing system 106 references the semantic scene graph for the digital image 2206 to determine the behaviors that have been assigned based on that relationship. In particular, the scene-based image editing system 106 references the behavior indicators associated with the relationship between the objects 2208a-2208b (e.g., the behavior indicators 2216a-2216b) to determine which behaviors are assigned to the objects 2208a-2208b based on their relationship. Thus, by determining the behaviors assigned to the object 2208b, the scene-based image editing system 106 determines how to respond to potential edits.
For instance, as shown in
As previously suggested, in some implementations, the scene-based image editing system 106 only adds an object to a selection if its assigned behavior specifies that it should be selected with another object. At least, in some cases, the scene-based image editing system 106 only adds the object before receiving any modification input if its assigned behavior specifies that it should be selected with another object. Indeed, in some instances, only a subset of potential edits to a first object are applicable to a second object based on the behaviors assigned to that second object. Thus, including the second object in the selection of the first object before receiving modification input risks violating the rules set forth by the behavioral policy graph via the semantic scene graph if there is not a behavior providing for automatic selection. To avoid this risk, in some implementations, the scene-based image editing system 106 waits until modification input has been received before determining whether to add the second object to the selection. In one or more embodiments, however—as suggested by
As further shown in
Additionally, as shown in
As further shown in
In one or more embodiments, the scene-based image editing system 106 provides the suggestion for adding the object 2308b to the selection based on determining the relationship between the objects 2308a-2308b via the semantic scene graph generated for the digital image 2306. In some cases, the scene-based image editing system 106 further provides the suggestion for adding the object 2308b based on the behaviors assigned to the object 2308b based on that relationship.
As suggested by
As shown in
As illustrated in
As shown in
As shown in
Indeed, as mentioned above, in one or more embodiments, the scene-based image editing system 106 waits upon receiving input to modify a first object before suggesting adding a second object (or automatically adding the second object). Accordingly, the scene-based image editing system 106 determines whether a relationship between the objects and the pending modification indicate that the second object should be added before including the second object in the selection.
To illustrate, in one or more embodiments, upon detecting the additional user interaction with the option 2412, the scene-based image editing system 106 references the semantic scene graph for the digital image 2406. Upon referencing the semantic scene graph, the scene-based image editing system 106 determines that the object 2408a has a relationship with the object 2408b. Further, the scene-based image editing system 106 determines that the behaviors assigned to the object 2408b based on that relationship indicate that the object 2408b should be deleted with the object 2408a. Accordingly, upon receiving the additional user interaction for deleting the object 2408a, the scene-based image editing system 106 determines that the object 2408b should also be deleted and then provides the suggestion to add the object 2408b (or automatically adds the object 2408b) to the selection.
As shown in
Though the above specifically discusses moving objects or deleting objects based on their relationships with other objects, it should be noted that the scene-based image editing system 106 implements various other types of relationship-aware object modifications in various embodiments. For example, in some cases, the scene-based image editing system 106 implements relationship-aware object modifications via resizing modifications, recoloring or retexturing modifications, or compositions. Further, as previously suggested, the behavioral policy graph utilized by the scene-based image editing system 106 is configurable in some embodiments. Thus, in some implementations, the relationship-aware object modifications implemented by the scene-based image editing system 106 change based on user preferences.
In one or more embodiments, in addition to modifying objects based on relationships as described within a behavioral policy graph that is incorporated into a semantic scene graph, the scene-based image editing system 106 modifies objects based on classification relationships. In particular, in some embodiments, the scene-based image editing system 106 modifies objects based on relationships as described by a real-world class description graph that is incorporated into a semantic scene graph. Indeed, as previously discussed, a real-world class description graph provides a hierarchy of object classifications for objects that may be portrayed in a digital image. Accordingly, in some implementations, the scene-based image editing system 106 modifies objects within digital images based on their relationship with other objects via their respective hierarchy of object classifications. For instance, in one or more embodiments, the scene-based image editing system 106 adds objects to a selection for modification based on their relationships with other objects via their respective hierarchy of object classifications.
In particular,
As shown in
As shown in
To illustrate, in some embodiments, in response to detecting the selection of the object 2508b and the object 2508e, the scene-based image editing system 106 references the semantic scene graph generated for the digital image 2506 (e.g., the semantic scene graph components that are associated with the object 2508b and the object 2508e). Based on referencing the semantic scene graph, the scene-based image editing system 106 determines that the object 2508b and the object 2508e are both part of the shoe class. Thus, the scene-based image editing system 106 determines that there is a classification relationship between the object 2508b and the object 2508e via the shoe class. In one or more embodiments, based on determining that the object 2508b and the object 2508e are both part of the shoe class, the scene-based image editing system 106 determines that the user interactions providing the selections are targeting all shoes within the digital image 2506. Thus, the scene-based image editing system 106 provides the text box 2528 suggesting adding the other shoes to the selection. In one or more embodiments, upon receiving a user interaction accepting the suggestion, the scene-based image editing system 106 adds the other shoes to the selection.
Similarly, as shown in
Further, as shown in
Thus, in one or more embodiments, the scene-based image editing system 106 anticipates the objects that are targeted user interactions and facilitates quicker selection of those objects based on their classification relationships. In some embodiments, upon selection of multiple objects via provided suggestions, the scene-based image editing system 106 modifies the selected objects in response to additional user interactions. Indeed, the scene-based image editing system 106 modifies the selected objects together. Thus, the scene-based image editing system 106 implements a graphical user interface that provides a more flexible and efficient approach to selecting and modifying multiple related objects using reduced user interactions.
Indeed, as previously mentioned, the scene-based image editing system 106 provides improved flexibility and efficiency when compared to conventional systems. For instance, by selecting (e.g., automatically or via suggestion) objects based on the selection of related objects, the scene-based image editing system 106 provides a flexible method of targeting multiple objects for modification. Indeed, the scene-based image editing system 106 flexibly identifies the related objects and includes them with the selection. Accordingly, the scene-based image editing system 106 implements a graphical user interface that reduces user interactions typically required under conventional system for selecting and modifying multiple objects.
In one or more embodiments, the scene-based image editing system 106 further pre-processes a digital image to aid in the removal of distracting objects. In particular, the scene-based image editing system 106 utilizes machine learning to identify objects in a digital image, classify one or more of the objects as distracting objects, and facilitate the removal of the distracting objects to provide a resulting image that is more visually cohesive and aesthetically pleasing. Further, in some cases, the scene-based image editing system 106 utilizes machine learning to facilitate the removal of shadows associated with distracting objects.
Many conventional systems are inflexible in the methods they use for removing distracting human in that they strip control away from users. For instance, conventional systems often remove humans they have classified as distracting automatically. Thus, when a digital image is received, such systems fail to provide the opportunity for user interactions to provide input regarding the removal process. For example, these systems fail to allow user interactions to remove human from the set of humans identified for removal.
Additionally, conventional systems typically fail to flexibly remove all types of distracting objects. For instance, many conventional systems fail to flexibly remove shadows cast by distracting objects and non-human objects. Indeed, while some existing systems identify and remove distracting humans from a digital image, these systems often fail to identify shadows cast by humans or other objects within the digital image. Accordingly, the resulting digital image will still include the influence of a distracting human as its shadow remains despite the distracting human itself being removed. This further causes these conventional systems to require additional user interactions to identify and remove these shadows.
The scene-based image editing system 106 addresses these issues by providing more user control in the removal process while reducing the interactions typically required to delete an object from a digital image. Indeed, as will be explained below, the scene-based image editing system 106 presents identified distracting objects for display as a set of objects selected for removal. The scene-based image editing system 106 further enables user interactions to add objects to this set, remove objects from the set, and/or determine when the selected objects are deleted. Thus, the scene-based image editing system 106 employs a flexible workflow for removing distracting objects based on machine learning and user interactions.
Further, the scene-based image editing system 106 flexibly identifies and removes shadows associated with distracting objects within a digital image. By removing shadows associated with distracting objects, the scene-based image editing system 106 provides a better image result in that distracting objects and additional aspects of their influence within a digital image are removed. This allows for reduced user interaction when compared to conventional systems as the scene-based image editing system 106 does not require additional user interactions to identify and remove shadows.
In one or more embodiments, the scene-based image editing system 106 utilizes, as the segmentation neural network 2604, one of the segmentation neural networks discussed above (e.g., the detection-masking neural network 300 discussed with reference to
As shown in
In one or more embodiments, the scene-based image editing system 106 utilizes a subset of the neural networks shown in
As illustrated, the heatmap network 2702 operates on an input image 2706 to generate heatmaps 2708. For instance, in some cases, the heatmap network 2702 generates a main subject heatmap representing possible main subject objects and a distractor heatmap representing possible distracting objects. In one or more embodiments, a heatmap (also referred to as a class activation map) includes a prediction made by a convolutional neural network that indicates a probability value, on a scale of zero to one, that a specific pixel of an image belongs to a particular class from a set of classes. As opposed to object detection, the goal of a heatmap network is to classify individual pixels as being part of the same region in some instances. In some cases, a region includes an area of a digital image where all pixels are of the same color or brightness.
In at least one implementation, the scene-based image editing system 106 trains the heatmap network 2702 on whole images, including digital images where there are no distracting objects and digital images that portray main subject objects and distracting objects.
In one or more embodiments, the heatmap network 2702 identifies features in a digital image that contribute to a conclusion that that a given region is more likely to be a distracting object or more likely to be a main subject object, such as body posture and orientation. For instance, in some cases, the heatmap network 2702 determines that objects with slouching postures as opposed to standing at attention postures are likely distracting objects and also that objects facing away from the camera are likely to be distracting objects. In some cases, the heatmap network 2702 considers other features, such as size, intensity, color, etc.
In some embodiments, the heatmap network 2702 classifies regions of the input image 2706 as being a main subject or a distractor and outputs the heatmaps 2708 based on the classifications. For example, in some embodiments, the heatmap network 2702 represents any pixel determined to be part of a main subject object as white within the main subject heatmap and represents any pixel determined to not be part of a main subject object as black (or vice versa). Likewise, in some cases, the heatmap network 2702 represents any pixel determined to be part of a distracting object as white within the distractor heatmap while representing any pixel determined to not be part of a distracting object as black (or vice versa).
In some implementations, the heatmap network 2702 further generates a background heatmap representing a possible background as part of the heatmaps 2708. For instance, in some cases, the heatmap network 2702 determines that the background includes areas that are not part of a main subject object or a distracting object. In some cases, the heatmap network 2702 represents any pixel determined to be part of the background as white within the background heatmap while representing any pixel determined to not be part of the background as black (or vice versa).
In one or more embodiments, the distractor detection neural network 2700 utilizes the heatmaps 2708 output by the heatmap network 2702 as a prior to the distractor classifier 2704 to indicate a probability that a specific region of the input image 2706 contains a distracting object or a main subject object.
In one or more embodiments, the distractor detection neural network 2700 utilizes the distractor classifier 2704 to consider the global information included in the heatmaps 2708 and the local information included in one or more individual objects 2710. To illustrate, in some embodiments, the distractor classifier 2704 generates a score for the classification of an object. If an object in a digital image appears to be a main subject object based on the local information, but the heatmaps 2708 indicate with a high probability that the object is a distracting object, the distractor classifier 2704 concludes that the object is indeed a distracting object in some cases. On the other hand, if the heatmaps 2708 point toward the object being a main subject object, the distractor classifier 2704 determines that the object has been confirmed as a main subject object.
As shown in
As illustrated by
As further shown, the distractor classifier 2704 also utilizes the crop generator 2712 to generate cropped heatmaps 2718 by cropping the heatmaps 2708 with respect to each detected object. For instance, in one or more embodiments, the crop generator 2712 generates—from each of the main subject heatmap, the distractor heatmap, and the background heatmap—one cropped heatmap for each of the detected objects based on a region within the heatmaps corresponding to the location of the detected objects.
In one or more embodiments, for each of the one or more individual objects 2710, the distractor classifier 2704 utilizes the hybrid classifier 2714 to operate on a corresponding cropped image (e.g., its features) and corresponding cropped heatmaps (e.g., their features) to determine whether the object is a main subject object or a distracting object. To illustrate, in some embodiments, for a detected object, the hybrid classifier 2714 performs an operation on the cropped image associated with the detected object and the cropped heatmaps associated with the detected object (e.g., the cropped heatmaps derived from the heatmaps 2708 based on a location of the detected object) to determine whether the detected object is a main subject object or a distracting object. In one or more embodiments, the distractor classifier 2704 combines the features of the cropped image for a detected object with the features of the corresponding cropped heatmaps (e.g., via concatenation or appending the features) and provides the combination to the hybrid classifier 2714. As shown in
To illustrate, in one or more embodiments, the scene-based image editing system 106 provides the features of a cropped image 2904 to the convolutional neural network 2902. Further, the scene-based image editing system 106 provides features of the cropped heatmaps 2906 corresponding to the object of the cropped image 2904 to an internal layer 2910 of the hybrid classifier 2900. In particular, as shown, in some cases, the scene-based image editing system 106 concatenates the features of the cropped heatmaps 2906 with the output of a prior internal layer (via the concatenation operation 2908) and provides the resulting feature map to the internal layer 2910 of the hybrid classifier 2900. In some embodiments, the feature map includes 2048+N channels, where N corresponds to the channels of the output of the heatmap network and 2048 corresponds to the channels of the output of the prior internal layer (though 2048 is an example).
As shown in
Additionally, as shown in
As shown in
In one or more embodiments, the scene-based image editing system 106 further provides the visual indicators 3014a-3014d to indicate that the objects 3010a-3010d have been selected for deletion. In some instances, the scene-based image editing system 106 also surfaces the pre-generated object masks for the objects 3010a-3010d in preparation of deleting the objects. Indeed, as has been discussed, the scene-based image editing system 106 pre-generates object masks and content fills for the objects of a digital image (e.g., utilizing the segmentation neural network 2604 and the inpainting neural network 2610 referenced above). Accordingly, the scene-based image editing system 106 has the object masks and content fills readily available for modifying the objects 3010a-3010d.
In one or more embodiments, the scene-based image editing system 106 enables user interactions to add to or remove from the selection of the objects for deletion. For instance, in some embodiments, upon detecting a user interaction with the object 3010a, the scene-based image editing system 106 determines to omit the object 3010a from the deletion operation. Further, the scene-based image editing system 106 removes the visual indication 3014a from the display of the graphical user interface 3002. On the other hand, in some implementations, the scene-based image editing system 106 detects a user interaction with the object 3008 and determines to include the object 3008 in the deletion operation in response. Further, in some cases, the scene-based image editing system 106 provides a visual indication for the object 3008 for display and/or surfaces a pre-generated object mask for the object 3008 in preparation for the deletion.
As further shown in
By enabling user interactions to control which objects are included in the deletion operation and to further choose when the selected objects are removed, the scene-based image editing system 106 provides more flexibility. Indeed, while conventional systems typically delete distracting objects automatically without user input, the scene-based image editing system 106 allows for the deletion of distracting objects in accordance with user preferences expressed via the user interactions. Thus, the scene-based image editing system 106 flexibly allow for control of the removal process via the user interactions.
In addition to removing distracting objects identified via a distractor detection neural network, the scene-based image editing system 106 provides other features for removing unwanted portions of a digital image in various embodiments. For instance, in some cases, the scene-based image editing system 106 provides a tool whereby user interactions can target arbitrary portions of a digital image for deletion.
In particular,
As further shown in
As indicated by
As further shown in
In one or more embodiments, the scene-based image editing system 106 further implements smart dilation when removing objects, such as distracting objects, from digital images. For instance, in some cases, the scene-based image editing system 106 utilizes smart dilation to remove objects that touch, overlap, or are proximate to other objects portrayed in a digital image.
Often, conventional systems remove objects from digital images utilizing tight masks (e.g., a mask that tightly adheres to the border of the corresponding object). In many cases, however, a digital image includes color bleeding or artifacts around the border of an object. For instance, there exist some image formats (JPEG) that are particularly susceptible to having format-related artifacts around object borders. Using tight masks when these issues are present causes undesirable effects in the resulting image. For example, inpainting models are typically sensitive to these image blemishes, creating large artifacts when operating directly on the segmentation output. Thus, the resulting modified images inaccurately capture the user intent in removing an object by creating additional image noise.
Thus, the scene-based image editing system 106 dilates (e.g., expands) the object mask of an object to avoid associated artifacts when removing the object. Dilating objects masks, however, presents the risk of removing portions of other objects portrayed in the digital image. For instance, where a first object to be removed overlaps, touches, or is proximate to a second object, a dilated mask for the first object will often extend into the space occupied by the second object. Thus, when removing the first object using the dilated object mask, significant portions of the second object are often removed and the resulting hole is filled in (generally improperly), causing undesirable effects in the resulting image. Accordingly, the scene-based image editing system 106 utilizes smart dilation to avoid significantly extending the object mask of an object to be removed into areas of the digital image occupied by other objects.
As shown in
As further illustrated in
Additionally, as shown in
After expanding the object mask 3208, the scene-based image editing system 106 performs an act 3214 of detecting overlap between the expanded object mask for the object 3202 and the object masks of the other detected objects 3206a-3206b (i.e., the combined object mask 3210). In particular, the scene-based image editing system 106 determines where pixels corresponding to the expanded representation of the object 3202 within the expanded object mask overlap pixels corresponding to the objects 3206a-3206b within the combined object mask 3210. In some cases, the scene-based image editing system 106 determines the union between the expanded object mask and the combined object mask 3210 and determines the overlap using the resulting union. The scene-based image editing system 106 further performs an act 3216 of removing the overlapping portion from the expanded object mask for the object 3202. In other words, the scene-based image editing system 106 removes pixels from the representation of the object 3202 within the expanded object mask that overlaps with the pixels corresponding to the object 3206a and/or the object 3206b within the combined object mask 3210.
Thus, as shown in
To describe it differently, in one or more embodiments, the scene-based image editing system 106 generates the smartly dilated object mask 3218 (e.g., an expanded object mask) by expanding the object mask 3208 for the object 3202 into areas not occupied by the object masks for the objects 3206a-3206b (e.g., areas not occupied by the objects 3206a-3206b themselves). For instance, in some cases, the scene-based image editing system 106 expands the object mask 3208 into portions of the digital image 3204 that abut the object mask 3208. In some cases, the scene-based image editing system 106 expands the object mask 3208 into the abutting portions by a set number of pixels. In some implementations, the scene-based image editing system 106 utilizes a different number of pixels for expanding the object mask 3208 into different abutting portions (e.g., based on detecting a region of overlap between the object mask 3208 and other object masks).
To illustrate, in one or more embodiments, the scene-based image editing system 106 expands the object mask 3208 into the foreground and the background of the digital image 3204. In particular, the scene-based image editing system 106 determines foreground by combining the object masks of objects not to be deleted. The scene-based image editing system 106 expands the object mask 3208 into the abutting foreground and background. In some implementations, the scene-based image editing system 106 expands the object mask 3208 into the foreground by a first amount and expands the object mask 3208 into the background by a second amount that differs from the first amount (e.g., the second amount is greater than the first amount). For example, in one or more implementations the scene-based image editing system 106 expands the object mask by twenty pixels into background areas and two pixels into foreground areas (into abutting object masks, such as the combined object mask 3210).
In one or more embodiments, the scene-based image editing system 106 determines the first amount to use for the expanding the object mask 3208 into the foreground by expanding the object mask 3208 into the foreground by the second amount—the same amount used to expand the object mask 3208 into the background. In other words, the scene-based image editing system 106 expands the object mask 3208 as a whole into the foreground and background by the same amount (e.g., using the same number of pixels). The scene-based image editing system 106 further determines a region of overlap between the expanded object mask and the object masks corresponding to the other objects 3206a-3206b (e.g., the combined object mask 3210). In one or more embodiments, the region of overlap exists in the foreground of the digital image 3204 abutting the object mask 3208. Accordingly, the scene-based image editing system 106 reduces the expansion of the object mask 3208 into the foreground so that the expansion corresponds to the second amount. Indeed, in some instances, the scene-based image editing system 106 removes the region of overlap from the expanded object mask for the object 3202 (e.g., removes the overlapping pixels). In some cases, scene-based image editing system 106 removes a portion of the region of overlap rather than the entire region of overlap, causing a reduced overlap between the expanded object mask for the object 3202 and the object masks corresponding to the objects 3206a-3206b.
In one or more embodiments, as removing the object 3202 includes removing foreground and background abutting the smartly dilated object mask 3218 (e.g., the expanded object mask) generated for the object 3202, the scene-based image editing system 106 inpaints a hole remaining after the removal. In particular, the scene-based image editing system 106 inpaints a hole with foreground pixels and background pixels. Indeed, in one or more embodiments, the scene-based image editing system 106 utilizes an inpainting neural network to generate foreground pixels and background pixels for the resulting hole and utilizes the generated pixels to inpaint the hole, resulting in a modified digital image (e.g., an inpainted digital image) where the object 3202 has been removed and the corresponding portion of the digital image 3204 has been filled in.
For example,
In contrast,
By generating smartly dilated object masks, the scene-based image editing system 106 provides improved image results when removing objects. Indeed, the scene-based image editing system 106 leverages expansion to remove artifacts, color bleeding, or other undesirable errors in a digital image but avoids removing significant portions of other objects that are remain in the digital image. Thus, the scene-based image editing system 106 is able to fill in holes left by removed objects without enhancing present errors where possible without needlessly replacing portions of other objects that remain.
As previously mentioned, in one or more embodiments, the scene-based image editing system 106 further utilizes a shadow detection neural network to detect shadows associated with distracting objects portrayed within a digital image.
In particular,
As shown in
In some embodiments, for each detected object, the scene-based image editing system 106 generates input for the second stage of the shadow detection neural network (i.e., the shadow prediction component).
In one or more embodiments, the scene-based image editing system 106 (e.g., via the object awareness component 3500 or some other component of the shadow detection neural network) generates the combined object mask 3512 using the union of separate object masks generated for the object 3504b and the object 3504c. In some instances, the object awareness component 3500 does not utilize the object-discriminative channel (e.g., the combined object mask 3512). Rather, the object awareness component 3500 generates the input 3506 using the input image 3508 and the object mask 3510. In some embodiments, however, using the object-discriminative channel provides better shadow prediction in the second stage of the shadow detection neural network.
Based on the outputs of the shadow segmentation model 3608, the shadow prediction component 3600 provides an object-shadow pair prediction 3614 for the object of interest. In other words, the shadow prediction component 3600 associates the object of interest with its shadow cast within the input image 3602. In one or more embodiments, the shadow prediction component 3600 similarly generates an object-shadow pair prediction for all other objects portrayed in the input image 3602. Thus, the shadow prediction component 3600 identifies shadows portrayed in a digital image and associates each shadow with its corresponding object.
In one or more embodiments, the shadow segmentation model 3608 utilized by the shadow prediction component 3600 includes a segmentation neural network. For instance, in some cases, the shadow segmentation model 3608 includes the detection-masking neural network 300 discussed above with reference to
In some implementations, the shadow detection neural network 3700 determines that an object portrayed within a digital image does not have an associated shadow. Indeed, in some cases, upon analyzing the digital image utilizing its various components, the shadow detection neural network 3700 determines that there is not a shadow portrayed within the digital image that is associated with the object. In some cases, the scene-based image editing system 106 provides feedback indicating the lack of a shadow. For example, in some cases, upon determining that there are no shadows portrayed within a digital image (or that there is not a shadow associated with a particular object), the scene-based image editing system 106 provides a message for display or other feedback indicating the lack of shadows. In some instances, the scene-based image editing system 106 does not provide explicit feedback but does not auto-select or provide a suggestion to include a shadow within a selection of an object as discussed below with reference to
In some implementations, the scene-based image editing system 106 utilizes the second stage of the shadow detection neural network to determine shadows associated with objects portrayed in a digital image when the objects masks of the objects have already been generated. Indeed,
As shown in
By providing direct access to the second stage of the shadow detection neural network, the scene-based image editing system 106 provides flexibility in the shadow detection process. Indeed, in some cases, an object mask will already have been created for an object portrayed in a digital image. For instance, in some cases, the scene-based image editing system 106 implements a separate segmentation neural network to generate an object mask for a digital image as part of a separate workflow. Accordingly, the object mask for the object already exists, and the scene-based image editing system 106 leverages the previous work in determining the shadow for the object. Thus, the scene-based image editing system 106 further provides efficiency as it avoids duplicating work by accessing the shadow prediction model of the shadow detection neural network directly.
In one or more embodiments, upon receiving the digital image 3906, the scene-based image editing system 106 utilizes a shadow detection neural network to analyze the digital image 3906. In particular, the scene-based image editing system 106 utilizes the shadow detection neural network to identify the object 3908, identify the shadow 3910 cast by the object 3908, and further associate the shadow 3910 with the object 3908. As previously mentioned, in some implementations, the scene-based image editing system 106 further utilizes the shadow detection neural network to generate object masks for the object 3908 and the shadow 3910.
As previously discussed with reference to
Indeed, as shown in
For instance, in some cases, the scene-based image editing system 106 receives a user selection of the object 3908 and automatically adds the shadow 3910 to the selection. In some implementations, the scene-based image editing system 106 receives a user selection of the object 3908 and provides a suggestion for display in the graphical user interface 3902, suggesting that the shadow 3910 be added to the selection. In response to receiving an additional user interaction, the scene-based image editing system 106 adds the shadow 3910.
As further shown in
Though
By identifying shadows cast by objects within digital images, the scene-based image editing system 106 provides improved flexibility when compared to conventional systems. Indeed, the scene-based image editing system 106 flexibly identifies objects within a digital image along with other aspects of those objects portrayed in the digital image (e.g., their shadows). Thus, the scene-based image editing system 106 provides a better image result when removing or moving objects as it accommodates these other aspects. This further leads to reduced user interaction with a graphical user interface as the scene-based image editing system 106 does not require user interactions for targeting the shadows of objects for movement or removal (e.g., user interactions to identify shadow pixels and/or tie the shadow pixels to the object).
In some implementations, the scene-based image editing system 106 implements one or more additional features to facilitate the modification of a digital image. In some embodiments, these features provide additional user-interface-based efficiency in that they reduce the amount of user interactions with a user interface typically required to perform some action in the context of image editing. In some instances, these features further aid in the deployment of the scene-based image editing system 106 on computing devices with limited screen space as they efficiently use the space available to aid in image modification without crowding the display with unnecessary visual elements.
As mentioned above, in some implementations, the scene-based image editing system 106 utilizes machine learning models to generate and/or expand semantic guides to modify or complete digital images. Given a digital portrait or photo, client devices often seek to remove unwanted accessories, unexpected occluders, or undesired leg/feet cropping. The scene-based image editing system 106 provides an authentic and flexible solution for efficient, accurate, and flexible portrait photo editing. Indeed, the scene-based image editing system 106 leverages semantic map guidance in machine learning models to remove accessories or occlusions, uncrop digital images, and control content creation. For example, the scene-based image editing system 106 can utilize a diffusion model (or other generative machine learning model) to complete a semantic map to progressively generate an image layout. Then, using the map as guidance, the scene-based image editing system can generate an RGB image with authenticity.
The scene-based image editing system 106 can perform a variety of editing processes. For example, in some implementations, the scene-based image editing system utilizes semantic map guidance and machine learning models for accessory removal. In particular, the scene-based image editing system can crop an occlusion or accessory from a digital image, utilize a first generative machine learning model to generate a semantic map of the missing region, and then utilize a second generative machine learning model to generate a complete digital image from the semantic map (with the accessory or occlusion removed from the final digital image).
The scene-based image editing system 106 can also uncrop or expand digital images. To illustrate, the scene-based image editing system can expand the size or scope of a digital image, utilize a first generative machine learning model to generate a semantic map of the expanded region, and then utilize a second generative machine learning model to generate a complete digital image. In this manner, the scene-based image editing system can expand a close up digital image of a person to include a more complete picture of the target subject (e.g., the person's legs, feet, or arms and corresponding background).
In one or more embodiments, the scene-based image editing system 106 performs progressive semantic completion and progressive image completion. For example, the scene-based image editing system 106 can iteratively utilize a diffusion model to generate progressively more accurate semantic maps and/or completed digital images. In this manner, the scene-based image editing system iteratively improves the accuracy of the resulting maps and images to generate a realistic/authentic result.
Moreover, the scene-based image editing system 106 can also provide more flexibility and controllability during editing. For example, the scene-based image editing system 106 can generate a diverse pool of possible solutions for user selection. Moreover, the scene-based image editing system 106 can allow a client device to provide user input of strokes, style guides, or color patches at a desired region to guide the image generation directly. Indeed, by including a style input within a region of a digital image, the scene-based image editing system 106 can utilize a machine learning model to expand/apply the style to the entire semantic region in generating a modified digital image. Similarly, by considering user input/modification of semantic regions, the scene-based image editing system 106 can flexibly guide generation of completed digital images to accurately reflect desired features.
For example,
Furthermore, the scene-based image editing system 106 provides options to generate the modified digital image 4012 within a user interface. For instance, the scene-based image editing system 106 provides client devices with more flexibility and controllability during editing. The user interface has various options to indicate the type of modification (e.g., expanding the digital image or removing an object) and the scene-based image editing system 106 further provides customizable options for modifying the digital image 4002. For example, the scene-based image editing system 106 can generate a diverse pool of possible solutions for user selection. Moreover, the scene-based image editing system 106 can allow a client device to provide user input of strokes, style guides, or color patches at a desired region to guide the image generation. Indeed, by including a style input within a region of digital image 4002, the scene-based image editing system 106 can utilize a machine learning model to expand/apply the style to the entire semantic region in generating the modified digital image 4012. Similarly, by considering user input/modification of semantic regions, the scene-based image editing system 106 can flexibly guide generation of completed digital images to accurately reflect desired features. More details relating to these features are given below starting at the description of
As mentioned above, the scene-based image editing system 106 generates the modified digital image 4012 from the digital image 4002. As shown,
As shown in
As mentioned, the scene-based image editing system 106 replaces or adds pixel values to a region. For example, the region indicated for the infill modification includes a portion of the digital image 4002, either from the existing digital image or from an expanded version of the digital image. In some instances, the region includes one or more objects within the digital image 4002. While in other instances (as already mentioned) the region includes expanding the frame of the digital image and the region encompasses the portion corresponding to the expanded frame. As shown in
Furthermore,
As just mentioned, the scene-based image editing system 106 generates the semantic map 4007. For example, the scene-based image editing system 106 generates the semantic map 4007 (e.g., an initial semantic map) based on the portrayal within the digital image 4002. In particular, the semantic map 4007 includes classifications of pixels within the digital image 4002. The scene-based image editing system 106 via the semantic map model 4005 detects the classifications and objects portrayed within the digital image 4002. For the semantic map 4007, the scene-based image editing system 106 highlights each objects of a single class with a different color shade (or other visual indicator) within a semantic boundary. To illustrate, the digital image 4002 depicts a human, the scene-based image editing system 106 generates a semantic map by segmenting the human from the background and highlighting each sub-portion of the human. For example, the semantic map 4007 as shown in
As mentioned above, the scene-based image editing system 106 detects and classifies various objects portrayed within the digital image 4002. For example, an object includes a collection of pixels in the digital image 4002 that depicts a person, place, or thing. To illustrate, in some embodiments, an object includes a person, an item, a natural object (e.g., a tree or rock formation) or a structure depicted in a digital image. In some instances, an object includes a plurality of elements that, collectively, can be distinguished from other elements depicted in the digital image 4002. For example, in some instances, an object includes a collection of buildings that make up a skyline. In some instances, an object more broadly includes a (portion of a) foreground or other element(s) depicted in the digital image 4002 as distinguished from a background. Specifically, for human semantic maps, the object includes sub-parts of the human such as body parts and articles of clothing associated with the human.
As further shown,
In one or more embodiments, machine learning models include generative machine learning models. In particular, generative machine learning models includes machine learning models that generate or create digital image content. For example, generative machine learning models include diffusion networks. Generative machine learning models also include generative adversarial neural networks (GANs). For instance, in a GAN, two machine learning models compete with each other in a zero-sum game.
As mentioned above, the scene-based image editing system 106 utilizes generative machine learning models. In particular, as already mentioned, the scene-based image editing system 106 utilizes the generative semantic machine learning model 4006. For instance, the generative semantic machine learning model 4006 is a generative machine learning model that generates semantic maps of the digital image. Specifically, the generative semantic machine learning model generates realistic semantic maps that fill in gaps of an expanded digital image or a semantic map of the digital image 4002. To illustrate, the generative semantic machine learning model 4006 generates the infill semantic map 4008 from the initially generated semantic map (e.g., the semantic map 4007).
As mentioned above, the scene-based image editing system 106 generates the infill semantic map 4008 that indicates semantic classifications corresponding to the digital image 4002 with the expanded portion infilled. For example, similar to the discussion above related to the semantic map 4007, the infill semantic map 4008 also includes classifications and boundaries of various objects within the digital image. However, the scene-based image editing system 106 generates the infill semantic map 4008 based on an infill modification that indicates a region to fill for an expanded digital image frame of the digital image 4002. To illustrate, if the infill modification indicates a region to fill that includes expanding the frame of the digital image 4002, then the scene-based image editing system 106 generates an infill semantic map that corresponds to the indicated region. In this instance, the infill semantic map 4008 includes semantic classifications and semantic boundaries for the expanded portion of the digital image. In other instances, when the infill modification indicates a region to fill that includes replacing a portion of the existing digital image, then the scene-based image editing system 106 generates an infill semantic map that generates a segmented portion corresponding to the portion to be replaced with corresponding semantic classifications and semantic boundaries. More details relating to semantic boundaries and classifications are given below in relation to
Additionally, as further shown,
As shown in the modified digital image 4012, the scene-based image editing system 106 infills the portion below the knees of the person portrayed in the digital image 4002 (which only portrays the human from the knees up). The results generated by the scene-based image editing system 106 illustrate an efficient, accurate, and high-quality completion of the digital image 4002 by completing the semantic map 4007 and utilizing the infill semantic map 4008 to generate the modified digital image 4012.
Conventional systems have leveraged recent computing advancements to modify digital images utilizing a variety of digital tools and models. To illustrate, conventional systems utilize computer-implemented models to uncrop, inpaint, and modify digital objects portrayed in digital images. Despite these advancements, however, conventional systems continue to suffer from a number of technical deficiencies, particularly with regard to accuracy, efficiency, and flexibility of implementing systems in generating modified digital images.
For instance, conventional systems are able to uncrop and inpaint digital images, but these systems often generate inaccurate and unrealistic results. Similarly, conventional systems typically require a large number of user interactions and user interfaces. The user interactions and interfaces required to add or remove regions consumes valuable computational resources and time. In addition, conventional systems provide limited functional flexibility in inpainting digital images. For example, conventional systems seek to generate pixels that will blend with surrounding pixels but fail to provide additional functional control with regard to features or factors in implementing underlying models.
As just mentioned, the scene-based image editing system 106 can improve upon accuracy of conventional systems. For example, the scene-based image editing system 106 utilizes the infill semantic map 4008 and the digital image 4002 to create a realistically infilled digital image (e.g., the modified digital image 4012). In particular, the scene-based image editing system 106 by generating the infill semantic map 4008 (utilizing a generative machine learning model) and generating the modified digital image 4012 from the infill semantic map (utilizing a generative machine learning model), the scene-based image editing system 106 generates digital images that accurately and realistically portray target objects. Indeed, in some embodiments, the scene-based image editing system 106 conditions a diffusion neural network to more accurately generate infill semantic maps and accurately resulting digital images. Thus, as illustrated in
Furthermore, in one or more implementations the scene-based image editing system 106 also improves upon efficiency relative to conventional systems. For example, the scene-based image editing system 106 can generate a modified digital image with reduced user interactions, user interfaces, and time-consuming corrections/edits associated with many conventional systems. For example, the scene-based image editing system 106 generates the semantic map 4007, the infill semantic map 4008, and the modified digital image 4012 with various machine learning models. Accordingly, the scene-based image editing system 106 saves valuable time and computational resources by intelligently generating the modified digital image 4012. Moreover, the scene-based image editing system 106 can provide streamlined interfaces for an intuitive and simplified flow to generate the infill semantic map 4008 and/or the modified digital image 4012. In particular, the scene-based image editing system 106 can provide a unified user interface for generating both an infill semantic map and a modified digital image to that reduces or eliminates the need to alternate between multiple interfaces as utilized by conventional systems.
In addition to the above, the scene-based image editing system 106 also improves upon functional flexibility. For example, the scene-based image editing system 106 allows client devices to expand the frame of digital images, remove objects portrayed within a digital image, or modify textures within portions of a digital images. Moreover, the scene-based image editing system 106 provides improved functionality with user interfaces that allow for unique control over implementing models, such as the number of infill segmentation maps to generate, the number of modified images to generate, the number of layers to utilizes (e.g., within a diffusion neural network), textures for conditioning the generative model, and semantic editing input to guide generation infill semantic maps and/or modified digital images.
As discussed above, the scene-based image editing system 106 utilizes various types of machine learning models. For example,
As mentioned above, the scene-based image editing system 106 utilizes a diffusion neural network. In particular, a diffusion neural network receives as input a digital image and adds noise to the digital image through a series of steps. For instance, the scene-based image editing system 106 via the diffusion neural network maps a digital image to a latent space utilizing a fixed Markov chain that adds noise to the data of the digital image. Furthermore, each step of the fixed Markov chain relies upon the previous step. Specifically, at each step, the fixed Markov chain adds Gaussian noise with variance, which produces a diffusion representation (e.g., diffusion latent vector, a diffusion noise map, or a diffusion inversion). Subsequent to adding noise to the digital image at various steps of the diffusion neural network, the scene-based image editing system 106 utilizes a trained denoising neural network to recover the original data from the digital image. Specifically, the scene-based image editing system 106 utilizes a denoising neural network with a length T equal to the length of the fixed Markov chain to reverse the process of the fixed Markov chain.
As mentioned earlier, in one or more embodiments the scene-based image editing system 106 generates an infill semantic map.
As illustrated,
Furthermore,
As just mentioned, the diffusion process 4108 adds noise at each step of the diffusion process 4108. Indeed, at each diffusion step, the diffusion process 4108 adds noise and generates a diffusion representation. Thus, for a diffusion process 4108 with five diffusion steps, the diffusion process 4108 generates five diffusion representations. As shown in
As shown,
In one or more implementations, the scene-based image editing system 106 trains the denoising neural networks in a supervised manner based on the diffusion representations generates at the diffusion process 4108. For example, the scene-based image editing system 106 compares (utilizing a loss function) a diffusion representation at a first step of the diffusion process 4108 with a final denoised representation generated by the final denoising neural network. Similarly, the scene-based image editing system 106 can compare (utilizing a loss function) a second diffusion representation from a second step of the diffusion process 4108 with a penultimate denoised representation generated by a penultimate denoising neural network. The scene-based image editing system 106 can thus utilize corresponding diffusion representations of the diffusion process 4108 to teach or train the denoising neural networks to denoise random Gaussian noise and generate realistic digital images.
The scene-based image editing system 106 can utilize a variety of neural network formulations for the denoising neural networks 4112a, 4112b. For example, in some implementations, the scene-based image editing system 106 utilizes a time conditional U-Net, as described by O. Ronneberger, P. Fischer, and T. Brox, in U-net: Convolutional networks for biomedical image segmentation, MICCAI (3), Vol. 9351 of Lecture Notes in Computer Science, p. 234-241 (2015), which is incorporated by reference herein in its entirety. In one or more embodiments, the scene-based image editing system 106 utilizes the diffusion architecture and training approach described by R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, in High-Resolution Image Synthesis with Latent Diffusion Models, arXiv:2112.10752v2, which is incorporated by reference in its entirety herein.
In addition to the training illustrated in
As shown, the scene-based image editing system 106 conditions the denoising neural network 4112a and the denoising neural network 4112b. For example,
For instance,
Furthermore, in response to the user of the client device providing this indication, the scene-based image editing system 106 processes the semantic editing input for conditioning. Accordingly, the scene-based image editing system 106 can condition the network based on a combination of one or more of the digital image, the semantic map 4119 of the digital image, and/or the semantic editing input. More details relating to the scene-based image editing system 106 providing the option to a user of a client device to indicate the semantic editing input is given below (e.g., in relation to
Furthermore, the scene-based image editing system 106 utilizes the conditioning representation generated from the encoder 4122 to perform the act 4120 of conditioning the layers of the denoising neural network 4112a and denoising neural network 4112b. Specifically, conditioning the layers of the network includes modifying input into the layers of the denoising neural networks to combine with the random noise input 4111. For instance, the scene-based image editing system 106 combines (e.g., concatenates) the conditioning representation generated from the encoder 4122 at different layers of the denoising neural networks. For instance, the scene-based image editing system 106 combines one or more conditioning vectors with the input 4111 of random noise, the first denoised representation 4114, or other intermediate denoised representations analyzed by denoising layers. Thus, the denoising process considers the semantic map 4119 and the digital image to generate denoised representations.
As shown,
Although
As mentioned above,
As shown,
As further shown,
Moreover,
As just mentioned, the scene-based image editing system 106 generates the modified digital image 4224 by conditioning layers of the denoising neural networks. In particular,
As also mentioned above, in one or more implementations the scene-based image editing system 106 generates a modified digital image utilizing an input texture. For example,
The scene-based image editing system 106 receives an indication to replace pixel values within a digital image 4315 and an input texture 4311 utilizing a trained denoising neural network. In particular, the scene-based image editing system 106 replaces the indicated pixel values with the input texture 4311. Specifically, the input texture 4311 includes a sample texture that portrays a pattern selected by a user of a client device. To illustrate, the scene-based image editing system 106 modifies an input digital image 4315 with the specifically selected pattern by localizing the input texture 4311 to the relevant region of the input digital image 4315. For example, the scene-based image editing system 106 receives the input texture 4311 from a client device or from a selection of a pre-defined texture option.
In one or more embodiments, the scene-based image editing system 106 utilizes a diffusion neural network to generate the modified digital image 4316 that includes the input texture 4311. Similar to the discussion above, the scene-based image editing system 106 also utilizes diffusion neural networks to replace a region in the digital image 4315 with the input texture 4311. In some implementations, the scene-based image editing system 106 isolates texture modifications to certain portions of the digital image within the diffusion neural network. In particular,
As mentioned above, the scene-based image editing system 106 can apply the denoising and conditioning process so that a texture is localized to certain portions or regions of the modified digital image. Indeed, applying conditioning vectors based on the input texture 4311 throughout the denoising process in a global fashion could result in adding texture to the entire modified digital image. The scene-based image editing system 106 can localize the texture to a certain region (i.e., the area of the dress) by applying a mask between the denoising layers and utilizing an unconditioned representation of the input digital image for regions outside of the mask.
For instance,
The scene-based image editing system 106 then combines the first masked denoised representation 4322a with untextured image representation 4310a. Indeed, as shown, the scene-based image editing system 106 identifies or generates the untextured image representation 4310a as a representation of the digital image 4315 without texture. The scene-based image editing system 106 can generate the untextured image representation 4310a in a variety of ways. For example, in some implementations, the scene-based image editing system 106 utilizes the output of a diffusion layer at the same level of a corresponding denoising layer as the untextured image representation 4310a. In some implementations, the scene-based image editing system 106 applies the denoising layers without condition vectors to generate the untextured image representation 4310a.
As shown, the scene-based image editing system 106 can combine the untextured image representation 4310a with the first masked denoised representation 4322a. Specifically, the scene-based image editing system 106 applies the mask 4320 to the untextured image representation 4310a (i.e., to extract the region outside of the dress) to generate a masked untextured image representation. The scene-based image editing system 106 then combines the masked untextured image representation with the first masked denoised representation 4322a. This results in a modified denoised representation where the area corresponding to the dress is conditioned on the input texture 4311 whereas the area outside the dress is not conditioned on the input texture 4311 (but the entire representation is at the same level of noise for input into the next denoising layer). The scene-based image editing system 106 then analyzes this modified denoised representation utilizing a second denoising layer 4318b.
As shown, the scene-based image editing system 106 repeats this approach at each denoising layer. For example, the second denoising layer 4318b generates a second denoised representation 4312b. The scene-based image editing system 106 applies the mask 4320 to the second denoised representation 4312b to generate a second masked denoised representation 4322b. The scene-based image editing system 106 generates a second untextured image representation 4310b (with the same level of noise as the second denoised representation 4312b) and applies the mask 4320 to generate a second masked untextured image representation. The scene-based image editing system 106 then combines the second masked untextured image representation with the second masked denoised representation 4322b to generate a second modified representation. This second modified representation is then utilized by the next denoising layer.
In one or more embodiments, the scene-based image editing system 106 applies a similar process to the output of the final denoising layer. In particular, the scene-based image editing system 106 applies a mask to the final denoised representation (to preserve the texture within the dress region) and applies a mask to an untextured representation (to preserve the background outside of the dress region without texture). The scene-based image editing system 106 then applies a decoder 4314 to the resulting latent vector to generate the modified digital image 4316. Thus,
As mentioned above, in one or more embodiments the scene-based image editing system 106 utilizes segmentation image neural networks. For example,
As shown,
The scene-based image editing system 106 can identify the user selection 4402 in a variety of ways. In some implementations, the scene-based image editing system 106 provides an editing tool to a client device and identifies the region to be removed based on user interaction with the editing tool. Specifically, the client device can identify a user interaction (e.g., a click, finger press, or drawing) to indicate scene-based image editing system 106 the portion to segment.
In some embodiments, the scene-based image editing system 106 utilizes the segmentation machine learning model 4404 to select the items. The segmentation machine learning model 4404 can also segment or select objects based on user input (or without user input). For example, in some implementations the scene-based image editing system 106 provides an option to the user of the client device to query a portion to be selected (via a selection or a natural language input). For instance, the user of the client device queries the scene-based image editing system 106 to select “bags” shown in the digital image 4400. Based on the query, the scene-based image editing system 106 utilizes the segmentation machine learning model 4404 to segment the portion of the digital image 4400 that portrays “bags.” In other embodiments, the user of the client device queries the scene-based image editing system 106 to select “pants” or “shirt” or “hair.”
In some implementations, the scene-based image editing system 106 utilizes the user selection 4402 with the segmentation machine learning model 4404 to segment or select an object in a digital image. For example, the scene-based image editing system 106 can utilize the segmentation machine learning model 4404 to analyze user inputs (e.g., clicks, drawings, outlines, or natural language) with the digital image to generate a segmentation. For example, the scene-based image editing system 106 can utilize the segmentation approach described in UTILIZING INTERACTIVE DEEP LEARNING TO SELECT OBJECTS IN DIGITAL VISUAL MEDIA, U.S. application Ser. No. 16/376,704, filed on Apr. 5, 2019, which is incorporated by reference herein in its entirety.
As mentioned previously in relation to
As further illustrated,
Accordingly, for the scene-based image editing system 106 providing the option to provide user input to guide the semantic map, the scene-based image editing system 106 provides an option to indicate the direction and position of the hands beneath the items removed (e.g., the bags). This is discussed in additional detail below (e.g., with regard to
In response to the scene-based image editing system 106 performing the act 4412 of providing the option to provide user input to guide the generation of the infill semantic map 4414, the scene-based image editing system 106 further utilizes the generative semantic machine learning model 4413 to generate the infill semantic map 4414 based on the user input. As indicated by
Moreover,
Although
As mentioned above, in some embodiments the scene-based image editing system 106 provides a user interface to a user of a client device to generate a modified digital image. For example,
Furthermore, the selectable expansion option 4500 includes an option to indicate a percentage by which to expand the digital image 4502 and a position of the digital image 4502 within an expanded frame 4504. For instance,
In response to an indication of the parameters to expand the digital image 4502, the scene-based image editing system 106 shows next to the digital image 4502 a semantic map 4506. Similar to the above discussion, the scene-based image editing system 106 generates the semantic map 4506 utilizing a semantic map model. Furthermore,
As just mentioned, the scene-based image editing system 106 receives a selection of the semantic completion option 4512. As illustrated,
In addition,
As also shown in
As shown,
In one or more embodiments, the scene-based image editing system 106 provides the client device 4608 with an option to select multiple semantic options. In particular, the scene-based image editing system 106 allows the user of the client device 4608 to select two or more semantic options. Furthermore, in response to selecting multiple semantic options and receiving a digital image completion selection (labeled “complete RGB”), the scene-based image editing system 106 generates a modified digital image for each semantic option selected.
In one or more implementations, the scene-based image editing system 106 generates multiple modified digital images from a single semantic map. For example,
The above discussion for
As mentioned above, the scene-based image editing system 106 can provider user interfaces for editing semantic maps in generating a modified digital image. For example,
Semantic classifications were previously mentioned in relation to the generation of the semantic map 4908. For example, generating semantic maps and infill semantic maps includes generating a semantic classification. In particular, a semantic classification includes a classification for an object of the digital image 4900. The scene-based image editing system 106 can include these classifications in a variety of ways, including numbers, text, textures, or other labels. In relation to
Accordingly, utilizing a semantic map model, scene-based image editing system 106 classifies each pixel of the digital image 4900. For the semantic classification, the scene-based image editing system 106 utilizes color shades to indicates objects such as hair, face, arms, shirt, torso, legs, pants, feet, shoes, and/or ankles. Moreover, for instances where generating the infill semantic map includes an expanded frame of the digital image, the scene-based image editing system 106 fills in the expanded frame for the infill semantic map by utilizing semantic classifications within the semantic map 4908. To illustrate, the digital image 4900 portrays a human from the knees up, and the scene-based image editing system 106 determines the infill modification as expanding the digital image 4900 to show the portrayal of the human below the knees. In such an instance, the scene-based image editing system 106 first generates the semantic map 4908 portraying the digital image of the human from the knees up. The semantic classification for pixels within the digital image 4900 guides the generative semantic machine learning model to fill-in the expanded portion of the digital image 4900. For instance, to complete the semantic map 4908 from the knees down, the scene-based image editing system 106 utilizes the semantic classification indicating the legs to complete the legs in the infill semantic map and add additional semantic classifications such as the feet.
As just mentioned above, the scene-based image editing system 106 generates semantic classifications when generating the semantic maps. In some embodiments, the scene-based image editing system 106 also generates a semantic boundary. In particular, the semantic boundary includes an indication of the shape or the form of pixels sharing a common semantic classification. For instance, a semantic boundary indicates an area of pixels sharing a pant semantic classification. Accordingly, in some implementations, the scene-based image editing system 106 identifies the semantic editing input 4906 to add/modify pixel classifications or to add/modify a classification boundary.
As just mentioned, the scene-based image editing system 106 provides a semantic editing input tool for the user of the client device to indicate the semantic boundaries.
Similar to the discussion in
As mentioned, the semantic editing input in
Moreover, the scene-based image editing system 106 provides a semantic editing option 4902 within the user interface to further edit the semantic. In particular, selecting the semantic editing option 4902 allows for modification of the semantic editing input 4906 to generate a different infill semantic map with different semantic classifications or semantic boundaries. For example, the scene-based image editing system 106 can modify semantic classifications of pixels/regions, change classification boundaries, or add new semantic classifications/boundaries.
As mentioned, the scene-based image editing system 106 provides options for selecting regions in a digital image to remove. For example,
Furthermore,
Although
Moreover,
As mentioned above, in one or more embodiments the scene-based image editing system 106 provides the option for a client device to replace a segmented portion within a digital image with an input texture. For example,
For example,
Specifically, for the texture selection option 5108
For example,
As mentioned above, in one or more implementations the scene-based image editing system 106 provides an option to a client device to query scene-based image editing system 106 and select a portion of the digital image to modify. For example,
In particular,
In one or more embodiments, the scene-based image editing system 106 has additional query options such as selecting only the apparel, selecting only the background, or selecting a specific sub-portion (the dress, the hat, the shoes, the shirt, etc.). In particular, based on the query, the scene-based image editing system 106 utilizes a segmentation image neural network to identify the relevant portions within the digital image 5200 and generates a identify the relevant portion to modify.
Moreover,
Although not shown in
In one or more embodiments, the scene-based image editing system 106 intelligently utilizes the features discussed in
Furthermore, in one or more embodiments, the scene-based image editing system 106 intelligently detects occluders or detractors in a digital image. For example, the scene-based image editing system 106 detects occluders/detractors covering part of a human portrayed within a digital image and removes the occluders/detractors. In particular, the scene-based image editing system 106 removes the occluders/detractors and generates an infill modification in place of the removed occluders/detractors. For instance, the scene-based image editing system 106 intelligently removes portions of a uploaded digital image and infills the removed portion.
Moreover, in one or more embodiments, the scene-based image editing system 106 intelligently detects textures within a digital image and recommends alternative textures. For example, the scene-based image editing system 106 detects apparel portrayed within a digital image and extracts the texture of the apparel. In particular, the scene-based image editing system 106 then provides a recommendation of texture options to replace the current texture shown within the digital image.
In addition, in one or more embodiments, the scene-based image editing system 106 intelligently detects other portions within a digital image and recommends options to replace detected portions. For example, the scene-based image editing system 106 detects apparel, background, hair, and other accessories (e.g., purse, backpack, hat, glasses, etc.). In particular, the scene-based image editing system 106 detects various portions of the digital image and provides multiple recommendations for each detected portion to replace the detected portion. Accordingly, for each of the above embodiments, the scene-based image editing system 106 intelligently and automatically performs infill modifications or infill recommendations to a user of a client device in real-time or near real-time (e.g., seconds).
As briefly mentioned above, in one or more implementations the scene-based image editing system 106 generates modified digital images by inpainting indicated portions of a digital image. Specifically, in one or more embodiments the scene-based image editing system 106 performs partial human inpainting with generative machine learning models specially trained to accommodate inpainting of human subjects from digital images. For example, for a digital image that portrays a human with an object covering (e.g., obstructing or occluding) part of the human, the scene-based image editing system 106 utilizes a human inpainting GAN to perform human inpainting. In particular, the scene-based image editing system 106 generates a modified digital image by utilizing a structural guidance map and a digital image portraying a human.
In addition, in one or more embodiments the scene-based image editing system 106 performs human inpainting to generate a modified digital image by utilizing two-dimensional modulation. For example, the scene-based image editing system 106 generates utilizing an encoder, a visual appearance encoding from the digital image portraying the human. Moreover, the scene-based image editing system 106 further utilizes a parameter neural network to generate a local appearance feature tensor, which includes spatially varying scaling and shifting tensors. In other words, the scene-based image editing system 106 modulates a structural encoding generated from a structure guidance map at the human inpainting GAN. The two-dimensional modulation aids in generating accurate and high-quality human inpainted digital images.
Further, in one or more embodiments, the scene-based image editing system 106 generates human inpainted digital images by utilizing hierarchical encoders. For example, the scene-based image editing system 106 utilizes a hierarchical encoder to generate the structural encoding and the visual appearance encoding. In particular, the scene-based image editing system 106 utilizes the hierarchical encoder which includes multiple downsampling and upsampling layers connected via skip connections (e.g., skip connections for matching resolutions between various layers). By utilizing the hierarchical encoder, the scene-based image editing system 106 can preserve nuance and details when generating the human inpainted digital images.
Moreover, as mentioned, in one or more embodiments the scene-based image editing system 106 utilizes structure guidance map(s) for generating the structural encoding. For example, the structure guidance maps include at least one of a keypoint map, a densepose map, a segmentation map, or a digital image. In particular, in some embodiments, the scene-based image editing system 106 utilizes a separate neural network branch to generate a segmentation map of a digital image for further use as the structure guidance map. For instance, the scene-based image editing system 106 generates an infill segmentation map of the digital image portraying the human with an unclassified region corresponding to the region to inpaint.
Furthermore, in other embodiments, the scene-based image editing system 106 utilizes multiple machine learning models for human inpainting and background inpainting. For example, the scene-based image editing system 106 utilizes a background GAN in addition to a human inpainting GAN. In particular, the scene-based image editing system 106 generates a modified background portion of the digital image utilizing a background GAN and generates modified pixels of the region of the human portrayed within the digital image with a human inpainting GAN. k
Moreover, the scene-based image editing system 106 can train a human inpainting GAN. For example, in some implementations, the scene-based image editing system 106 utilizes a combination of partial reconstruction loss and adversarial loss for training the human inpainting GAN. Indeed, because occluder images often fail to include underlying ground truth information behind the occlusion, the scene-based image editing system 106 can utilize a partial reconstruction loss where the measure of loss is based on the portions of the human outside of the region to inpaint.
As mentioned above, the scene-based image editing system 106 performs acts for generating modified digital images. For example, the scene-based image editing system 106 utilizes multiple machine learning model instances to generate a modified digital image by performing partial human inpainting. In particular,
As shown,
As shown, for the input digital image 5300, the scene-based image editing system 106 determines a region to inpaint corresponding with the depiction of the human. In particular, the region of the human includes a portion of the input digital image 5300 that either only covers a part of the depicted human or a portion of the input digital image 5300 that overlaps with part of the depicted human and another part of the input digital image 5300. Specifically, the input digital image 5300 depicts a human and also depicts the human holding an object, the region of the human to inpaint in this instance includes the object held by the human.
As also shown,
Moreover, as further shown,
In one or more embodiments, the scene-based image editing system 106 segments the input digital image 5300 utilizing a segmentation model. In particular, the scene-based image editing system 106 utilizes a segmentation image neural network (e.g., segmentation machine learning model 4404 discussed in
Conventional systems have leveraged recent computing advancements to modify digital images utilizing a variety of digital tools and models. To illustrate, conventional systems utilize computer-implemented models to inpaint digital images. Despite these advancements, however, conventional systems continue to suffer from a number of technical deficiencies, particularly with regard to accuracy, efficiency, and flexibility of implementing systems in generating inpainted digital images.
For example,
The scene-based image editing system 106 can improve accuracy relative to conventional systems. For example, in one or more implementations the scene-based image editing system 106 improves upon the accuracy of conventional systems by utilizing a local appearance feature tensor (e.g., spatially variant parameters) to generate smooth, natural, realistic, and accurate human inpainting. Unlike conventional machine learning models that merely utilize global features of a digital image (e.g., feature representations that do not change or vary across pixels of the digital image), the use of the local appearance feature tensor can add more nuance, depth, and details to generating the fills. Accordingly, in some implementations the scene-based image editing system 106 more accurately generates inpainted digital images.
Moreover, the scene-based image editing system 106 can improve upon accuracy of conventional systems by utilizing the human inpainting GAN 5302. In particular, in one or more implementations the scene-based image editing system 106 utilizes the human inpainting GAN 5302 which is trained specifically for performing human inpainting. Thus, unlike conventional systems which are generally trained for filling in regions, the scene-based image editing system 106 can utilize a specifically trained and specialized generative model for inpainting humans portrayed within digital images.
Additionally, the scene-based image editing system 106 can improve upon accuracy by utilizing structure guidance maps. Specifically, the scene-based image editing system 106 utilizes structural guidance maps, such as keypoint maps, pose maps, or segmentation maps to inform the human inpainting GAN 5302 to more accurately generate modified digital images.
In addition to improving accuracy, the scene-based image editing system 106 improves upon the efficiency of conventional systems. For example, the scene-based image editing system 106 reduces the need for excessive user interactions for generating modified digital images. Indeed, by intelligently and accurately inpainting regions of a digital image, the scene-based image editing system 106 reduces user interactions and user interfaces needed to remove occluders, correct digital images, and generate accurate portrayals of human subjects.
As illustrated,
As shown in
In addition to generating the structural encoding 5410,
As shown,
In some embodiments, the scene-based image editing system 106 utilizes multiple encoders. For example,
As further shown,
As mentioned above, in some implementations the scene-based image editing system 106 generates a local appearance feature tensor to improve the accuracy of human inpainting. For example,
In particular,
In some embodiments, the scene-based image editing system 106 utilizes a pose map. In particular, a pose map includes a dense pose representation. For instance, in generating the pose map, the scene-based image editing system 106 maps known pixels of the digital image 5504 to a 3-dimensional surface of a human body. The pose map includes a dense keypoint sequence representation of the digital image 5504. In some embodiments, the scene-based image editing system 106 generates the dense keypoint sequence representation utilizing a pose estimation algorithm such as DensePose. For example, the scene-based image editing system 106 can utilize a pose estimation algorithm that maps pixels portraying a human within a digital image to a 3D surface of the human body (e.g., by quantizing UV values).
Furthermore, in one or more embodiments, the scene-based image editing system 106 utilizes segmentation maps (such as a semantic map as discussed above). For example, the segmentation map utilized as the structure guidance map 5502 includes the depiction of the human within the digital image 5504 segmented according to various classifications. For example, the segmentation map includes a segmentation between the background and the human. In particular, the segmentation map further segments sub-portions of the human such as a jacket the human is wearing, pants, shoes, head, and hair.
Moreover, in some embodiments, the scene-based image editing system 106 utilizes the digital image 5504 (or a portion of the digital image 5504) as the structure guidance map 5502. For example, the scene-based image editing system 106 utilizes the digital image 5504 that depicts the human as the structure guidance map 5502. In particular, the digital image 5504 contains structural information such as the boundary of the human and the various sub-portions of the human reflected by different color shades within the digital image 5504. As such, the scene-based image editing system 106 utilizes the human from the digital image 5504 as the structure guidance map 5502.
As further shown, the scene-based image editing system 106 utilizes an encoder 5506 to generate a structural encoding 5508 and a visual appearance encoding 5510 from the structure guidance map 5502 and the digital image 5504. Specifically, the scene-based image editing system 106 generates the structural encoding 5508 and the visual appearance encoding 5510 for each resolution of the digital image 5504. More details relating to generating the structural encoding 5508 and the visual appearance encoding 5510 are given below in the description of
Additionally,
For example, a feature tensor includes a multi-dimensional array of numerical values that represent features or characteristics of underlying data (such as a digital image, structural representation, and/or visual appearance representation). The scene-based image editing system 106 utilizes a feature tensor for modulating style blocks within a human inpainting GAN.
In one or more embodiments, the human inpainting GAN utilizes a StyleGAN or StyleGAN2 architecture (as discussed and incorporated above with regard to StyleGAN2). For instance, the scene-based image editing system 106 utilizes convolutional layers (e.g., multiple fully connected convolutional layers) to convert a digital image into intermediate latent vectors (e.g., w vectors in a W or W+ space). In turn, the scene-based image editing system 106 converts the Intermediate latent vectors into latent style vectors utilizing a series of learned transformations.
The scene-based image editing system 106 converts style vectors into a feature representation utilizing a series of layers referred to as style blocks. In particular, the scene-based image editing system 106 utilizes a first style block to process a first style vector. The first style block generates a first intermediate feature vector and passes the first intermediate feature vector to a second style block. The second style block processes the first intermediate feature vector and the second style vector to generate a second intermediate feature vector. The second style block generates the second intermediate feature vector. Thus, the scene-based image editing system 106 iteratively utilizes the style blocks to generate a series of intermediate feature vectors. The scene-based image editing system 106 can also modulate these style blocks by combining additional representations with the style vectors/and/or intermediate feature vectors.
As mentioned above, the scene-based image editing system 106 generates the local appearance feature tensor 5516. For example, the local appearance feature tensor 5516 includes a scaling and/or a shifting tensor. In other words, depending on the pixel value location of the digital image 5504, the scene-based image editing system 106 utilizes a different scaling and shifting tensor to modulate the feature representations for a particular style block. Accordingly, the scene-based image editing system 106 utilizes the local appearance feature tensor 5516 from the visual appearance encoding 5510 that corresponds to a specific resolution of a particular style block to modulate (scale and shift) the feature representations of the human inpainting GAN 5514. Moreover, as shown, the scene-based image editing system 106 via the human inpainting GAN 5514 generates the modified digital image 5518.
For example, in one or more embodiments, the scene-based image editing system utilizes a scaling tensor and shifting tensor to perform 2D modulation as described by B. AlBahar, J. Lu, J. Yang, Z. Shu, E. Shechtman, and J. Huang in Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGAN, SIGGRAPH Asia 2021 (Pose with Style), which is incorporated by reference herein in its entirety. Unlike Pose with Style, in one or more embodiments, the scene-based image editing system 106 performs modulation with regard to different features and inputs, with a different architecture, and with regard to generating different results.
As mentioned above,
Although the description above illustrates the scene-based image editing system 106 utilizing a single structure guidance map, in one or more embodiments, the scene-based image editing system 106 utilizes multiple structure guidance maps. In particular, the scene-based image editing system 106 generates the structural encoding 5508 from a combination of a keypoint map and a pose map. In other instances, the scene-based image editing system 106 generates the structural encoding from a combination of the segmentation map and a digital image. Moreover, the scene-based image editing system 106 can utilize various combinations of the available structure guidance maps to generate the structural encoding 5508.
As mentioned above,
As shown in
For example, upsampling includes moving from a lower resolution to a higher resolution corresponding with the style blocks of the human inpainting GAN. In particular, prior to moving to a next upsampling layer, the scene-based image editing system 106 utilizes the aforementioned skip connections to receive the encodings from a corresponding down sampling layer. In response to receiving the encodings from the downsampling layers 5604 via skip connections, the scene-based image editing system 106 combines the encodings. Specifically, the scene-based image editing system 106 concatenates the encodings from the corresponding downsampling layer and upsampling layer. The scene-based image editing system 106 then passes the combined encoding to the next upsampling layer and the process repeats.
As
As mentioned previously, the hierarchical encoder generates an encoding for a variety of resolutions (i.e., each resolution of the channels/style blocks of the human inpainting GAN). As illustrated,
As an example,
As shown in
Similarly, the scene-based image editing system 106 can modulate a third region of a second intermediate feature vector of a second style block (corresponding to a second resolution) utilizing a third scaling value and a third shifting value. Moreover, the scene-based image editing system 106 can modulate a fourth region of the second intermediate feature vector of the second style block utilizing a fourth scaling value and a fourth shifting value.
As mentioned above,
For example, as shown, the scene-based image editing system 106 utilizes the human inpainting GAN with multiple style blocks. In particular,
Modulation in a neural network refers to the process of adjusting the strength or influence of certain inputs or pathways in the network. This can be done by adjusting the weights or biases of specific neurons or groups of neurons. Modulation can be used to improve the performance of a network, such as by increasing the sensitivity of certain neurons to important features in the input data, or by reducing the influence of noise or irrelevant information. It can also be used to control the behavior of a network, such as by adjusting the strength of certain pathways to control the output of the network. For example, in
As mentioned above, for the human inpainting GAN, each style block contains multiple layers (e.g., the modulation layer, the convolutional layer, and the demodulation layer). The scene-based image editing system 106 modulates a first vector at the modulation layer, passes the modulated structural encoding to the convolutional layer, then to the demodulation layer (or normalization layer) to generate the intermediate vector 5810. Indeed, in one or more implementations, the scene-based image editing system 106 modulates the mean and standard deviation of the features (e.g., the style vectors) using the scaling and shifting tensors (at the modulation layer), applies the convolutional layer, and normalizes (at the demodulation layer) the output of the convolution to zero mean and unit standard deviation before adding the bias and StyleGAN 2 noise broadcast operation. In some embodiments, the scene-based image editing system 106 utilizes the modulation approach described in Pose with Style.
As further shown, the scene-based image editing system 106 utilizes the human inpainting GAN at the second style block 5812 to generate an additional intermediate vector 5814 from the intermediate vector 5810 (and/or style vectors analyzed/processed by each style block but unshown in
As mentioned previously, the scene-based image editing system 106 generates segmented digital images (e.g., segmented digital image 4406 discussed in the description of
Furthermore, as shown,
As just mentioned, the scene-based image editing system 106 generates the initial segmentation map. However, due to a hole or occlusion of the human portrayed within the digital image 5902, the segmentation map includes an unclassified region. In particular, the unclassified region corresponds to the region of the human to inpaint. The scene-based image editing system 106 utilizes a generative semantic machine learning model (e.g., generative semantic machine learning model 4006 discussed in the description of
Accordingly, the scene-based image editing system 106 utilizes a generative model specifically for segmentation maps to generate a realistic and accurate infill segmentation for the determined region to inpaint. For the infill segmentation map 5908 the scene-based image editing system 106 assigns a label to various pixels within the digital image 5902. Specifically, the scene-based image editing system 106 assigns labels to every pixel within the digital image 5902. For instance, the scene-based image editing system 106 assigns a label to pixels in a manner that groups pixels together that share certain characteristics (e.g., background portion or foreground portion). Furthermore, the infill segmentation map 5908 assigns labels to the region determined to be inpainted. Accordingly, various models utilize the infill segmentation map 5908 to determine how to infill the region indicated to be inpainted.
For example, as mentioned, the scene-based image editing system 106 generates the infill segmentation map and, for the region determined for inpainting, the scene-based image editing system 106 assigns a human segmentation classification. In particular, the inpainting of a region corresponds with a human portrayed within the digital image 5902, as such, the region for inpainting includes a human segmentation classification. Specifically, the human segmentation classification corresponds with a sub-portion of the human such as the hands, feet, arms, legs, torso, or head. Further as shown, the scene-based image editing system 106 utilizes an encoder 5912 that generates outputs based on the infill segmentation map 5908 and the digital image 5902. In particular, the scene-based image editing system 106 further utilizes an output of the encoder 5912 and via a human inpainting GAN 5914, the scene-based image editing system 106 generates a modified digital image 5916.
As mentioned earlier, in one or more embodiments the scene-based image editing system 106 utilizes a human inpainting GAN and background GAN.
The scene-based image editing system 106 utilizes the modified background portion of the input digital image 6002 to generate the modified digital image 6014. The background GAN 6010 includes a GAN that generates a modified background portion of a digital image. In particular, the background GAN 6010 replaces or adds pixel values to the background portion 6004 to generate a realistic-looking background portion 6004. For example, in one or more embodiments, the scene-based image editing system 106 utilizes a cascaded modulation inpainting neural network (or another architecture such as CoModGAN) as the background GAN 6010.
In addition to determining the background portion 6004, the scene-based image editing system 106 also determines a human portion 6006 of the input digital image 6002. For example, the scene-based image editing system 106 determines the human portion 6006 and generates modified pixels of the region corresponding to the human. In particular, the scene-based image editing system 106 utilizes a human inpainting GAN 6012 to generate the modified pixels for the human portion 6006 portrayed within the input digital image 6002. As shown, the scene-based image editing system 106 utilizes a combination of the modified background portion and the modified human portion to generate the modified digital image 6014. Thus, as just discussed in some embodiments, the scene-based image editing system 106 utilizes separate generative models to modify the background portion 6004 and the human portion 6006.
The scene-based image editing system 106 can utilize a variety of approaches to combine the background and human portions of a digital image generated by the human inpainting GAN 6012 and the background GAN 6010. In some implementations, the scene-based image editing system 106 masks an indicated object for removal, inpaints background pixels utilizing the background GAN 6010, and inpaints human pixels (for the hole or infill portion) utilizing the human inpainting GAN 6012. In particular, the scene-based image editing system 106 inpaints the remaining background pixels independent of the pixels corresponding to the human inpainted region.
In some circumstances, this approach can lead to bleeding of the human region into the background pixels. In one or more embodiments, the scene-based image editing system 106 utilizes a different approach to account for this potential bleeding. For example, in one or more implementations, the scene-based image editing system 106 utilizes the human inpainting GAN 6012 to complete the human pixels, but segments and removes the human from the digital image. The scene-based image editing system 106 utilizes the background GAN 6010 to inpaint background pixels (without the human in the digital image), and then reinserts the human within the digital image.
To illustrate, the scene-based image editing system 106 generates the modified digital image 6014 by generating an intermediate digital image. For instance, the scene-based image editing system 106 generates the intermediate digital image by removing the region of the human in the input digital image 6002. Specifically, the scene-based image editing system 106 masks the region corresponding with the human and removes the masked region. Furthermore, for the intermediate digital image, the scene-based image editing system 106 inpaints the remaining portion which includes the background portion 6004 utilizing the background GAN 6010. In doing so, the intermediate digital image contains the background portion 6004 with the human portion 6006 temporarily removed.
In addition, the scene-based image editing system 106 generates modified pixels for the region corresponding to the human portion 6006 (e.g., the human segmentation classification) by utilizing the human inpainting GAN 6012. For example, the scene-based image editing system 106 generates modified pixels for the human portion 6006 separately and inserts the modified pixels corresponding to the human portion 6006 into the intermediate digital image. In doing so, the scene-based image editing system 106 generates the modified digital image 6014.
In some circumstances, this approach can also lead to artifacts along the boundary of the human and background pixels, particularly where the segmentation is not precise. As shown in
In one or more embodiments, the scene-based image editing system utilizes the human inpainting GAN 6012 to generate modified pixel values corresponding to the human portion 6006 and the overlapping portion 6008. Further, the scene-based image editing system 106 utilizes the background inpainting GAN on the background portion 6004. In particular, the scene-based image editing system 106 performs background inpainting for the background portion 6004 based on the human inpainted pixels (e.g., the scene-based image editing system 106 utilizing the human inpainting GAN 6012 to generate modified pixel values corresponding to the human portion 6006 and the overlapping portion 6008). Thus, the combination of the inpainted background portion, the inpainted human portion, and the inpainted overlapping portion generates the modified digital image 6014. In other words, the scene-based image editing system 106 first inpaints the pixels corresponding to the human portion 6006 and then the scene-based image editing system 106 inpaints the background portion 6004 conditioned on the inpainting performed for the human portion 6006.
As mentioned, the scene-based image editing system 106 determines various portions of the input digital image 6002. In one or more embodiments, the scene-based image editing system 106 masks the various determined portions of the input digital image 6002. For example, the scene-based image editing system 106 generates masks for the region of the human portion 6006 and a mask for the background portion 6004. In particular, the mask includes segmenting a portion of the input digital image 6002. For instance, the scene-based image editing system 106 generates a mask by utilizing a segmentation machine learning model to identify pixels corresponding to an indicated region (e.g., the human portion 6006, the background portion 6004, and/or the overlapping portion 6008).
As just mentioned, the scene-based image editing system 106 utilizes masks for the various portions of the input digital image 6002. In particular, the scene-based image editing system 106 first determines the human portion 6006 and the overlapping portion 6008 (the human and the background overlapping), generates masks, and inpaints the masked human portion and the masked overlapping portion utilizing the human inpainting GAN 6012. Subsequently, the scene-based image editing system 106 inpaints the masked background portion utilizing the background GAN 6010. Moreover, the scene-based image editing system 106 combines the inpainted masked background portion with the inpainted masked human portion and masked overlapping portion to generate the modified digital image 6014.
As shown,
For example,
Reconstruction loss refers to a measure a similarity or fidelity between a digital image and a generated digital image. For example, a reconstruction loss includes the degree of closeness for a decoder output relative to the original input. In one or more implementations, the scene-based image editing system 106 determines a reconstruction loss utilizing a loss function, such as a mean-squared error or other measure of loss.
In one or more embodiments, the scene-based image editing system 106 determines the partial reconstruction loss 6120 by focusing on a particular portion of the digital image. For example, in one or more implementations, the scene-based image editing system 106 determines the partial reconstruction loss 6120 between the region of the digital image 6104 outside of the hole/occlusion. For instance, the scene-based image editing system 106 in utilizing the human inpainting GAN 6108, inpaints a portion of the digital image 6104. In doing so, the scene-based image editing system 106 determines a reconstruction loss between the non-inpainted portion and the corresponding portion (non-inpainted portion) of the digital image 6104. Thus, as illustrated, the scene-based image editing system 106 compares the modified digital image 6110 with a digital image with a hole 6116 to generate the partial reconstruction loss 6120.
Upon determining the partial reconstruction loss 6120, the scene-based image editing system 106 utilizes the partial reconstruction loss to train the human inpainting GAN 6108. For example, the scene-based image editing system 106 modifies parameters of the human inpainting GAN 6108 to reduce the partial reconstruction loss 6120. In one or more implementations, the scene-based image editing system 106 back propagates the partial reconstruction loss 6120 to the human inpainting GAN 6108.
As further shown, the scene-based image editing system 106 also determines the adversarial loss 6114. As mentioned previously, a discriminator 6112 and generative model attempt to generate a realistic-looking digital image. For example, the scene-based image editing system 106 determines adversarial loss for a generative model. In particular, as mentioned, the adversarial loss includes the generative model and the discriminator 6112 attempting to trick one another in a zero-sum game. As shown in
As mentioned above, the scene-based image editing system 106 utilizes various structure guidance maps in accordance with one or more embodiments. For example,
In one or more embodiments, the scene-based image editing system 106 intelligently utilizes the features discussed in
In some implementations, the scene-based image editing system 106 receives a user selection of an area, occluder, or object (or a user input expanding a frame) and utilizes the human inpainting GAN 6108 to inpaint a human portrayed in the digital image. Thus, based on a single user interaction, the scene-based image editing system 106 can generate a modified digital image that more accurately inpaints a portion of a human.
In some embodiments, the scene-based image editing system 106 detects the human and an obstruction/occlusion within a digital image and provides recommendations. For example, the scene-based image editing system 106 provides various recommendations to remove detected obstructions/occlusions and provides an option for a user of a client device to select the provided recommendations. In particular, the scene-based image editing system 106 can also provide a generated sample of various objects removed from a digital image and inpainting of a human and/or background.
Moreover, in some embodiments, the scene-based image editing system 106 intelligently detects an upload of a digital image portraying a human in addition to an existing digital image being edited within an image editing application. In particular, the scene-based image editing system 106 receives the upload of the digital image portraying a human and automatically removes objects (e.g., objects occluding or obstructing the human) and performs human inpainting. Moreover, in this instance, the scene-based image editing system 106 can also perform background inpainting to conform with the already existing digital image. Furthermore, in response to receiving additionally uploaded digital images, the scene-based image editing system 106 performs human inpainting for humans portrayed within the existing digital image being edited within the image editing application.
As also briefly mentioned above, in one or more embodiments the scene-based image editing system 106 generates modified digital images by reposing subjects portrayed within a digital image. For example, the scene-based image editing system 106 reposes human subjects (or other objects) portrayed in a source digital image according to a target pose. In particular, the scene-based image editing system 106 utilizes a human (or object) portrayed within the source digital image to depict the human (or object) with a different pose. For instance, the scene-based image editing system 106, in one or more embodiments, utilizes a global texture map appearance vector and a local appearance feature tensor. Moreover, the scene-based image editing system 106 combines the global texture map appearance vector with the local appearance feature tensor and utilizes a reposing model to generate a modified digital image.
As just mentioned, in one or more embodiments the scene-based image editing system 106 combines a global texture map appearance vector and a local appearance feature tensor. Specifically, the scene-based image editing system 106 combines these representations to generate a globally modified local appearance feature tensor which includes spatially variant information and spatially invariant information. Furthermore, the scene-based image editing system 106 utilizes the globally modified local appearance feature tensor to modulate a reposing model, such as a reposing GAN. Accordingly, modulating the reposing GAN results in the scene-based image editing system 106 generating a high-quality and nuanced reposing of the human portrayed within the source digital image.
Furthermore, in one or more embodiments the scene-based image editing system 106 generates a warped digital image from a texture map and a target pose map. In particular, the scene-based image editing system 106 further generates a warped pose feature map from the warped digital image. For instance, the scene-based image editing system 106 generates vectors utilizing the warped digital image and synthesizes a modified digital image based on the warped pose feature map. Moreover, in one or more embodiments the scene-based image editing system 106 utilizes a coordinate inpainting GAN to generate an inpainted warped digital image. The inpainted warped digital image can further assist the scene-based image editing system 106 in generating high-quality reposing of humans portrayed within digital images.
In addition, in one or more implementations the scene-based image editing system 106 trains the reposing GAN by utilizing an unpaired image dataset. For example, the scene-based image editing system 106 generates incomplete digital images and synthesized inpainted digital images from the incomplete digital images and target poses. Based on the synthesized inpainted digital images, the scene-based image editing system 106 determines a measure of loss to modify parameters of the reposing GAN. Moreover, the scene-based image editing system 106 generates masks for different portions of a modified digital image. In particular, the scene-based image editing system 106 generates masks for the body and/or for sub-parts of the body to determine different loss weights. For instance, the scene-based image editing system 106 modifies parameters of the reposing GAN based on the various loss weights.
For example,
In addition,
For instance, the scene-based image editing system 106 can generate the target pose map 6300 by mapping UV coordinates from a digital image that depicts a human to represent the structural pose of the human. Furthermore, the target pose map 6300 represents rendered dense body UV coordinates. Specifically, the scene-based image editing system 106 utilizes a parametric 3D body model to render the dense UV of the body captured in a target digital image. For example, the scene-based image editing system 106 utilizes the parametric 3D body model to map the human appearance within a digital image to a UV space. In particular, the 3D body model utilized by the scene-based image editing system 106 includes a set of interconnected 3D meshes for defining body shape and pose. The scene-based image editing system 106 can utilize a variety of computer-implemented models. For instance, in one or more embodiments the scene-based image editing system 106 generates the target pose map 6300 by utilizing the methods described by Kripasindhu Sarkar, Vladislav Golyanik, Lingjie Liu, and Christian Theobalt in Style and pose control for image synthesis of humans from a single monocular view, 2021, which is incorporated in its entirety by reference herein.
Moreover, in some embodiments, the scene-based image editing system 106 receives an upload of a digital image from the user of the client device and generates the target pose map 6300 from the uploaded digital image. In other embodiments, the scene-based image editing system 106 provides pre-defined target poses and in response to a selection of a pre-defined target pose, the scene-based image editing system 106 utilizes a corresponding target pose map with the selected pre-defined target pose.
Furthermore,
As shown in
As shown, the reposing model 6306 generates the modified digital image 6312, which portrays the human within the source digital image 6302 reposed according to the target pose map 6300. As compared to conventional systems, the scene-based image editing system 106 improves upon accuracy and efficiency in generating modified digital images, specifically reposing subjects portrayed within digital images. Indeed, conventional systems suffer from a number of technical deficiencies, particularly with regard to accuracy and efficiency in performing reposing in a digital image.
For instance, conventional systems can repose subjects shown in a digital image, however conventional systems often generate inaccurate and unrealistic results. In particular, conventional systems performing reposing generally suffers from a lack of nuance and detail. Specifically, conventional systems utilize global feature modifications which fail to accurately and efficiently capture the details for reposing a subject within a digital image. In addition, conventional systems typically require excessive user interactions and user interfaces for a user of a client device to accurately generate a reposed subject within a modified digital image.
The scene-based image editing system 106 can improve upon accuracy of conventional systems by utilizing both global and local feature representations. For example, in one or more embodiments the scene-based image editing system 106 utilizes the global texture map appearance vector 6310 and the local appearance feature tensor 6308 to generate the to generate a more accurate and realistic digital image. For instance, the scene-based image editing system 106 modulates the reposing model 6306 to capture the nuance and details of reposing the subject from the source digital image 6302.
Furthermore, in one or more embodiments, the scene-based image editing system 106 improves accuracy by utilizing a warped digital image generated from a texture map and a target pose map. In particular, the scene-based image editing system 106 rearranges the pixels from the source image with the spatial structure provided by the target to generate the warped image. The system can combine the target pose and the warped image as input to focus on the detail hallucination instead of synthesizing the 2D appearance based on appearance code alone.
Additionally, the scene-based image editing system 106 can improve upon accuracy by utilizing an inpainted warped digital image. For example, the scene-based image editing system 106 utilizes a coordinate inpainting GAN for inpainting a warped digital image. In one or more implementations, the scene-based image editing system 106 utilizes the coordinate inpainting GAN to address holes and missing parts in the projected texture map.
As mentioned above, the scene-based image editing system 106 can also improve efficiency. For example, as mentioned above, the scene-based image editing system 106 eliminates excessive user interactions and user interfaces for performing reposing of a subject within a digital image. Specifically, the scene-based image editing system 106 incorporates the nuance and details from the target pose map 6300, the source digital image 6302, and the texture map 6304 to automatically and efficiently generate a high-quality and accurate modified digital image.
As mentioned above,
In one or more embodiments, the pose encoder 6406 includes a hierarchical pose encoder. In particular, the hierarchical pose encoder includes multiple layers to encode the target pose map 6400 and the source digital image 6404 in a hierarchical manner. Each layer of the hierarchical pose encoder extracts a different level of abstraction from the target pose map 6400 and the source digital image 6404. Specifically, the hierarchical pose encoder extracts features, passes the features to the next layer until generating the pose feature map 6412.
Moreover, as discussed above, the hierarchical pose encoder includes downsampling layers and upsampling layers. Specifically, the scene-based image editing system 106 utilizes the hierarchical pose encoder to process the target pose map 6400 and the source digital image 6404 via a downsampling layer of the hierarchical pose encoder. Further, the scene-based image editing system 106 via the downsampling layer generates an intermediate downsampled feature vector with a resolution corresponding to that downsampling layer. Furthermore, the scene-based image editing system 106 generates utilizing an additional downsampling layer of the hierarchical pose encoder, an additional intermediate downsampled feature vector corresponding to an additional resolution.
In addition, to the downsampling layers, the hierarchical pose encoder also utilizes an upsampling layer to generate an intermediate upsampled feature vector with a resolution corresponding to that upsampled layer. Further, the scene-based image editing system 106 utilizes skip connections between the downsampling layer and the upsampling layer to generate a combined feature vector from the intermediate downsampled feature vector and the intermediate upsampled feature vector. Moreover, the scene-based image editing system 106 generates utilizing an additional upsampling layer of the hierarchical pose encoder from the combined feature vector (of the intermediate downsampled feature vector and the intermediate upsampled feature vector) an additional intermediate upsampled feature vector corresponding to an additional resolution. Furthermore, via skip connections, the scene-based image editing system 106 combines the additional intermediate downsampled feature vector with the additional intermediate upsampled feature vector to generate the pose feature map 6412. Indeed, in one or more embodiments, the scene-based image editing system 106 utilizes the architecture illustrated in
For example, as mentioned above, the scene-based image editing system 106 generates the pose feature map 6412. In particular, the pose feature map 6412 includes an encoding, embedding, or feature representation of the source digital image 6404 and the target pose map 6400. For instance, the pose feature map 6412 includes an indication of the different features found within the source digital image 6404 and the target pose map 6400. Accordingly, because the scene-based image editing system 106 generates the pose feature map 6412 from the target pose map 6400 and the source digital image 6404, the pose feature map 6412 contains feature of both the target pose map 6400 and the source digital image 6404.
Furthermore,
As shown in
Moreover, although not shown, the scene-based image editing system 106 passes the local appearance vector generated from the image appearance encoder 6408 to a parameters neural network 6414. For example, in one or more embodiments, the scene-based image editing system 106 utilizes the parameter neural network described in relation to
Furthermore,
In particular, the global texture map appearance vector 6416 includes a spatially invariant vector. In other words, the global texture map appearance vector 6416 globally applies to the entire source digital image 6404. In contrast to the local appearance feature tensor 6418, as mentioned, the global texture map appearance vector 6416 includes a spatially invariant vector. Specifically, the global texture map appearance vector 6416 does not change depending on the location of the pixel value within the source digital image 6404 (or feature representations of the source digital image 6404. Thus, the global texture map appearance vector 6416 includes a one-dimensional appearance vector.
In addition to generating the global texture map appearance vector 6416 from the texture map appearance encoder 6410,
Moreover, although
As mentioned,
As shown, the scene-based image editing system 106 utilizes a texture map model 6508 to generate a texture map 6510 from the pose map 6504 of the source digital image 6500 and the source digital image 6500. Specifically, the texture map model 6508 generates the texture map 6510 by utilizing pixel values from the source digital image 6500 and the pose map 6504 of the source digital image 6500. In other words, the scene-based image editing system 106 generates the texture map 6510 by projecting the pixel values of the source digital image 6500 guided by the pose map 6504 of the source digital image 6500. For instance, in some implementations, the scene-based image editing system 106 utilizes the methods disclosed in Style and Pose Control for Image Synthesis of Humans from a Single Monocular View, 2021 which was mentioned above. Furthermore, in some embodiments, the scene-based image editing system 106 utilizes the methods discussed in Pose With Style for generating texture maps.
Additionally, in one or more implementations, the scene-based image editing system 106 generates the texture map 6510 utilizing VGG normalization. For instance, prior to projecting the pixel values of the source digital image 6500 guided by the pose map 6504, the scene-based image editing system 106 normalizes the source digital image 6500 to match a distribution used when training a VGG network, e.g., VGG Net described by Karen Simonyan and Andrew Zisserman in Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014 which is incorporated in its entirety by reference herein.
As mentioned, in one or more embodiments, the scene-based image editing system 106 modulates the reposing GAN. For example,
As further shown, the scene-based image editing system 106 combines a global texture map appearance vector 6620 with the local appearance feature tensor 6604. For instance, in one or more implementations, the scene-based image editing system 106 concatenates the global texture map appearance vector 6620 (i.e., a 1D appearance vector) to each location of the local appearance feature tensor 6604 (i.e., a 2D appearance tensor) to pass the global information of the texture map into the generator. For example, by combining the global texture map appearance vector 6620 and the local appearance feature tensor 6604, the scene-based image editing system 106 generates a globally modified local appearance feature tensor 6606. In particular, the scene-based image editing system 106 modulates the reposing GAN with the globally modified local appearance feature tensor 6606. The globally modified local appearance feature tensor 6606 includes spatially variant information and spatially invariant information for modulating the pose feature map at each block of the reposing GAN.
As mentioned above (e.g., with regard to
Furthermore, the scene-based image editing system 106 utilizes as input a corresponding resolution of the second style block 6612 and modulates the corresponding feature representations (e.g., style vector or other vector representations) of the second style block 6612 with the globally modified local appearance feature tensor 6606. As shown, for the second style block 6612, the scene-based image editing system 106 utilizes a different scaling and shifting tensor at the second style block 6612 than at the first style block 6608. Similar to the above, the corresponding resolution of the second style block 6612 passes through the modulation layer, convolutional layer, and demodulation layer to generate an additional intermediate vector 6614. As similarly shown, the scene-based image editing system 106 also utilizes at a third style block 6616 a corresponding resolution of the third style block 6616 for modulation by the globally modified local appearance feature tensor 6606. Similarly, the scene-based image editing system 106 via the third style block 6616 generates an additional intermediate vector 6618.
As mentioned earlier, in one or more embodiments the scene-based image editing system 106 also utilizes a warped digital image. For example,
In particular,
Furthermore,
As shown,
Although not shown in
The coordinate inpainting generative neural network can learn correlations between different UV positions. During run time, the missing pixels in the texture map can be synthesized from the visible pixels using such correlation prior. The scene-based image editing system 106 can utilize a variety of coordinate inpainting models. In one or more implementations, the scene-based image editing system 106 utilizes a coordinate inpainting model as described by Artur Grigorev, Artem Sevastopolsky, Alexander Vakhitov, and Victor Lempitsky. Coordinate-based texture inpainting for pose-guided image generation, arXiv preprint arXiv:1811.11459, 2018, which is incorporated by reference in its entirety herein).
Accordingly, the scene-based image editing system 106 generates the warped digital image 6706 which includes missing pixel values. As such, the scene-based image editing system 106 utilizes the coordinate inpainting generative neural network to generate an inpainted warped digital image.
As mentioned above, the scene-based image editing system 106 generates the inpainted warped digital image. Because the warped digital image 6706 often includes corrupted or missing parts, the scene-based image editing system 106 utilizes the coordinate inpainting generative neural network to generate an inpainted version of the warped digital image 6706. Furthermore, utilizing the inpainted warped digital image, the scene-based image editing system 106 generates via the image appearance encoder 6710 a warped image feature vector. In particular, the warped image appearance encoder extracts features of the inpainted warped digital image and generates the warped image feature vector. Similar to the discussion above, the scene-based image editing system 106 utilizes the inpainted warped digital image to generate the warped pose feature map 6722 and the local appearance vector 6718 and to further generate the modified digital image 6728.
In addition,
Moreover,
In addition,
Further,
Moreover,
As also shown,
As mentioned above, the scene-based image editing system 106 trains the reposing GAN 6900 utilizing various weights. For example, the scene-based image editing system 106 can apply a weight of 0.1 to the body measure of loss, apply a weight of 0.9 to the body part measure of loss, while applying a weight of 0 for the background measure of loss. The scene-based image editing system 106 can apply a variety of weight combinations (e.g., ⅓, ⅓, ⅓ or 0.5, 0.25, 0.25, or 1, 0, 0).
By applying masks (and weights) during training of the reposing GAN 6900, the scene-based image editing system 106 improves the synthesis quality of the modified digital image 6902. Indeed, the scene-based image editing system 106 can focus training on the body as a whole (e.g., utilizing a body mask), certain sub-parts of the body that are difficult to synthesize (e.g., hands or harms), or other portions of the digital image (e.g., the background). Furthermore, as shown, in one or more embodiments, the scene-based image editing system 106 optionally performs the act 6904 and the act 6906 (e.g., performs different combinations of one or more masks in training). In particular, the scene-based image editing system 106 performs the act of applying a body mask to the modified digital image 6902 and does not perform the act of applying the body part mask to the modified digital image 6902 or vice-versa.
In addition to training the reposing GAN 6900 by applying masks, the scene-based image editing system 106 in
As shown,
Indeed, as shown, the scene-based image editing system 106 compares the synthesized inpainted digital image 7010 and a source digital image 7012 in the unpaired image dataset 7000. In particular, the scene-based image editing system 106 compares the synthesized inpainted digital image 7010 and the source digital image 7012 (e.g., utilizing a loss function) to determine a measure of loss 7014. Furthermore, the scene-based image editing system 106 utilizes the determined measure of loss 7014 (e.g., via back propagation) to modify parameters of the reposing GAN 7008. Similar to above, although not shown in
Although not shown in
Moreover, in one or more embodiments, the scene-based image editing system 106 utilizes wild image datasets which contains no paired digital images or ground truth digital images for comparison. Furthermore, the scene-based image editing system 106 in some instances utilizes paired digital image datasets which includes ground truth digital images in target poses.
Furthermore, as shown,
To illustrate,
Furthermore, as mentioned
Moreover,
In one or more embodiments, the scene-based image editing system 106 intelligently utilizes the features discussed in
Further, in other embodiments, the scene-based image editing system 106 receives an upload of a digital image, detects the uploaded digital image portrays a human and makes intelligent recommendations. For example, the scene-based image editing system 106 provides alternative pose options for the human within the digital image and provides the options to a user of a client device. In particular, the scene-based image editing system 106 indicates to a user of a client device possible options for reposing a human portrayed within an uploaded digital image. Accordingly, the scene-based image editing system 106 can make intelligent adjustments and/or recommendations in real-time or near real-time to a user of a client device (e.g., seconds).
The scene-based image editing system 106 can also match a target pose in generating a composite digital image. For example, consider an initial digital image portraying a human with closed eyes or awkward facial expression. The scene-based image editing system 106 can extract a representation of the human from a second digital image and repose the representation from the second digital image to align to the target pose of the source digital image. The scene-based image editing system 106 can then generate a composite image by adding the modified representation to the initial image.
The scene-based image editing system 106 can also provide a variety of user interfaces for modifying poses. For example, the scene-based image editing system 106 can provide a user interface with a selectable option for selecting a source image. The scene-based image editing system 106 can also provide selectable options for selecting or creating a target pose. For instance, the scene-based image editing system 106 can provide a user interface element for uploading a target image with a target pose.
Similarly, the scene-based image editing system 106 can provide an interactive pose element (e.g., as an overlay to a human in a digital image) that allows a client device to modify the pose element and create a target pose. For instance, the scene-based image editing system 106 can receive a user interaction moving a member or joint to a new location, rotating a pose representation, or making another modification to the interactive pose element. The scene-based image editing system 106 can utilize the user interaction to determine a target pose and generate a modified digital image according to the target pose.
As also briefly mentioned above, in one or more embodiments the scene-based image editing system 106 generates modified digital images by transferring facial expression and pose or animating a static digital image. For example, the scene-based image editing system 106 transfers a facial expression and a pose from a target digital image to a source digital image. In particular, the scene-based image editing system 106 utilizes the face of a human portrayed within the source digital image and modifies the face and pose to match a different facial expression and a different pose. For instance, the scene-based image editing system 106, in one or more embodiments, generates various embeddings (e.g., target facial expression embedding, source shape embedding, and target pose embedding) to utilize within an end-to-end network for generating a modified source digital image.
Further, in one or more embodiments, the scene-based image editing system 106 animates a static source digital image. For example, the scene-based image editing system 106 performs facial animation by animating a static source digital image according to a target digital video. In particular, the scene-based image editing system 106 extracts various embeddings from a target digital video and generates an animation that portrays a source face from the source digital image. Moreover, in some embodiments, the scene-based image editing system 106 utilizes a diffusion neural network to generate the animation that portrays the source face. Additionally, the scene-based image editing system 106 accurately and efficiently performs both the facial expression transfer and the facial animation for a variety of races and ethnicities without a drop-off in quality.
Furthermore, in one or more embodiments, the scene-based image editing system 106 provides to a user of a client device via a user interface a variety of intuitive options for performing facial expression and/or pose transfer and animation. For example, the scene-based image editing system 106 provides via a user interface of a client device a variety of selectable element options such as selecting a source digital image and/or a target digital image. In particular, the scene-based image editing system 106 provides the selectable elements for a user of a client device to upload various images to transfer facial expressions and/or pose or animate static digital images. Additionally, the scene-based image editing system 106 also provides pre-defined elements. For instance, pre-defined elements include already defined target facial expressions (e.g., happy, sad, smile, cry, angry, etc.). In doing so, the scene-based image editing system 106 can receive from a user of a client device a selection of a pre-defined target facial expression or pose to utilize for transferring a facial expression and/or pose, or animating a static source digital image.
As mentioned above, in some embodiments the scene-based image editing system 106 generates a modified source digital image utilizing a facial expression transfer model in accordance with one or more embodiments. For example,
For example, the scene-based image editing system 106 utilizes the source digital image 7302 for facial expression transfer as a starting reference point for modification. In particular, the scene-based image editing system 106 utilizes the source digital image 7302 as the digital image to be modified by the target digital image 7304. Moreover, the scene-based image editing system 106 can receive the source digital image 7302 from a variety of sources, such as a camera roll of a client device, upload from a client device, or a cloud repository of digital images.
Furthermore,
Additionally, as mentioned, the scene-based image editing system 106 utilizes the facial expression transfer model 7308 to process the source digital image 7302 and the target digital image 7304 and generate the modified source digital image 7310. For example, the modified source digital image 7310 includes one or more pixel values that differ from the source digital image 7302. In particular, the scene-based image editing system 106 via the facial expression transfer model 7308 modifies pixels relating to the facial expression portrayed within the source digital image 7302 according to the target digital image 7304. More details relating to the facial expression transfer model 7308 is given below.
Conventional systems have leveraged recent computing advancements to modify digital images utilizing a variety of digital tools and models. To illustrate, conventional systems utilize computer-implemented models to generate digital images with various facial expressions. Despite these advancements, however, conventional systems continue to suffer from a number of technical deficiencies, particularly with regard to accuracy and efficiency in generating digital images with modified facial expressions.
For instance, to generate modified digital images with various facial expressions, conventional systems utilize models that fail to accurately capture the nuance and detail of a facial expression and realistically transfer that facial expression to another digital image. In particular, conventional systems render various intermediate digital images for utilization in transferring a facial expression from one digital image to another. Moreover, conventional systems often require a large number of user interactions and user interfaces for transferring facial expressions.
As mentioned above, the scene-based image editing system 106 can improve upon accuracy of conventional systems. For example, in one or more implementations the scene-based image editing system 106 improves upon the accuracy of conventional systems by generating embeddings and utilizing the generated embeddings to further generate a modified source digital image. In particular, in some embodiments, the scene-based image editing system 106 generates a target facial expression embedding and a source shape embedding. Further, the scene-based image editing system 106 generates a modified source digital image that portrays a target facial expression from the target digital image 7304 by conditioning layers of the facial expression transfer model 7308 with the target facial expression embedding and the source shape embedding.
For instance, the scene-based image editing system 106, unlike conventional systems, can directly utilize generated embeddings for further generating the modified source digital image 7310. Specifically, the scene-based image editing system 106 need not render intermediate digital images to aid in facial expression transfer. In one or more implementations, the scene-based image editing system 106 utilizes an end-to-end network to generate embeddings and utilize the embeddings to further generate the modified source digital image 7310. Moreover, the scene-based image editing system 106 can utilize a combination of various embeddings such as the facial expression embedding, the target pose embedding, and the source shape embedding to condition layers of the facial expression transfer model 7308. The combined embedding further improves upon accuracy in generating modified source digital images.
Furthermore, in some implementations the scene-based image editing system 106 improves upon accuracy of conventional systems by generating an animation that portrays a source face from a source digital image. In particular, the scene-based image editing system 106 extracts various embeddings (e.g., the target facial expression animation embeddings from a target digital video) and generates the animation from the embeddings. Similar to the facial expression transfer, the scene-based image editing system 106 generates the animation utilizing an end-to-end network without rendering intermediate digital images. In doing so, the scene-based image editing system 106 increases the quality and nuance of animations.
In addition to the accuracy improvements, the scene-based image editing system 106 can also improve upon efficiency and functionality. In particular, the scene-based image editing system 106 can provide a user interface to select a source digital image and a target digital image/video for generating modified digital images or animations. For instance, the scene-based image editing system 106 can provide efficient user interface implementations that reduce excessive user interactions and user interfaces while providing improved functionality and flexibility in generating modified digital images or animations from source content.
As mentioned above,
In relation to
Moreover, the source digital image 7402 includes a source pose and a source facial expression. In particular, the source pose includes a position, or an arrangement of a person or face portrayed within the source digital image 7402. Specifically, the source pose includes the position or arrangement of the person's face portrayed within the source digital image 7402. Furthermore, the source facial expression includes the facial expression shown in the original digital image (e.g., the source digital image 7402) for modification by the target facial expression. For instance, the source facial expression includes an expression of the human face (e.g., a combination of facial elements with particular arrangements or features) shown in the source digital image 7402. Furthermore, the scene-based image editing system 106 extracts the source facial expression of the source digital image 7402 to modify the source facial expression by the target facial expression.
As shown, the scene-based image editing system 106 utilizes the three-dimensional encoder 7404 and the three-dimensional encoder 7410. In particular, the three-dimensional encoders extract features from a digital image. For instance, the scene-based image editing system 106 utilizes the three-dimensional encoders to process the source digital image 7402 and/or the target digital image 7408. Furthermore, the three-dimensional encoders map the image pixels to a higher-dimensional vector representation (e.g., an embedding) to capture features portrayed within the source digital image 7402 and the target digital image 7408.
As further shown, the scene-based image editing system 106 utilizes the three-dimensional encoder 7404 to generate a source shape embedding 7406. For example, the scene-based image editing system 106 generates the source shape embedding 7406 from the source digital image 7402. In particular, the scene-based image editing system 106 extracts features from the source digital image 7402 relating to the shape of the face and human portrayed within the source digital image 7402 to generate the source shape embedding 7406.
In addition,
The scene-based image editing system 106 can utilize a variety of three-dimensional machine learning models. In one or more embodiments, the scene-based image editing system 106 utilizes one or more three-dimensional morphable models (e.g., to generate 3D morphable model parameters). For instance, in some implementations, the scene-based image editing system 106 utilizes a 3D morphable model, as described by V. Blanz and T. Vetter in A Morphable Model for the Synthesis of 3D Faces, SIGGRAPH '99 (187-194) or by J. Booth, A. Roussos, E. Ververas, E. Antonakos, S. Ploumpis, Y. Panagakis, and S. Zafeiriou in Lage Scale 3D Morphable Models, Int. J. Comput. Vision, 126 (2-4):233-254 (2018), which are incorporated by reference herein in their entirety. In some embodiments, the scene-based image editing system 106 utilizes detailed expression capture and animation model (e.g., DECA) as described by Yao Feng, Haiwen Feng, Michael J. Black, Timo Bolkar in Learning an Animatable Detailed 3D Face Model from In-The-Wild Images, arXiv:2012.04012v2, which is incorporated by reference herein in its entirety.
As mentioned,
As also mentioned previously, the scene-based image editing system 106 generates a combined embedding. For example,
Furthermore, although not shown in
Although
To further illustrate, for a second resolution (corresponding to a convolutional layer), the scene-based image editing system 106 generates a second source shape embedding, a second target facial expression embedding, and a second target pose embedding (corresponding to the second resolution). Likewise, the scene-based image editing system 106 combines the second source shape embedding, the second target facial expression embedding and the second target pose embedding to generate a second combined embedding corresponding to the second resolution. With each of the combined embeddings (the first combined embedding and the second combined embedding) the scene-based image editing system 106 utilizes the facial expression GAN 7418 to process the source digital image 7402 and conditions layers of the facial expression GAN 7418 (having matching resolutions) with the combined embeddings.
Moreover, although
As mentioned above, the scene-based image editing system 106 also generates animations of a face portrayed within a static source digital image in accordance with one or more embodiments. For example,
As previously discussed, in one or more embodiments the scene-based image editing system 106 generates a source shape embedding by processing a static source digital image.
Furthermore,
In contrast to
For example, as mentioned, the image editing system 104 also generates target pose animation embeddings 7514. In particular, generating target pose animation embeddings 7514 include the scene-based image editing system 106 extracting pose animation information from the target digital video 7508. For instance, the target pose animation embeddings 7514 include extracting information such as the arrangement and/or position of a human portrayed within the target digital video 7508. Furthermore, the scene-based image editing system 106 generates utilizing the three-dimensional encoder 7510 the target pose animation embeddings 7514 based on the extracted pose animation information.
As mentioned above, the scene-based image editing system 106 generates multiple target facial expression animation embeddings, multiple target pose animation embeddings, and multiple source shape embeddings. For example, the scene-based image editing system 106 utilizes the three-dimensional encoder 7504 to generate a first source shape embedding corresponding with a first resolution of the static source digital image 7502. Moreover, the scene-based image editing system 106 then utilizes the three-dimensional encoder 7504 to generate a second source shape embedding corresponding with a second resolution of the static source digital image 7502. Specifically, the scene-based image editing system 106 iteratively repeats the process for each resolution of the static source digital image 7502. In doing so, the scene-based image editing system 106 generates the source shape embeddings 7506.
Furthermore, the scene-based image editing system 106 utilizes the three-dimensional encoder 7510 to process a first frame of the target digital video 7508 and generate a first target facial expression animation embedding and a first target pose animation embedding corresponding to a first resolution of the first frame. Moreover, the scene-based image editing system 106 further utilizes the three-dimensional encoder 7510 to generate a second target facial expression animation embedding and a second target pose animation embedding corresponding to a second resolution of the first frame. Furthermore, for a second frame of the target digital video 7508, the scene-based image editing system 106 generates a third target facial expression animation embedding and a third target pose animation embedding corresponding to a first resolution of the second frame. As mentioned, the scene-based image editing system 106 iteratively repeats this process for each frame of the target digital video 7508 and each resolution of each frame. Accordingly, the scene-based image editing system 106 generates the target facial expression animation embeddings 7512 and the target pose animation embeddings 7514.
Moreover, as shown, the scene-based image editing system 106 generates combined embeddings 7516. In particular, the scene-based image editing system 106 combines the source shape embeddings 7506, the target facial expression animation embeddings 7512, and the target pose animation embeddings 7514. Further, the scene-based image editing system 106 utilizes the facial animation GAN 7518 to process the combined embeddings 7516. Moreover, the scene-based image editing system 106 utilizes the facial animation GAN 7518 to generate an animation 7520. For instance, the animation portrays the source face from the static source digital image 7502 according to the target animation from the target digital video 7508. In one or more embodiments, the scene-based image editing system 106 implements the architecture of CoModGAN (or another GAN described herein) as the facial animation GAN 7518. In some embodiments, the scene-based image editing system 106 implements the architecture of a diffusion neural network as the facial animation GAN 7518.
Although
As mentioned above, the scene-based image editing system 106 in one or more embodiments utilizes a diffusion neural network to generate an animation. Diffusion neural networks were discussed above (e.g., with regard to
Specifically,
As mentioned, the scene-based image editing system 106 performs an act 7624 of conditioning layers of the denoising neural network 7614 and a denoising neural network 7618. For example, the scene-based image editing system 106 utilizes an encoder 7630 to process a static source digital image 7626 and a target digital video 7628. In particular, the scene-based image editing system 106 generates utilizing the encoder 7630 source shape embeddings 7632, target pose animation embeddings 7634, and target facial expression animation embeddings 7636. Conditioning was previously discussed. As discussed, the scene-based image editing system 106 conditions layers of the denoising neural network 7614 and the denoising neural network 7618 with the source shape embeddings 7632, the target pose animation embeddings 7634, and/or the target facial expression animation embeddings 7636 (e.g., a combined embedding). Additionally, as shown, the scene-based image editing system 106 generates a final denoised representation 7620 and processes the final denoised representation 7620 with a decoder 7621. In particular, the scene-based image editing system 106 utilizes the decoder 7621 to generate an animation 7622 (or a digital image in each iteration that is combined to generate the animation 7622) that portrays the source face.
Although
As mentioned above, the scene-based image editing system 106 provides an intuitive and efficient user interface for performing facial expression transfer in accordance with one or more embodiments. For example,
Furthermore,
Moreover, in addition to the source image selectable element 7702 and the target face selectable element 7704, the scene-based image editing system 106 provides pre-defined target facial expressions 7706. For example, the scene-based image editing system 106 provides to a user of the client device 7716 the pre-defined target facial expressions 7706. In particular, the pre-defined target facial expressions 7706 includes expressions (i.e., embeddings for target expressions/poses) previously generated by the scene-based image editing system 106. For instance, the pre-defined target facial expressions 7706 include expressions such as frowning, smiling, angry, crying, or happy. Although
Similar to
Similar to the discussion above relating to pre-defined target facial expressions, the scene-based image editing system 106 also provides pre-defined target animations 7806. For example, pre-defined target animations 7806 include the scene-based image editing system 106 providing in the user interface 7816, options to select animations to animate the static source digital image with. In particular, the scene-based image editing system 106 previously receives multiple digital videos and extracts various types of animations from the digital videos. In doing so, the scene-based image editing system 106 provides the pre-defined target animations 7806. In other embodiments, rather than selecting from the pre-defined target animations 7806, the scene-based image editing system 106 receives a selection of a target animation from a camera roll. In particular, the camera roll includes a plurality of target animations on the client device 7814.
Furthermore, the scene-based image editing system 106 trains a generative model to generate facial expressions from a digital video dataset in accordance with one or more embodiments. For example,
As shown,
Moreover,
Furthermore, as shown, the scene-based image editing system 106 utilizes a GAN 7916 to process the combined embedding 7914 guided by the first frame 7900 of the digital video from the digital video dataset 7902. Moreover, the scene-based image editing system 106 utilizes the GAN 7916 to generate a modified source digital image 7918. Additionally, the scene-based image editing system 106 compares the modified source digital image 7918 with the second frame 7922 of the digital video from the digital video dataset 7902 to determine a measure of loss 7924. Accordingly, the scene-based image editing system 106 back propagates the determined measure of loss to modify parameters of the GAN 7916. Furthermore, although not shown, the scene-based image editing system 106 in a similar manner trains a GAN for generating animations. In particular, the scene-based image editing system 106 utilizes the digital video dataset 7902 and a static source digital image (e.g., the first frame 7900). For instance, in a similar manner, the scene-based image editing system 106 determines a measure of loss by comparing a generated animation with multiple frames of the digital video from the digital video dataset 7902. Furthermore, the scene-based image editing system 106 back propagates the measure of loss to the GAN 7916.
As mentioned above,
In one or more embodiments, the scene-based image editing system 106 intelligently utilizes the features discussed in
In one or more embodiments, the scene-based image editing system 106 detects faces within a digital image being edited and provides recommendations for i) transferring a different facial expression to the portrayed faces within the digital image or ii) animating one or more faces shown in the digital image. In particular, for providing recommendations to transfer a facial expression or animate a face in the digital image, the scene-based image editing system 106 provides an overlay option with pre-defined facial expression or animation options (e.g., happy sad, angry, crying, etc.) for a user to select from. In response to a selection of one of the pre-defined options, the scene-based image editing system 106 modifies the faces shown within the digital image.
In other embodiments, the scene-based image editing system 106 detects the upload of a digital image in addition to an existing digital image being edited. In particular, in response to detecting the upload of the digital image, the scene-based image editing system 106 detects one or more facial expressions within the existing digital image. Further, after detecting the one or more facial expressions within the existing digital image, the scene-based image editing system 106 automatically provides modifications to the uploaded digital image to transfer the facial expressions from the existing digital image to the uploaded digital image. For example, in a family portrait, the scene-based image editing system 106 can detect that a first facial expression (a frown) differs from other facial expressions (other family members smiling). The scene-based image editing system 106 can modify the first facial expression based on one or more other facial expressions in the image. In doing so, the scene-based image editing system 106 intelligently matches uploaded digital image facial expressions with an existing digital image being edited. In other instances however, the scene-based image editing system 106 does not automatically transfer the facial expression, but rather provides an option for the user of the client device to confirm whether the scene-based image editing system 106 should perform a facial expression transfer.
Turning to
As just mentioned, and as illustrated in
Additionally, as shown in
As shown in
Further, as shown in
Additionally, as shown in
Each of the components 8202-8216 of the scene-based image editing system 106 optionally include software, hardware, or both. For example, the components 8202-8216 include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices, such as a client device or server device. When executed by the one or more processors, the computer-executable instructions of the scene-based image editing system 106 cause the computing device(s) to perform the methods described herein. Alternatively, the components 8202-8216 include hardware, such as a special-purpose processing device to perform a certain function or group of functions. Alternatively, the components 8202-8216 of the scene-based image editing system 106 include a combination of computer-executable instructions and hardware.
Furthermore, the components 8202-8216 of the scene-based image editing system 106 may, for example, be implemented as one or more operating systems, as one or more stand-alone applications, as one or more modules of an application, as one or more plug-ins, as one or more library functions or functions that may be called by other applications, and/or as a cloud-computing model. Thus, the components 8202-8216 of the scene-based image editing system 106 may be implemented as a stand-alone application, such as a desktop or mobile application. Furthermore, the components 8202-8216 of the scene-based image editing system 106 may be implemented as one or more web-based applications hosted on a remote server. Alternatively, or additionally, the components 8202-8216 of the scene-based image editing system 106 may be implemented in a suite of mobile device applications or “apps.” For example, in one or more embodiments, the scene-based image editing system 106 comprises or operates in connection with digital software applications such as ADOBE® PHOTOSHOP® or ADOBE® ILLUSTRATOR®. The foregoing are either registered trademarks or trademarks of Adobe Inc. in the United States and/or other countries.
As shown in
In particular, the act 8302 can include determining an infill modification indicating a region to fill for a digital image, the act 8304 can include generating, utilizing a generative semantic machine learning model, an infill semantic map from the digital image and the infill modification, and the act 8306 can include generating, utilizing a generative image machine learning model, a modified digital image from the infill semantic map and the digital image.
For example, the series of acts 8300 also includes generating, by utilizing a semantic map model, a semantic map from the digital image and generating, utilizing the generative semantic machine learning model, the infill semantic map from the semantic map and the infill modification. Moreover, the series of acts 8300 includes determining the region to fill by generating, utilizing a segmentation neural network, a segmented digital image from the digital image. In addition, the series of acts 8300 includes comprises segmenting, utilizing a segmentation neural network, an object to remove from the digital image.
Also, the series of acts 8300 includes comprising generating the infill semantic map from the digital image to remove the object from the digital image by generating, utilizing the generative semantic machine learning model, semantic classifications for pixels within the region to be filled. Further, the series of acts 8300 includes generating, utilizing a semantic map model, an initial semantic map from the digital image, and generating, utilizing a semantic map diffusion neural network, the infill semantic map from the initial semantic map.
Further, the series of acts 8300 includes receiving a semantic editing input corresponding to the initial semantic map and generating the infill semantic map from the initial semantic map utilizing the semantic map diffusion neural network conditioned on the semantic editing input. Moreover, the acts 8300 includes generating utilizing digital image diffusion neural network, the modified digital image, wherein the digital image diffusion neural network is conditioned by the infill semantic map.
Similar to
In particular, the act 8402 includes generating, utilizing the semantic map model, a semantic map from the input digital image, the act 8404 includes determining an infill modification indicating a region to fill for the input digital image, the act 8406 includes generating, utilizing the generative semantic machine learning model, an infill semantic map from the infill modification and the semantic map, and the act 8408 includes generating, utilizing the generative image machine learning model, a modified digital image from the infill semantic map and the input digital image.
For example, the series of acts 8400 includes determining an input texture for the region and generating the modified digital image from the input digital image by utilizing a digital image diffusion neural network conditioned on the infill semantic map and the input texture. Further, the series of acts 8400 also includes segmenting, utilizing a segmentation neural network, an object to remove from the input digital image and generating the infill semantic map from the input digital image and the object to remove from the input digital image. Moreover, the series of acts includes generating the infill semantic map by generating, utilizing the generative semantic machine learning model, semantic classifications of pixels within the region to be filled.
In addition, the series of acts 8400 includes generating, utilizing a semantic map model, an initial semantic map and generating an infill semantic map from the initial semantic map by utilizing a semantic map diffusion neural network. Moreover, the series of acts 8400 includes utilizing a digital image diffusion neural network conditioned on the infill semantic map and the input digital image.
Moving on,
In particular, the act 8502 includes generating a semantic map corresponding to a digital image, the act 8504 includes determining an infill modification indicating a region to fill for the digital image, the act 8506 includes generating, utilizing a semantic map diffusion neural network, an infill semantic map from the semantic map and the infill modification, and the act 8508 includes generating, utilizing a digital image diffusion neural network, a modified digital image from the infill semantic map.
For example, the series of acts 8500 includes generating the modified digital image from the digital image by utilizing the digital image diffusion neural network conditioned on the infill semantic map. In addition, the series of acts 8500 includes generating, utilizing denoising steps of the semantic map diffusion neural network, the infill semantic map conditioned by the semantic map.
Moreover, the series of acts 8500 includes receiving a semantic editing input corresponding to the semantic map and generating the infill semantic map from the semantic map by conditioning denoising steps of the semantic map diffusion neural network with the semantic editing input. Further, the series of acts 8500 includes determine an input texture for modifying the digital image and generating the modified digital image from the infill semantic map by conditioning denoising steps of the digital image diffusion neural network with the input texture. In addition, the series of acts 8500 includes generating semantic classifications for the infill semantic map and generating the modified digital image by conditioning denoising steps of the digital image diffusion neural network with the semantic classifications.
Turning to
In particular, the act 8602 includes providing, for display via a user interface of a client device, an input digital image and a semantic map of the input digital image, the act 8604 includes receiving, via the user interface, an indication of an infill modification comprising a region corresponding to the input digital image to fill, the act 8606 includes provide, for display via the user interface, an infill semantic map comprising semantic classifications of pixels within the region of the input digital image to fill, and the act 8608 includes in response to an image completion interaction via the user interface, generating a modified digital image comprising infilled pixels for the region according to the infill semantic map.
For example, the series of acts 8600 includes receiving the indication of the infill modification by receiving a user selection of an expanded region beyond the input digital image and providing the infill semantic map by providing, for display via the user interface, the infill semantic map of the input digital image within an expanded digital image frame corresponding to the expanded region. Further, the series of acts 8600 includes generating, in response to an infill completion interaction via the user interface, the infill semantic map, wherein the semantic classifications of the pixels extends into the expanded digital image frame.
In addition, the series of acts 8600 includes providing the infill semantic map comprises providing, for display via the user interface, a plurality of infill semantic maps and in response to user selection of the infill semantic map from the plurality of infill semantic maps, generating the modified digital image. Further, the series of acts 8600 includes providing, for display via the user interface, the infill semantic map comprising the semantic classifications within a semantic boundary and providing, for display via the user interface, an additional infill semantic map comprising additional semantic classifications within an additional semantic boundary.
As another example, the series of acts 8600 includes generating a plurality of modified digital images according to the infill semantic map, providing, for display via the user interface, the plurality of modified digital images, and receiving a user selection of the modified digital image from the plurality of modified digital images generated according to the infill semantic map. Moreover, the series of acts 8600 includes receiving, via the user interface, a semantic editing input for a region of the semantic map, generating the infill semantic map guided by the semantic editing input, and providing, for display via the user interface, the infill semantic map guided by the semantic editing input.
Further, the series of acts 8600 includes providing, for display via the user interface, a semantic editing tool and determining the semantic editing input comprises receiving an input semantic classification or an input semantic boundary based on user interaction with the semantic editing tool. Moreover, the series of acts 8600 includes providing, via the user interface, a segmentation option to apply a segmentation model to the input digital image, in response to receiving a selection of the segmentation option, generating a segmented digital image, and providing, via the user interface, the segmented digital image, wherein the segmented digital image indicates the region of the input digital image to fill.
Additionally, the series of acts 8600 includes providing, for display via the user interface, a selectable texture option, in response to receiving a user interaction with the selectable texture option, identifying an input texture, and generating the modified digital image by filling the region corresponding to the indicated infill modification guided by the selected texture option. Moreover, the series of acts 8600 includes providing, for display via the user interface, a diffusion iteration option and in response to user interaction with the diffusion iteration option: determining a number of diffusion iterations and generating the modified digital image by utilizing a diffusion neural network comprising the number of diffusion iterations.
In particular, the act 8702 includes providing, for display via a user interface of a client device, a digital image and a selectable texture option, the act 8704 includes based on user interaction with the selectable texture option, identifying an input texture, the act 8706 includes receiving, via the user interface, an indication of an infill modification comprising a region of the digital image to fill, the act 8708 includes generating, utilizing a generative image machine learning model, a modified digital image by filling the region of the digital image conditioned on the input texture, and the act 8710 includes providing the modified digital image for display via the user interface.
For example, the series of acts 8700 includes receiving, via the user interface, an indication of an infill modification comprising an object portrayed in the digital image to replace and providing, for display via the user interface of the client device, a plurality of selectable texture options comprising texture patches for replacing the object. Further, the series of acts 8700 includes receiving, via the user interface, a selection of a selectable texture option corresponding to the input texture and generating, utilizing the generative image machine learning model, a modified digital image by replacing the object depicted in the digital image based on a texture patch. Moreover, the series of acts 8700 includes providing, via the user interface, a segmentation option to apply a segmentation model to the digital image, in response to receiving a selection of the segmentation option, generating a segmented digital image, and providing, via the user interface, the segmented digital image, wherein the segmented digital image indicates the region of the digital image to fill conditioned on the input texture.
In particular, the act 8802 includes providing, for display via a user interface of a client device, an input digital image and a semantic map of the input digital image, the act 8804 includes receiving, via the user interface, an indication of a semantic editing input for a region of the semantic map, the act 8806 includes generating, utilizing a generative semantic machine learning model, an infill semantic map from the semantic map and the indication of the semantic editing input, the act 8808 includes providing, for display via the user interface, the infill semantic map comprising the semantic editing input, and the act 8810 includes in response to user interaction with an image completion element, generating a modified digital image comprising infilled pixels for the region.
Further the series of acts 8800 includes receiving an input semantic classification or an input semantic boundary. Additionally, the series of acts 8800 includes providing, for display via the user interface, a semantic editing tool and receiving, via the semantic editing tool, an indication of an input semantic classification or an input semantic boundary. Moreover, the series of acts 8800 includes providing, for display via the user interface, a diffusion iteration option and in response to user interaction with the diffusion iteration option: determining a number of diffusion iterations and generating the infill semantic map by utilizing a diffusion neural network comprising the number of diffusion iterations guided by the indication of the semantic editing input. Additionally, the series of acts 8800 includes providing, via the user interface, a segmentation option to apply a segmentation model to the input digital image, in response to receiving a selection of the segmentation option, generating a segmented digital image, and generating the modified digital image from the segmented digital image indicating the region of the input digital image to fill and the semantic editing input.
In particular, the act 8902 includes determining, from a digital image, a depiction of a human and a region of the human to inpaint, the act 8904 includes generating, utilizing an encoder, a structural encoding from a structure guidance map of the human, the act 8906 includes generating, utilizing the encoder, a visual appearance encoding from the human portrayed in the digital image, and the act 8908 includes generating, utilizing a human inpainting generative adversarial neural network, a modified digital image comprising modified pixels of the region from the structural encoding and the visual appearance encoding of the human.
For example, the series of acts 8900 include generating, utilizing a hierarchical encoder comprising a plurality of downsampling layers and upsampling layers of matching resolutions connected via skip connections, the structural encoding from the structure guidance map of the human and the visual appearance encoding from the human portrayed in the digital image. Additionally, the series of acts 8900 include generating, utilizing a parameter neural network, a local appearance feature tensor from the visual appearance encoding. Further, the series of acts 8900 include generating the local appearance feature tensor comprises generating, utilizing the parameter neural network, a spatially varying scaling tensor and a spatially varying shifting tensor. Moreover, the series of acts 8900 include generating, utilizing a first style block of the human inpainting generative adversarial neural network, an intermediate feature vector by modulating the structural encoding utilizing the spatially varying scaling tensor and the spatially varying shifting tensor.
Further, the series of acts 8900 include utilizing a modulation layer, a convolutional layer, and a demodulation layer to generate the intermediate feature vector. Moreover, the series of acts 8900 include generating, utilizing a second style block of the human inpainting generative adversarial neural network, an additional intermediate feature vector from the intermediate feature vector, an additional spatially varying scaling tensor, and an additional spatially varying shifting tensor. Additionally, the series of acts 8900 include generating the structural encoding from at least one of a keypoint map or a pose map. Further, the series of acts 8900 includes generating the structural encoding from at least one of a segmentation map or a digital image.
In addition the series of acts 8900 also include determining a human portrayed in the digital image and a region of the human to inpaint, generating, utilizing an encoder, a structural encoding from the structure guidance map of the human, generating, utilizing the encoder, a visual appearance encoding of the human, generating, utilizing the parameter neural network, local appearance feature tensor from the visual appearance encoding and generating, utilizing a human inpainting generative adversarial neural network, a modified digital image comprising modified pixels of the region from the structural encoding and the local appearance feature tensor.
Moreover, the series of acts include generating a spatially varying scaling tensor and a spatially varying shifting tensor by utilizing the parameter neural network. Further, the series of acts 8900 include modulating the structural encoding utilizing the spatially varying scaling tensor and the spatially varying shifting tensor. Additionally, the series of acts 8900 include utilizing a first style block of the human inpainting generative adversarial neural network, an intermediate feature vector by utilizing a modulation layer, a convolutional layer, and a demodulation layer.
Further, the series of acts 8900 include generating the structural encoding from at least one of a keypoint map, a segmentation map, or a pose map. Moreover, the series of acts 8900 include determining a partial reconstruction loss for a portion of the digital image that does not include the region of the human to inpaint and modifying parameters of the human inpainting generative adversarial neural network based on the partial reconstruction loss. In addition, the series of acts 8900 include determining an adversarial loss for the region of the human to inpaint and modifying parameters of the human inpainting generative adversarial neural network based on the adversarial loss and the partial reconstruction loss. Further, the series of acts 8900 include determining an indication of at least one of removing an object, expanding a digital image frame of the digital image, or removing an object from the region of the human.
In one or more embodiments, the series of acts 8900 includes determining a human portrayed in a digital image and a region of the human to inpaint, generating, utilizing a first encoder, a structural encoding from a structure guidance map of a depiction of the human from the digital image, generating, utilizing a second encoder, a visual appearance encoding from the depiction of the human from the digital image, and generating, utilizing a human inpainting generative adversarial neural network, a modified digital image comprising modified pixels of the region from the structural encoding by modulating style blocks of the human inpainting generative adversarial neural network based on the visual appearance encoding.
For example, the series of acts 8900 includes comprises generating, utilizing a hierarchical encoder, the structural encoding from the structure guidance map of the human and the visual appearance encoding from the human portrayed in the digital image, wherein the hierarchical encoder comprises a plurality of downsampling layers and upsampling layers of matching resolutions connected via skip connections. Further, the series of acts 8900 includes generating an intermediate feature vector for each style block of the human inpainting generative adversarial neural network by modulating the structural encoding utilizing a spatially varying scaling tensor and a spatially varying shifting tensor corresponding to a style block.
In particular, the act 9002 includes determining a region of a human portrayed within a digital image to inpaint, the act 9004 includes generating, utilizing a generative segmentation machine learning model, an infill segmentation map from the digital image, the infill segmentation map comprising a human segmentation classification for the region, the act 9006 includes generating, utilizing a human inpainting generative adversarial neural network, a modified digital image from the digital image and the infill segmentation map, wherein the modified digital image comprises modified pixels for the region corresponding to the human segmentation classification.
For example, the series of acts 9000 includes generating, utilizing a segmentation machine learning model, an initial segmentation map that comprises an unclassified region corresponding to the region of the human to inpaint. Further, the series of acts 9000 includes generating, utilizing the generative segmentation machine learning model, the infill segmentation map from the initial segmentation map and wherein generating the initial segmentation map comprises generating a segmentation classification for the unclassified region.
Further, the series of acts 9000 includes generating a structural encoding from the infill segmentation map and a visual appearance encoding from the digital image. Additionally, the series of acts 9000 includes generating a structural encoding and a visual appearance encoding further comprises utilizing a hierarchical encoder comprising a plurality of downsampling layers and upsampling layers connected via skip connections. Moreover, the series of acts 9000 includes generating, utilizing a background generative adversarial neural network, a modified background portion of the digital image from the digital image and generating, utilizing the human inpainting generative adversarial neural network, the modified pixels of the region of the human portrayed within the digital image.
Additionally, the series of acts 9000 includes inpainting the human portion utilizing the human inpainting generative adversarial neural network, and inpainting the background portion based on the inpainted human portion and utilizing the background generative adversarial neural network. Further, the series of acts 9000 includes generating an intermediate digital image by removing the region of the human and inpainting the background portion utilizing the background generative adversarial neural network, generating, utilizing the human inpainting generative adversarial neural network, the modified pixels for the region corresponding to the human segmentation classification, and generating the modified digital image by inserting the modified pixels into the intermediate digital image.
The series of acts 9000 also includes determining a region of the digital image to inpaint, the region comprising a human portion of the digital image and a background portion of the digital image, generating, utilizing the generative segmentation machine learning model, an infill segmentation map from the digital image, the infill segmentation map comprising a human segmentation classification for the human portion, generating, utilizing the human inpainting generative adversarial neural network, a modified human portion of the digital image from the digital image and the infill segmentation map, and generating, utilizing the background generative adversarial neural network, a modified background portion of the digital image from the digital image.
Further, the series of acts 9000 includes generating a modified digital image by combining the modified background portion and the modified human portion of the digital image from the digital image. Additionally, the series of acts 9000 includes generating, utilizing an encoder, a structural encoding from the infill segmentation map, generating, utilizing the encoder, a visual appearance encoding based on the human portion of the digital image, and generating the modified digital image from the modified background portion, the structural encoding, and the visual appearance encoding.
Moreover, the series of acts 9000 includes generating a mask for the region of the human portion in the digital image and a mask for the background portion of the digital image. In addition, the series of acts 9000 modifying pixels for the region corresponding to the human portion of the digital image utilizing the mask for the region of the human portion, and modifying pixels for the region corresponding to the background portion of the digital image utilizing the mask for the background portion. Further, the series of acts 9000 includes generating, utilizing a segmentation machine learning model, an initial segmentation map that comprises an unclassified region corresponding to the region of the human portion to inpaint and generating, utilizing the generative segmentation machine learning model, the infill segmentation map from the initial segmentation map.
The series of acts 9000 also includes determining a region of a human portrayed within a digital image to inpaint, generating, utilizing a generative segmentation machine learning model, an infill segmentation map from the digital image, the infill segmentation map comprising a human segmentation classification for the region, generating, utilizing a hierarchical encoder comprising a plurality of downsampling layers and upsampling layers connected via skip connections, a structural encoding from the infill segmentation map, and generating, utilizing a human inpainting generative adversarial neural network, a modified digital image from the digital image and the structural encoding, the modified digital image comprising modified pixels for the region corresponding to the human segmentation classification.
Further, the series of acts 9000 also includes generating, utilizing the hierarchical encoder, a visual appearance encoding based on the digital image and generating, utilizing the human inpainting generative adversarial neural network, the modified digital image from the visual appearance encoding and the structural encoding. Moreover, the series of acts 9000 include identifying a background portion of the digital image, generating, utilizing a background generative adversarial neural network, a modified background portion of the digital image from the digital image, and generating the modified digital image by combining the modified background portion and the modified pixels of the region of the human from the structural encoding and the visual appearance encoding.
In addition, the series of acts 9000 includes generating a mask for the region of the human portrayed in the digital image and a mask for the background portion of the digital image, and generating the modified digital image by utilizing the mask for the region of the human and the mask for the background portion. Moreover, the series of acts 9000 includes generating, utilizing a segmentation machine learning model, an initial segmentation map that comprises an unclassified region corresponding to the region of the human to inpaint and wherein generating the initial segmentation map comprises generating a segmentation classification for the unclassified region. Additionally, the series of acts 9000 includes generating, utilizing the generative segmentation machine learning model, the infill segmentation map from the initial segmentation map.
In particular, the act 9102 includes generating, utilizing a pose encoder, a pose feature map from a target pose map and a source digital image that depicts a human, the act 9104 includes generating, utilizing a texture map appearance encoder, a global texture map appearance vector from a texture map of the source digital image, the act 9106 includes generating, utilizing a parameter neural network, a local appearance feature tensor from the source digital image, and the act 9108 includes synthesizing, utilizing a reposing generative adversarial neural network, a modified digital image that depicts the human according to the target pose map based on the pose feature map, the local appearance feature tensor, and the global texture map appearance vector.
For example, the series of acts 9100 includes generating the pose feature map utilizing a hierarchical pose encoder by: generating, utilizing a downsampling layer of the hierarchical pose encoder, an intermediate downsampled feature vector having a resolution, generating utilizing an upsampling layer of the hierarchical pose encoder, an intermediate upsampled feature vector having the resolution, and generating, via a skip connection, a combined feature vector from the intermediate downsampled feature vector and the intermediate upsampled feature vector.
In addition, the series of acts 9100 includes generating, utilizing an additional downsampling layer of the hierarchical pose encoder, an additional intermediate downsampled feature vector having an additional resolution, generating utilizing an additional upsampling layer of the hierarchical pose encoder from the combined feature vector, an additional intermediate upsampled feature vector having the additional resolution, and combining, via a skip connection, the additional intermediate downsampled feature vector and the additional intermediate upsampled feature vector to generate the pose feature map. Further, the series of acts 9100 includes generating a one-dimensional appearance vector utilizing a hierarchical appearance encoder.
Further, the series of acts 9100 includes generating, utilizing an image appearance encoder, a local appearance vector from the source digital image, and generating, utilizing the parameter neural network, the local appearance feature tensor from the local appearance vector, the local appearance feature tensor comprising a spatially varying scaling tensor and a spatially varying shifting tensor. Moreover, the series of acts 9100 includes generating the modified digital image from the pose feature map by: generating globally modified local appearance feature tensor by combining the local appearance feature tensor and the global texture map appearance vector and modulating the reposing generative adversarial neural network utilizing the globally modified local appearance feature tensor.
In addition, the series of acts 9100 includes training the reposing generative adversarial neural network by: generating incomplete digital images from an unpaired image dataset by applying masks to a plurality of source digital images portraying humans in the unpaired image dataset and synthesizing, utilizing the reposing generative adversarial neural network, a plurality of inpainted digital images from the incomplete digital images and target pose maps of the plurality of source digital images. Additionally, the series of acts 9100 includes modifying parameters of the reposing generative adversarial neural network based on measures of loss between the plurality of inpainted digital images and the plurality of source digital images. Moreover, the series of acts 9100 includes training the reposing generative adversarial neural network utilizing an unpaired dataset comprising the source digital image by: determining, utilizing a discriminator neural network, an adversarial loss from the modified digital image and modifying parameters of the reposing generative adversarial neural network based on the adversarial loss from the modified digital image generated from the unpaired dataset.
The series of acts 9100 also includes generating, utilizing a texture map appearance encoder of the reposing neural network, a global texture map appearance vector from the texture map, generating, utilizing an image appearance encoder of the reposing neural network, a local appearance vector from the source digital image, generating, utilizing a parameter neural network, local appearance feature tensor from the local appearance vector, and synthesize, utilizing the reposing neural network, a modified digital image that depicts the human according to the target pose map based on the source digital image, the local appearance feature tensor, and the global texture map appearance vector.
Moreover, the series of acts 9100 includes generating the global texture map appearance vector from the texture map of the source digital image comprises generating a one-dimensional appearance vector utilizing a hierarchical appearance encoder and generating the local appearance feature tensor from the local appearance vector comprises a spatially varying scaling tensor and a spatially varying shifting tensor. Further, the series of acts 9100 includes generating globally modified local appearance feature tensor by combining the global texture map appearance vector with the local appearance feature tensor.
Additionally, the series of acts 9100 includes modulate the reposing neural network utilizing the globally modified local appearance feature tensor. Further, the series of acts 9100 includes generating the texture map from pixel values of the source digital image and a pose map of the source digital image. Moreover, the series of acts 9100 includes training the reposing neural network by generating incomplete digital images from an unpaired image dataset by applying masks to a plurality of source digital images portraying humans in the unpaired image dataset, synthesizing, utilizing the reposing neural network, a plurality of inpainted digital images from the incomplete digital images and target pose maps of the plurality of source digital images, and modifying parameters of the reposing neural network based on measures of loss between the plurality of inpainted digital images and the plurality of source digital images.
The series of acts 9100 also include generating, utilizing a texture map appearance encoder of a reposing neural network, a global texture map appearance vector from a texture map, generating, utilizing an image appearance encoder of the reposing neural network, a local appearance vector from a source digital image that depicts a human, generating, a combined vector from the global texture map appearance vector and the local appearance vector, and synthesizing, utilizing a reposing generative adversarial neural network, a modified digital image that depicts the human according to a target pose map based on the combined vector. Additionally, the series of acts 9100 includes generating the texture map from pixel values of the source digital image and a pose map of the source digital image, and generating the global texture map appearance vector from the texture map of the source digital image comprises generating a one-dimensional appearance vector utilizing a hierarchical appearance encoder.
Moreover, the series of acts 9100 includes generating a local appearance feature tensor from the local appearance vector, the local appearance feature tensor comprises a spatially varying scaling tensor and a spatially varying shifting tensor and generating the combined vector by combining the global texture map appearance vector with the local appearance feature tensor. Further, the series of acts 9100 includes synthesizing the modified digital image by modulating a style block of the reposing neural network utilizing the combined vector. Additionally, the series of acts 9100 includes training the reposing neural network by modifying parameters of the reposing neural network based on an adversarial loss from the modified digital image generated from an unpaired dataset comprising the source digital image.
In particular, the act 9202 includes generating a texture map from pixel values of a source digital image that depicts a human and from a pose map of the source digital image, the act 9204 includes generating a warped digital image by warping the texture map with a target pose map, the act 9206 includes generating, utilizing a pose encoder, a warped pose feature map from the warped digital image and the target pose map, the act 9208 includes generating, utilizing a texture map appearance encoder, a global texture map appearance vector from the texture map, and the act 9210 includes synthesizing, utilizing a reposing generative adversarial neural network, a modified digital image that depicts the human according to the target pose map based on the warped pose feature map and the global texture map appearance vector.
For example, the series of acts 9200 includes generating, utilizing a three-dimensional model, the pose map from the source digital image and generating the texture map by projecting the pixel values of the source digital image guided by the pose map of the source digital image. Further, the series of acts 9200 includes determining UV coordinates from the target pose map and generating the warped digital image by rearranging the pixel values from the source digital image in the texture map based on the UV coordinates from the target pose map. Moreover, the series of acts 9200 includes generating, utilizing an image appearance encoder of the reposing generative adversarial neural network, a local appearance vector from the warped digital image, generating, utilizing a parameter neural network, a local appearance feature tensor from the local appearance vector, and synthesizing, utilizing the reposing generative adversarial neural network, the modified digital image that depicts the human according to the target pose map based on the warped pose feature map, the global texture map appearance vector, and the local appearance feature tensor.
In addition, the series of acts 9200 includes generating a globally modified local appearance feature tensor by combining the global texture map appearance vector and the local appearance feature tensor. Moreover, the series of acts 9200 includes generating, utilizing a style block of the reposing generative adversarial neural network, an intermediate feature vector by modulating the warped pose feature map utilizing the globally modified local appearance feature tensor. Furthermore, the series of acts 9200 includes generating the warped digital image further comprises generating, utilizing a coordinate inpainting generative neural network, an inpainted warped digital image and generating the warped pose feature map comprises generating, utilizing the pose encoder, the warped pose feature map from the inpainted warped digital image.
Additionally, the series of acts 9200 include training the reposing generative adversarial neural network by: generating a body mask for the human depicted in the modified digital image and determining a measure of loss for a portion of the modified digital image within the body mask utilizing a first loss weight. Further, the series of acts 9200 includes training the reposing generative adversarial neural network by: determining an additional measure of loss for an additional portion of the modified digital image outside the body mask utilizing a second loss weight and modifying parameters of the reposing generative adversarial neural network based on the measure of loss and the additional measure of loss.
Moreover, the series of acts 9200 includes generating a texture map from pixel values of the source digital image that depicts a human and a pose map of the source digital image, generating a warped digital image by warping the texture map with the target pose map, generating, utilizing a pose encoder, a warped pose feature map from the warped digital image and the target pose map, and synthesizing, utilizing the reposing neural network from the warped pose feature map, a modified digital image that depicts the human according to the target pose map.
In addition, the series of acts 9200 includes generating, utilizing a three-dimensional model, the pose map from the source digital image, generating the texture map by projecting the pixel values of the source digital image guided by the pose map of the source digital image, and generating the warped digital image by rearranging the pixel values from the source digital image in the texture map with UV coordinates from the target pose map.
Further, the series of acts 9200 includes generating, utilizing a coordinate inpainting generative neural network, an inpainted warped digital image from the warped digital image, and generating, utilizing the pose encoder, the warped pose feature map from the inpainted warped digital image. Moreover, the series of acts 9200 includes generating, utilizing a texture map appearance encoder, a global texture map appearance vector from the texture map and generating, utilizing an image appearance encoder, a warped image feature vector from the warped digital image.
Additionally, the series of acts 9200 includes generating, utilizing a parameter neural network, a local appearance feature tensor from the warped image feature vector and generating a globally modified local appearance feature tensor by combining the global texture map appearance vector and the local appearance feature tensor. Moreover, the series of acts 9200 includes synthesizing the modified digital image that depicts the human according to the target pose map utilizing the reposing neural network from the warped pose feature map and the globally modified local appearance feature tensor.
The series of acts 9200 also includes generating a warped digital image by warping a texture map of a digital image with a target pose map, generating, utilizing a pose encoder, a warped pose feature map from the warped digital image and the target pose map, generating, utilizing a parameter neural network, local appearance feature tensor from the warped digital image, and synthesizing, utilizing a reposing generative adversarial neural network, a modified digital image that depicts a human according to the target pose map based on the warped pose feature map and the local appearance feature tensor.
Additionally, the series of acts 9200 includes generating the warped digital image by rearranging pixel values from the digital image in the texture map with UV coordinates from the target pose map. Moreover, the series of acts 9200 includes generating a globally modified local appearance feature tensor from the local appearance feature tensor and a global texture map appearance vector from the texture map and modulating the warped pose feature map via the reposing generative adversarial neural network utilizing the globally modified local appearance feature tensor. Further, the series of acts 9200 includes utilizing a coordinate inpainting generative neural network, an inpainted warped digital image. Moreover, the series of acts 9200 includes generating, utilizing the pose encoder, the warped pose feature map from the inpainted warped digital image.
In particular, the act 9302 includes generating, utilizing a three-dimensional encoder, a target facial expression embedding from a target digital image portraying a face having a target facial expression, the act 9304 includes generating, utilizing the three-dimensional encoder, a source shape embedding from a source digital image portraying a source face having a source pose and a source facial expression, and the act 9306 includes generating, utilizing a facial expression generative neural network from the source digital image, a modified source digital image that portrays the source face of the source digital image with the target facial expression from the target digital image by conditioning layers of the facial expression generative neural network based on the target facial expression embedding and the source shape embedding.
For example, the series of acts 9300 include generating, utilizing the three-dimensional encoder, a target pose embedding from the target digital image, wherein the face comprises a target pose. Further, the series of acts 9300 include generating, utilizing the facial expression generative neural network from the source digital image, the modified source digital image that portrays the source face of the source digital image with the target pose and the target facial expression from the target digital image by conditioning layers of the facial expression generative neural network based on the target pose embedding, the target facial expression embedding, and the source shape embedding.
Additionally, the series of acts 9300 include generating a combined embedding from the target pose embedding, the target facial expression embedding, and the source shape embedding. Further, the series of acts 9300 include generating the modified source digital image from the source digital image by conditioning layers of the facial expression generative neural network with the combined embedding. Moreover, the series of acts 9300 include providing, for display via a user interface of a client device, a source image selectable element and a target face selectable element, determining the source digital image based on user interaction with the source image selectable element, and determining the target facial expression embedding based on user interaction with the target face selectable element.
Furthermore, the series of acts 9300 includes receiving a selection, via the client device, of the target digital image or receiving a selection, via the client device, of a pre-defined target facial expression. Additionally, the series of acts 9300 includes training the facial expression generative neural network utilizing a digital video dataset by: generating the source shape embedding from a first frame of a digital video of the digital video dataset and generating a target pose embedding and the target facial expression embedding from a second frame of the digital video of the digital video dataset.
Moreover, the series of acts 9300 includes generating the modified source digital image that portrays the source face from the first frame of the digital video with the target pose and the target facial expression from the second frame of the digital video and modifying parameters of the facial expression generative neural network by comparing the modified source digital image with the second frame of the digital video.
The series of acts 9300 includes generating, utilizing a facial expression generative neural network guided by the source digital image the target pose embedding, and the source shape embedding, a modified source digital image that portrays the source face of the source digital image with the target pose from the target digital image. In addition, the series of acts 9300 includes generating, utilizing the three-dimensional encoder, a target facial expression embedding from the target digital image, wherein the face comprises a target facial expression, and generating, utilizing the facial expression generative neural network from the source digital image, the modified source digital image that portrays the source face of the source digital image with the target pose and the target facial expression from the target digital image by conditioning layers of the facial expression generative neural network based on the target pose embedding, the target facial expression embedding, and the source shape embedding.
Further, the series of acts 9300 includes generating, a combined embedding from the target pose embedding, the target facial expression embedding, and the source shape embedding and generating the modified source digital image from the source digital image by conditioning layers of the facial expression generative neural network with the combined embedding. Moreover, the series of acts 9300 includes providing, for display via a user interface of a client device, a source image selectable element and a target face selectable element.
In addition, the series of acts 9300 includes determining a target facial expression embedding by receiving a selection, via the client device, of a pre-defined target facial expression. Further, the series of acts 9300 includes determining a target facial expression embedding by receiving a selection, via the client device, of the target digital image.
The series of acts 9300 includes providing, for display via a user interface of a client device, source image selection element and a target face selection element, in response to interaction with the source image selection element, determining a source digital image, in response to user interaction with the target face selection element, determining a target pose embedding and a target facial expression embedding, generating, utilizing a three-dimensional encoder, a source shape embedding from the source digital image portraying a source face having a source pose and a source facial expression, and generating, utilizing a facial expression generative neural network from the source digital image, a modified source digital image that portrays the source face of the source digital image with a target pose and target facial expression by conditioning style blocks of the facial expression generative neural network based on the target pose embedding, the target facial expression embedding, and the source shape embedding.
Further, the series of acts 9300 includes providing an option to select a target digital image that portrays the target pose and the target facial expression or an option to select a pre-defined target facial expression. Moreover, the series of acts 9300 includes training the facial expression generative neural network utilizing a digital video dataset by generating the source shape embedding from a first frame of a digital video of the digital video dataset. Additionally, the series of acts 9300 includes training the facial expression generative neural network utilizing a digital video dataset by generating the target pose embedding and the target facial expression embedding from a second frame of the digital video of the digital video dataset. Moreover, the series of acts 9300 includes training the facial expression generative neural network utilizing a digital video dataset by generating the modified source digital image that portrays the source face from the first frame of the digital video with the target pose and the target facial expression from the second frame of the digital video to modify parameters of the facial expression generative neural network by comparing the modified source digital image with the second frame of the digital video.
In particular, the act 9402 includes extracting, utilizing a three-dimensional encoder, target facial expression animation embeddings from a target digital video portraying a target animation of a face, the act 9404 includes identifying a static source digital image portraying a source face having a source shape and facial expression, and the act 9406 includes generating, utilizing a facial animation generative neural network from the static source digital image and the target facial expression animation embeddings, an animation that portrays the source face animated according to the target animation from the target digital video.
For example, the series of acts 9400 includes utilizing the three-dimensional encoder, target pose animation embeddings from the target digital video portraying the target animation of the face. Moreover, the series of acts 9400 includes generating the animation utilizing the facial animation generative neural network from the static source digital image, the target facial expression animation embeddings, and the target pose animation embeddings. Further, the series of acts 9400 includes generating a first target facial expression animation embedding from a first frame of the target digital video and a second target facial expression embedding from a second frame of the target digital video.
Additionally, the series of acts 9400 includes generating the animation that portrays the source face animated according to the target animation from the target digital video by utilizing a facial animation diffusion neural network. Further, the series of acts 9400 includes conditioning the facial animation diffusion neural network with a target pose animation embeddings, the target facial expression animation embeddings, and a source shape embedding from the source face. Moreover, the series of acts includes providing, for display via a user interface of a client device, a source image selection element and a target animation selection element and based on user interaction with the source image selection element and the target animation selection element, identify the static source digital image portraying the source face and the target facial expression animation embeddings. Further, the series of acts 9400 includes providing a plurality of pre-defined target animations for display. In addition the series of acts 9400 includes identifying the target animation from a camera roll of the client device based on user interaction with the target animation selection element.
The series of acts 9400 also includes providing, for display via a user interface of a client device, a source image selection element and a target animation selection element, based on user interaction with the source image selection element and the target animation selection element, identifying a static source image portraying a source face, and target pose animation embeddings, and target facial expression animation embeddings of a target animation of a target face, generating, utilizing a facial animation generative neural network from the static source digital image, the target pose animation embeddings, and the target facial expression animation embeddings, an animation that portrays the source face animated according to the target animation of the target face, and providing the animation for display via the user interface of the client device.
Further, the series of acts 9400 includes providing, for display via the user interface of the client device an option to select from a plurality of pre-defined target animations. Additionally, the series of acts 9400 includes providing, for display via the user interface of the client device an option to select a digital image from a camera roll of the client device. Moreover, the series of acts 9400 includes identifying the static source image portraying the source face by receiving a selection of the digital image from the camera roll of the client device. Additionally, the series of acts 9400 includes generating the target facial expression animation embeddings by: generating, utilizing a three-dimensional encoder, a first target facial expression animation embedding from a first frame of the target digital video and generating, utilizing the three-dimensional encoder, a second target facial expression animation embedding from a second frame of the target digital video.
Moreover, the series of acts 9400 includes generating the animation utilizing the facial animation generative neural network by: conditioning a facial animation diffusion neural network with the first target facial expression animation embedding and the second target facial expression animation embedding.
The series of acts 9400 also includes extracting, utilizing a three-dimensional encoder, target pose animation embeddings and target facial expression animation embeddings from a target digital video portraying a target animation of a face, extracting, utilizing a three-dimensional encoder, a source shape embedding from a static source digital image that portrays a source face, generating, utilizing a facial animation generative neural network from the static source digital image, an animation that portrays the source face animated according to an animation selection element by conditioning style blocks of the facial animation generative neural network with the target pose animation embeddings, the target facial expression animation embeddings, and the source shape embedding, and providing the animation for display via a user interface of a client device.
Further the series of acts 9400 includes generating combined embeddings from the target pose animation embeddings, the target facial expression animation embeddings, and the source shape embedding and generating the animation by conditioning style blocks of the facial animation generative neural network with the combined embeddings. Moreover, the series of acts 9400 includes providing, for display via a user interface of a client device, a source image selection element and a target animation selection element, wherein the target animation selection element comprises providing a plurality of pre-defined target animations for display. Also, the series of acts 9400 includes identifying the source digital image based on user interaction with the source image selection element comprising an option to select a digital image from a camera roll of the client device. Additionally, the series of acts 9400 include utilizing a three-dimensional encoder to generate the target pose animation embedding and the target facial expression animation embeddings.
Embodiments of the present disclosure may comprise or utilize a special purpose or general-purpose computer including computer hardware, such as, for example, one or more processors and system memory, as discussed in greater detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. In particular, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). In general, a processor (e.g., a microprocessor) receives instructions, from a non-transitory computer-readable medium, (e.g., a memory), and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.
Computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer system. Computer-readable media that store computer-executable instructions are non-transitory computer-readable storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, embodiments of the disclosure can comprise at least two distinctly different kinds of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.
Non-transitory computer-readable storage media (devices) includes RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.
A “network” is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media can include a network and/or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of computer-readable media.
Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to non-transitory computer-readable storage media (devices) (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a “MC”), and then eventually transferred to computer system RAM and/or to less volatile computer storage media (devices) at a computer system. Thus, it should be understood that non-transitory computer-readable storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.
Computer-executable instructions comprise, for example, instructions and data which, when executed by a processor, cause a general-purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. In some embodiments, computer-executable instructions are executed on a general-purpose computer to turn the general-purpose computer into a special purpose computer implementing elements of the disclosure. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.
Those skilled in the art will appreciate that the disclosure may be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, and the like. The disclosure may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.
Embodiments of the present disclosure can also be implemented in cloud computing environments. In this description, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be employed in the marketplace to offer ubiquitous and convenient on-demand access to the shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly provisioned via virtualization and released with low management effort or service provider interaction, and then scaled accordingly.
A cloud-computing model can be composed of various characteristics such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, and so forth. A cloud-computing model can also expose various service models, such as, for example, Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). A cloud-computing model can also be deployed using different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, and so forth. In this description and in the claims, a “cloud-computing environment” is an environment in which cloud computing is employed.
As shown in
In particular embodiments, the processor(s) 9502 includes hardware for executing instructions, such as those making up a computer program. As an example, and not by way of limitation, to execute instructions, the processor(s) 9502 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 9504, or a storage device 9506 and decode and execute them.
The computing device 9500 includes memory 9504, which is coupled to the processor(s) 9502. The memory 9504 may be used for storing data, metadata, and programs for execution by the processor(s). The memory 9504 may include one or more of volatile and non-volatile memories, such as Random-Access Memory (“RAM”), Read-Only Memory (“ROM”), a solid-state disk (“SSD”), Flash, Phase Change Memory (“PCM”), or other types of data storage. The memory 9504 may be internal or distributed memory.
The computing device 9500 includes a storage device 9506 including storage for storing data or instructions. As an example, and not by way of limitation, the storage device 9506 can include a non-transitory storage medium described above. The storage device 9506 may include a hard disk drive (HDD), flash memory, a Universal Serial Bus (USB) drive or a combination these or other storage devices.
As shown, the computing device 9500 includes one or more I/O interfaces 9508, which are provided to allow a user to provide input to (such as user strokes), receive output from, and otherwise transfer data to and from the computing device 9500. These I/O interfaces 9508 may include a mouse, keypad or a keyboard, a touch screen, camera, optical scanner, network interface, modem, other known I/O devices or a combination of such I/O interfaces 9508. The touch screen may be activated with a stylus or a finger.
The I/O interfaces 9508 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, I/O interfaces 9508 are configured to provide graphical data to a display for presentation to a user. The graphical data may be representative of one or more graphical user interfaces and/or any other graphical content as may serve a particular implementation.
The computing device 9500 can further include a communication interface 9510. The communication interface 9510 can include hardware, software, or both. The communication interface 9510 provides one or more interfaces for communication (such as, for example, packet-based communication) between the computing device and one or more other computing devices or one or more networks. As an example, and not by way of limitation, communication interface 9510 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI. The computing device 9500 can further include a bus 9512. The bus 9512 can include hardware, software, or both that connects components of computing device 9500 to each other.
In the foregoing specification, the invention has been described with reference to specific example embodiments thereof. Various embodiments and aspects of the invention(s) are described with reference to details discussed herein, and the accompanying drawings illustrate the various embodiments. The description above and drawings are illustrative of the invention and are not to be construed as limiting the invention. Numerous specific details are described to provide a thorough understanding of various embodiments of the present invention.
The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. For example, the methods described herein may be performed with less or more steps/acts or the steps/acts may be performed in differing orders. Additionally, the steps/acts described herein may be repeated or performed in parallel to one another or in parallel to different instances of the same or similar steps/acts. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.
This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63/378,616 filed Oct. 6, 2022, which is incorporated herein by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63378616 | Oct 2022 | US |
