Patent Application
20220365657
The present invention relates to lighting probe placement, in particular using machine learning.
The “background” description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description which may not otherwise qualify as prior art at the time of filing, are neither expressly or impliedly admitted as prior art against the present invention.
Currently, global illumination (GI) and reflection probe placement tends to be a long process that starts with an initial placement of probes, for example in a grid pattern, followed by a process of elimination for example to remove probes that collide with geometry, cause light leaking, and so on. This requires the supervision of an artist or designer who examines the scene and moves or eliminates probes in an effort to improve the appearance of a lit scene. This process is then repeated in an ad-hoc manner for each scene in a game (or other rendered environment, such as in a film).
The results may be visually satisfying, but are laborious and typically still include an unnecessary number or placement of probes, increasing computational overhead.
The present invention seeks to address or mitigate this problem.
In a first aspect, a lighting probe placement system for estimating lighting probe positions within a virtual environment is provided in accordance with claim 1.
In another aspect, a method of estimating lighting probe positions within a virtual environment is provided in accordance with claim 13.
In another aspect, a method of training a machine learning system to use in any one of the preceding aspects is provided in accordance with claim 15.
Various aspects and features of the present invention are defined in the appended claims and within the text of the accompanying description.
It is to be understood that both the foregoing general description of the invention and the following detailed description are exemplary, but are not restrictive, of the invention.
A more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:
A system and method of lighting probe placement are disclosed. In the following description, a number of specific details are presented in order to provide a thorough understanding of the embodiments of the present description. It will be apparent, however, to a person skilled in the art that these specific details need not be employed to practice the present invention. Conversely, specific details known to the person skilled in the art are omitted for the purposes of clarity where appropriate.
In an example embodiment of the present description, an entertainment system such as a videogame console, or equivalently a development kit for such an entertainment system, or a computer or server used in development for such an entertainment system may implement a method of illumination and reflection probe placement as described elsewhere herein.
Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views, then
The entertainment system 10 comprises a central processor 20. This may be a single or multi core processor, for example comprising eight cores. The entertainment system also comprises a graphical processing unit or GPU 30. The GPU can be physically separate to the CPU, or integrated with the CPU as a system on a chip (SoC).
The entertainment device also comprises RAM 40, and may either have separate RAM for each of the CPU and GPU, or shared RAM. The or each RAM can be physically separate, or integrated as part of an SoC. Further storage is provided by a disk 50, either as an external or internal hard drive, or as an external solid state drive, or an internal solid state drive.
The entertainment device may transmit or receive data via one or more data ports 60, such as a USB port, Ethernet® port, WiFi® port, Bluetooth® port or similar, as appropriate. It may also optionally receive data via an optical drive 70.
Interaction with the system is typically provided using one or more handheld controllers 80, but may be via a mouse/keyboard, gestures or other suitable user interfaces.
Audio/visual outputs from the entertainment device are typically provided through one or more A/V ports 90, or through one or more of the wired or wireless data ports 60.
Where components are not integrated, they may be connected as appropriate either by a dedicated data link or via a bus 100.
Operating under suitable software instruction, the entertainment device or equivalent development kit, computer or server, may then implement a method of global illumination and reflection probe placement as follows.
It will be appreciated that a typical virtual scene, which may be for use in a video game, or as a TV or movie set, may comprise ambient light (for example generated by a so-called sky box) and/or directional light (typically from in-environment light sources).
However a further form of lighting comes from global illumination (‘GI’), which accounts for secondary light sources such as bounced or reflective light, or optionally emissive light from textured objects. Similarly reflection can redirect light toward other surfaces and toward the virtual camera.
The calculation of such secondary light can be performed using global illumination probes (‘GI probes’) and optionally also reflection probes, which collectively may be more generally referred to as lighting probes.
The lighting interaction is typically with static components within the scene, although not necessarily so.
In a typical example, a red cube on a white surface will result in a pink/red tint to the white surface around the cube due to GI. Meanwhile depending on the surface properties of the cube, the surface, or another object, a sharp or diffuse reflection may be seen on the surface of one or more of them.
As noted above, the placement of probes to enable good quality GI, and optionally reflection, can be laborious, and the process of calculating global illumination, and optionally reflection, can be time consuming and is advantageously reduced if there are fewer iterations of GI probe placement or optionally reflection probe placement in this process.
As noted previously, current implementations of GI and reflection probe placement are slow, and involve iteratively moving probes (and sometimes removing or adding probes), under the supervision of an artist or designer, followed each time by re-computation of the lighting in the scene—which itself can be slow—and rendering an image for review. This has to be done ad hoc for every scene in every game. The results are usually visually satisfying, but not always optimal in terms of the number or position of probes. Hence both the process itself and the final result are typically more computationally intensive than is necessary.
Accordingly, embodiments of the present description seek to address or mitigate this issue by determining a suitable GI and optionally reflection probe placement with much less artist supervision required.
Automatic probe placement is performed by a machine learning (‘ML’) system trained on probe placement examples that reduce and preferably minimise graphics artefacts, and similarly reduce and preferably minimise the number of probes needed for a scene.
The ML system is provided as input with descriptive data of the virtual scene.
Typically this comprises geometry data for that part of the scene to be illuminated based on the current set of probes. This may represent only a part of the scene (e.g. for a large environment using successive sets of probes) or the full scene (e.g. for a smaller environment such as a typical room). Preferably the geometry data is the same as that used for rendering the scene.
Such geometry data can be presented to the ML system for example as a scene graph structure.
Optionally other data may also be provided, such as surface colour data associated with the geometry or derived from textures to be applied to the geometry in the rendering process.
Optionally, this input data can be subjected to any suitable pre-processing, such as dimension reduction (e.g. using an autoencoder, or a non-square matrix multiplication) and other techniques common to machine learning schemes. Dimension reduction, for example, can help to provide a consistent input data size for the machine learning system.
The ML system is trained to output a vector or other data structure comprising position information for one or more probes, and optionally one or more other items of information that may be associated with a probe, such as metadata for occlusions, or one or more probe's influence radius.
The ML system may be trained to output data just for GI probes, or GI and reflection probes, or just for reflection probes. In other words, a machine learning system can be trained just for GI probe placement (with optional additional probe data) or for both GI and reflection probes, either a second ML system or a combined ML system can be used.
Hence the ML system comprises an input suitable to receive geometry data and optionally additional data, or a pre-processed and/or dimension-reduced version thereof, and comprises an output suitable to provide position data and optionally additional data (such as probe influence radius) for the placement of a suitable number of probes.
A typical architecture for such an ML may thus comprise an input layer for the inputs, at least a first hidden layer, and an output layer for the outputs, with a connectivity scheme between the layers that enables an internal representation of the relationship between virtual scenes and probe placements to be modelled. Examples of such an ML include a supervised or semi-supervised neural network architecture, or a reinforcement learning neural network architecture. A deep learning neural network may similarly be considered suitable.
As noted above, the ML uses geometry based input data and generate probe position data as an output. Consequently training data for such an ML comprises at least scene geometry data and preferred probe position data.
The training data may be obtained using one or more of the following techniques.
Firstly, where a virtual scene or set for a TV, film, interactive experience, or game has already been processed to have GI and optionally reflection probes, for example using artistic supervision/curation, then the scene geometry and any optional data, together with the curated probe positions, may be used as training data. Preferably training data from a plurality of such virtual scenes may be used, and more preferably training data from a plurality of different TV, film, interactive experience, or game titles may be used, as these are likely to reflect different lighting requirements, geometric layouts, and optionally colour palettes (e.g. if textures or representations thereof are part of the input).
This approach may be of particular benefit when using similar assets and environments to a previous virtual scene, for example when developing a sequel to or expansion of existing content.
Alternatively or in addition, an ML system may be trained with new training data based for example on a representative sample of virtual scenes from the new content. This new training data may also be generated in the supervised and curated manner described previously herein. However, for a system intended to minimise or avoid the need for such supervision, it may be reasonable to assume that such supervised curation may not be available for the purposes of generating training data.
Therefore it is preferable to be able to generate such training data automatically. It will be appreciated that if it is possible to generate probe position data automatically, then the approach used to generate the training data itself could in theory be used to generate actual probe placement solutions for the virtual scene. However, as explained below, the available techniques are computationally very expensive.
Firstly, for a given training pair of inputs and outputs, it is useful to generate a ground truth or ideal image in which the virtual scene is rendered fully including lighting and optionally reflection, for example by using ray tracing.
Any subsequent automatic process for approximating this image using global illumination probes and optionally reflection probes during a render can then be evaluated against this ground truth. A particular distribution of global illumination probes and optionally reflection probes may then be deemed the target distribution if the resulting rendered image best matches the ground truth image, or matches it to a threshold degree. In this latter case, it will be appreciated that potentially a number of probe distributions for the same scene may generate rendered images that match the ground truth image to a threshold degree; in this case optionally some or all of these distributions may be used for training, which in turn may improve the resilience and robustness of the machine learning system, but possibly at a cost of a slight reduction in fidelity due to also learning acceptable but less optimal distributions than that of the best matching rendered image.
Algorithms for generating such probe distributions include the following.
i. Brute Force Algorithm
Such an algorithm starts with the scene and places probes randomly or following a predetermined sampling pattern such a uniform sampling, e.g. on a volumetric grid. Alternatively the initial random or grid based distribution can be chosen responsive to the existing scene geometry, for example with the probe density of a given volume being a function of the number of polygons of the scene geometry falling within that volume, with more geometry giving rise to more (or more densely placed) probes.
The scene is then rendered using the preferred rendering method (typically the production method, i.e. that which will be used during use of the virtual scene, such as traditional rasterization based rendering) and an image is produced.
This image is compared with the ray traced ground truth image, rendered from the same camera position, and an error or difference between the two images is calculated, as described later herein. This is repeated until the error is minimised or is less than a threshold amount, and the corresponding probe number and placement is recorded.
An example of such an algorithm is the random search algorithm. After having found a probe placement for that scene with the best (or threshold-acceptable) result, the probe positions and optionally metadata of the ideal placement are used as output targets of the training data for the ML model in conjunction with the geometry of the scene (without probes) as the input.
ii. Genetic or Random Noise Algorithm
Such an algorithm starts in a similar manner to the brute force algorithm with regards to initial placement and rendering, but subsequently the positions of one or more probes are perturbed (e.g. randomly) by a small amount (e.g. up to half the separation distance from neighbouring probes) between each test render. Perturbations that reduce the error are kept, whilst those that do not are discarded. The size of the perturbations can be reduced either as a function of the number of successive tests and/or as a function of the error size, becoming smaller as the error becomes smaller.
iii. Mock Gradient Descent Algorithm
Whilst it is difficult to directly relate a probe-position based gradient to the error in a rendered image, it will be appreciated that when a probe is moved in a certain direction, and the error is reduced, it makes sense to move it in that direction again (or optionally in a random direction falling within a cone having an axis parallel with that direction and a bounding surface at a predetermined angle, such as 15, 30, 45 or 60 degrees).
It will also be appreciated however that when multiple probes are moved at the same time, the error may reduce but not necessarily due to the contribution from any one probe's movement, which may actually be detrimental. Therefore the algorithm comprises moving a random subset of probes for each iteration, determining if the error goes up or down, and then moving another random subset of probes—but where a probe is selected again, either moving it in the same direction or within the cone if its previous use corresponded with a reduction in error, and either in the opposite direction or randomly (optionally randomly in a range outside the cone or in a hemisphere opposite the cone) if its previous use corresponded with an increase in error. This approach acts as a form of noisy simultaneous equation approximating a gradient descent, causing probes to tend to move in a direction that reduces the error whenever they are next selected.
Optionally for any of the above techniques, a probe may be randomly turned off, or a new probe randomly added. This may occur at a longer interval than each image iteration. As it is preferable to use fewer probes (to reduce computational overhead), then if turning a probe off does not increase the error (or only increases it by less than a second predetermined amount), optionally by after N subsequent iterations, then it may be kept off. Similarly if adding a probe (for example midway between three other probes in a volume) does not decrease the error by more than a third predetermined amount, optionally by after N subsequent iterations, then it may be removed again.
Each of the above algorithms may optionally start with a small number of probes (for example between 50 and 80% of the number normally used by a human supervisor in a scene of a similar size and/or polygon count) and slowly increase the number over time, evaluating the corresponding return on error performance until for example the marginal benefit of the last probe is below the third predetermined threshold.
Optionally for any of the above techniques, the training data may also include a probe's influence radius. This may be fixed, so that achieving a threshold error is based primarily on placement and number of probes, or may also be make a variable that can be changed according to the same principles as placement in each algorithm, so that the radius of influence of a given probe can also be adjusted. Optionally different algorithms can be employed to refine placement and radius of influence. If a probe is turned on, it can be initialised with a default radius, or if it was previously turned off, optionally can use its previous radius value.
The above algorithms use an error measurement between the rendered image and the ground truth image to determine the suitability of a candidate probe distribution in the virtual scene.
This error measurement may be any suitable comparative measure between two data sets (e.g. image pixels), such as an absolute pixel value difference (because the images are of the same scene and from the same viewpoint, the only difference in pixel values will be due to the lighting).
Preferably a perceptually based comparison of the images may be employed, on the basis that minimising such a metric may also minimise a perceived difference by a user.
Known metrics suitable for perceptually based comparison are the so-called L1 and L2, which are specific instances of the Minkowski metric (See https://en.wikipedia.org/wiki/Minkowski_distance). A description of its use for perceptually based image comparison may be found for example in ‘A perceptually based comparison of image similarity metrics’, Sinha, P and Russel, R; Perception, 2011, volume 40, pages 1269-1281 (http://web.mit.edu/sinhalab/Papers/sinha_russell2011.pdf), incorporated herein by reference.
In summary, the L1 metric is the Manhattan distance or the sum of absolute error, whilst L2 is the Euclidean distance of sum of square error, and hence can be seen as variants on the above approach of absolute pixel value difference. L1 is computationally cheaper to implement. Preference studies also suggest that L1 will produce a slightly better approximation of user perception preferences. Hence preferably the error is based on the L1 metric, but it will be appreciated that L2, or absolute pixel value difference, or any other comparison scheme may be used.
As an example of an alternative scheme, an adversarial network may be trained on target images (e.g. images from existing games, or real-world images) to recognise images that look real, in terms of their lighting. Images rendered using a current distribution of probes could then be scored by the network, with higher scoring images being deemed more realistic. Again, because the scene and viewpoint would not change between iterations of probe placement, any change in score would effectively be independent of the content of the scene itself, and be due to the lighting.
Using any suitable combination of the above approaches, may thus be used to generate training images for the machine learning system. It will be appreciated the as a result the training set is computationally expensive to produce (though less than using these techniques for every scene or viewpoint in production). As a result the training set may be relatively small. To increase the training set size, as noted above images that are within a threshold error of the ground truth image may be used, thereby creating multiple examples for each ground truth image generated and compared against, and similarly data from other projects (typically using the same game engine) can be pooled to increase the size of the training set.
In this latter case, different developers may not want to share geometry data or image data with each other. However, note that the training set itself only requires the target probe placements, not the final image, and the input geometry data can be subjected to a dimension reduction scheme as part of pre-processing, making the geometry effectively unusable by the game engine itself but still suitable for training.
As also noted above, additional sources of training data include previously supervised and curated probe placements, which may be particularly of use for example when developing a sequel or expansion to previous content
The ML is then trained using the training set to output predicted numbers of GI probes, and optionally reflection probes, and further optionally any suitable metadata associated with such probes. The number of generated probes may be fixed (e.g. within the training set and subsequent output) or may vary. In this case, a null placement (for example a placement within a particular region of the scene such as within an object volume, or outside of a scene wall) may signify non-use of a probe both during training and output, when the nodes or other output scheme of the ML are otherwise fixed.
The trained ML can then be integrated into a game engine editor or similar, and provide good quality probe placement for any subsequent content that follows the same physically based lighting model as that used in the ray tracing scheme. Optionally an artist may still then adjust the probe positions if desired, but this process will take much less time than if starting from an initial random or grid based layout. It will also be appreciated that any such changes may also be used to refine the training of the ML.
As noted elsewhere herein, traditionally lighting probe placement has to be done ad hoc for every scene in every game, under manual supervision, resulting in slow throughput and sometimes less than optimal probe usage. Hence both the process itself and the final result are typically more computationally intensive than is necessary.
Embodiments of the present description seek to address or mitigate this issue by determining a suitable GI and optionally reflection probe placement with much less artist supervision required. This advantageously reduces computational load in the selection of probe placement and optionally improves the number of probes selected. Furthermore it typically also improves memory usage, either by virtue of fewer probes or because fewer iterations of probe layout need to be stored during the decision making process. In particular with fewer probes, less computation is required both at so-called ‘baking time’ and also when traversing the virtual environment to find probes.
In a summary embodiment of the present description, a lighting probe placement system (such as entertainment system 10 or a development kit thereof) for estimating lighting probe positions within a virtual environment comprising the following.
Firstly, a geometry input processor (such as CPU 20, GPU 30, or any suitable combination thereof) configured (for example by suitable software instruction) to receive geometry of at least part of the virtual environment and prepare it for input to a machine learning system, as described elsewhere herein.
Secondly, a processor (such as CPU 20, GPU 30, or any suitable combination thereof) configured (for example by suitable software instruction) to implement the machine learning system, which has been trained to output lighting probe positions upon receiving input geometry of the at least part of the virtual environment, as described elsewhere herein.
And thirdly, an association processor (such as CPU 20, GPU 30, or any suitable combination thereof) configured (for example by suitable software instruction) to associate the output lighting probe positions with the at least part of the virtual environment for subsequent rendering, as described elsewhere herein.
It will be apparent to a person skilled in the art that variations in the above system corresponding to operation of the various embodiments of the method and/or apparatus as described and claimed herein are considered within the scope of the present disclosure, including but not limited to that:
Meanwhile a virtual content editing system may comprise the lighting probe placement system of any configuration described above, together with a rendering processor operable to render an image of at the least part of the virtual environment using an associated distribution of lighting probe positions generated by the machine learning system, and an output processor operable to output the rendered image for display to a user, as described elsewhere herein.
Turning now to
In a first step s210, receiving geometry of at least part of the virtual environment and preparing it for input to a machine learning system, as described elsewhere herein.
In a second step s220, providing the input geometry to a machine learning system that has been trained to output lighting probe positions, as described elsewhere herein.
And in a third step s230, associating the output lighting probe positions with the at least part of the virtual environment for subsequent rendering, as described elsewhere herein.
Again, it will be apparent to a person skilled in the art that variations in the above method corresponding to operation of the various embodiments of the method and/or apparatus as described and claimed herein are considered within the scope of the present disclosure.
Turning next to
In a first step s310, automatically generating a training set of input geometry and corresponding target probe positions using a lighting probe position evaluation scheme, as described elsewhere herein, the scheme comprising the following sub-steps.
And in a second step s320, training the machine learning system using input geometry and distributions of lighting probe positions associated with the corresponding at least part of the virtual environment, as described elsewhere herein.
Again, it will be apparent to a person skilled in the art that variations in the above method corresponding to operation of the various embodiments of the method and/or apparatus as described and claimed herein are considered within the scope of the present disclosure, including but not limited to that:
It will be appreciated that the above methods may be carried out on conventional hardware (such as the entertainment device 10 or a development kit thereof) suitably adapted as applicable by software instruction or by the inclusion or substitution of dedicated hardware.
Thus the required adaptation to existing parts of a conventional equivalent device may be implemented in the form of a computer program product comprising processor implementable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, solid state disk, PROM, RAM, flash memory or any combination of these or other storage media, or realised in hardware as an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array) or other configurable circuit suitable to use in adapting the conventional equivalent device. Separately, such a computer program may be transmitted via data signals on a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.
The foregoing discussion discloses and describes merely exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the disclosure of the present description is intended to be illustrative, but not limiting of the scope of the invention, as well as other claims. The disclosure, including any readily discernible variants of the teachings herein, defines, in part, the scope of the foregoing claim terminology such that no inventive subject matter is dedicated to the public.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2106698.0 | May 2021 | GB | national |
