Patent Application
20230162424
This invention relates to a computer implemented method of simulating a three-dimensional (3D) object for display in a browser on a client's device. This invention further relates to a computer program product, and an in-browser robotics 3D object simulator.
Simulation of three-dimensional objects, and in particular simulation of robotics, is a multi-billion-dollar business with substantial potential. It is highly advantageous to provide customers with the ability to render their robotics in a web browser where they can create, view and share their simulations. In this way, the simulations can be used for a wide array of different purposes, from training to testing, evaluation to education.
There are however a number of problems with the existing offerings. Heretofore, the existing offerings have been quite limited. Generally speaking, due to the computational burden placed on the processor, the simulations for all but the most basic robots and three-dimensional objects are necessarily at a low frame rate when being executed on a customer's machine. This low frame rate tends to lead to a jittery, stop-start simulation that is difficult to properly evaluate and appreciate.
Secondly, heretofore, the simulations for delivery though a browser have been limited by the use of relatively old physics engines, such as, but not limited to, Bullet v.2.82 (released 2013). Although useful for many projects, Bullet 2.82 is quite limited in its application compared with newer physics engines and does not support functionality typically supported by more modern non-web-based physics engines. Furthermore, it is generally accepted that development with in-browser physics engines was a difficult task that required large amounts of complex code. This further slowed down the processing of the simulations.
It is an object of the present invention to provide a computer implemented method, a computer program product, and an in-browser robotics simulator that overcome at least one of the above-identified problems. It is a further object of the present invention to provide a useful alternative choice for the consumer.
According to the invention there is provided a computer implemented method of simulating a three-dimensional (3D) object for display in a browser on a client's device, the method comprising the initial steps of:
on a web page load request issued through the browser on the client's device:
By having such a method, it will be possible to provide an in-browser simulation that can operate at approximately three times the frames per second (FPS) rate than previous offerings could achieve. This is due in part to the simulation API linking the physics engine and the rendering engine that will reduce the simulation code by up to in the region of 80%. In other words, the code will be of the order of 20% the size of the former code. This places less of a computational burden on the client device and it is possible to run at a higher FPS rate. In addition, due to the fact that the simulation runs client side and in browser, it is scalable and more cost effective to provide than previous offerings.
In one embodiment of the invention there is provided a computer implemented method in which the step of importing the physics library to the client's device comprises importing a subset of the available physics libraries to the client's device. In this way, the memory requirement on the client device will be reduced as well as the bandwidth required to transmit the physics libraries.
In one embodiment of the invention there is provided a computer implemented method in which the method comprises the step of determining which of the physics libraries is required for simulation of the 3D object and importing only those physics libraries required for the simulation of that 3D object. This is seen as a simple yet effective way of reducing the memory and bandwidth requirements.
In one embodiment of the invention there is provided a computer implemented method in which the method comprises storing the 3D object and the physics libraries on a content provider's web server. This is seen as a useful implementation as the content provider, the operator of the web server, can then deliver the content from their own web server in a more efficient manner. The content provider can also determine who has access to the content and when.
In one embodiment of the invention there is provided a computer implemented method in which the method comprises the step of storing the simulation application programming interface (API) and the simulation computer program code in a content delivery network (CDN). In this way, it will be possible to modify and update the API and simulation computer program code without recourse to the end users. Furthermore, this will also make it easier for the operator of the web server as they do not have to update existing files on the web server.
In one embodiment of the invention there is provided a computer implemented method comprising the step of running the physics engine in a web worker. This is seen as a useful implementation of the invention as the physics engine can be operating in the background and will be unimpeded by other operations on the web page.
In one embodiment of the invention there is provided a computer implemented method comprising the step of simulating the 3D object using a multi-threading technique in the physics engine. By using a multi-threading technique, this can help to provide a smoother simulation for the user with more effective use of available resources.
In one embodiment of the invention there is provided a computer implemented method comprising the step of compiling a version for each of a plurality of thread configurations, and on start of simulation, determining the version that is most suited to the client computer and running that version on the client computer. The number of available threads will depend on the user machine. By providing a plurality of versions, the version with the number of threads most suited to the machine it is running on can be determined and that version used in the simulation.
In one embodiment of the invention there is provided a computer implemented method comprising the step of exporting the simulation as a shareable web link. This is seen as a simple way of sharing the simulation.
In one embodiment of the invention there is provided a in-browser robotics 3D object simulator comprising:
By having such a simulator, it will be possible to provide an in-browser simulation that can operate at approximately three times the frames per second (FPS) rate than previous offerings could achieve. This is due in part to the simulation API linking the physics engine and the rendering engine that will reduce the simulation code to of the order of 20% the size of the former code. This places less of a computational burden on the client device and it is possible to run at a higher FPS rate. In addition, due to the fact that the simulation can run client side and in browser, it is scalable and more cost effective to provide than previous offerings.
In one embodiment of the invention there is provided an in-browser robotics 3D object simulator in which the physics engine is a port of a C++ physics engine. This is seen as a particularly useful implementation of the present invention as a modern, more powerful physics engine can be used to good effect. This will enhance the functionality of the offering.
In one embodiment of the invention there is provided an in-browser robotics 3D object simulator in which the physics engine is an emscripten port of a C++ physics engine. The emscripten port of the C++ engine will be particularly useful as it will allow the direct use of the physics engine in the web browser.
In one embodiment of the invention there is provided an in-browser robotics 3D object simulator in which the physics engine is PhysX. The PhysX physics engine, for example the PhysX 4.1 produced by Nvidia Corporation of Santa Clara, Calif., United States of America, is seen as a particularly suitable physics engine to use in the present invention that will allow for a richer, more immersive experience for the user.
In one embodiment of the invention there is provided an in-browser robotics 3D object simulator in which the rendering engine comprises a web graphics library (WebGL).
In one embodiment of the invention there is provided an in-browser robotics 3D object simulator in which the rendering engine comprises Three.js. The combination of PhysX and Three.js is seen as particularly preferred and leads to up to three times more FPS than other solutions.
In one embodiment of the invention there is provided an in-browser robotics 3D object simulator in which the physics engine is configured to support multi-threading. By using a multi-threading technique, this can help to provide a smoother simulation for the user with more effective use of available resources.
In one embodiment of the invention there is provided an in-browser robotics 3D object simulator in which the physics engine is configured to operate in a web worker. This is seen as a useful implementation of the invention as the physics engine can be operating in the background and will be unimpeded by other operations on the web page. The use of a web worker will increase code efficiency. The web worker will make it possible to run a script operation in a background thread separate from the main execution thread of a web application. The advantage of this is that intensive processing may be performed in the background in a separate thread, allowing the main thread to run unimpeded.
In one embodiment of the invention there is provided an in-browser robotics 3D object simulator in which there is provided an accessible memory and a plurality of 3D objects stored in the accessible memory.
In one embodiment of the invention there is provided a computer program product having program instructions loaded thereon that when executed on a computer, cause the computer to carry out one or more of the method steps of:
By having such a computer program product, it will be possible to provide an in-browser simulation that can operate at approximately three times the frames per second (FPS) rate than previous offerings could achieve. This is due in part to the simulation API linking the physics engine and the rendering engine that will reduce the simulation code to of the order of 20% the size of the former code. This places less of a computational burden on the client device and it is possible to run at a higher FPS rate. In addition, due to the fact that the simulation can run client side and in browser, it is scalable and more cost effective to provide than previous offerings.
In one embodiment of the invention there is provided a computer program product comprising program instructions loaded thereon that when executed on a computer, cause the computer to carry out one or more of the preparatory method steps of:
The invention will now be more clearly understood from the following description of some embodiments thereof given by way of example only with reference to the accompanying drawings, in which:—
Referring to
In use, the method comprises the initial steps of the operator generating a 3D object and storing the 3D object in memory 103. Furthermore, the operator stores a plurality of physics libraries in memory 103 along with a simulation application programming interface (API) linking a physics engine and a rendering engine, and a simulation computer program code. The simulation API and simulation computer program code may be stored in the CDN 109, whereas the 3D object and physics libraries may be stored on web server 119.
The simulation API is the code library, portions of reusable code that can be run using the API. The simulation computer program code is the code that is written to create a particular simulation of a 3D object. The simulation computer program code makes use of the simulation API library and will contain functions that can be found in the API library. Therefore, it will be necessary to load the API in order to run the simulation code. As indicated above, the API reduces the amount of code required. By this, what is meant is that the amount of code that is required to be written by the developer, i.e. the simulation computer program code. Strictly speaking, it is the features of the code library that allow for the code reduction as the API is the point of contact used for interaction between two running codes. The code library is written using best practices in that it is as efficient as possible.
The client issues a web page load request using a browser (not shown) on the client's device 105. The method then comprises the steps of loading the simulation application programming interface (API) and the simulation computer program code onto the client's device 105 from the memory 103, (or from the CDN 109), importing the 3D object and at least one of the physics libraries to the client's device 105 from the web server 119, and simulating the 3D object using the simulation API, the rendering engine and the physics engine having access to the imported physics library. The 3D object is rendered in the client's browser, and, for each subsequent frame, the rendered 3D object is updated by using the physics engine to generate position and rotation data for the 3D object, that position and rotation data is sent to the rendering engine, and the rendering engine updates the rendered 3D object based on the position and rotation data received from the physics engine before displaying the updated rendered 3D object (as will be described in more detail below with respect to
Instead of client device 105, the client may use any of the client devices 111, 113, 115 shown, or indeed another client device capable of accessing the web server and displaying a web page. For example, a smartphone, an internet enabled television, a tablet, a phablet, a smart watch or the like could also be used to view the simulation.
Once the operator has completed the simulation of the content provider's robot, the operator provides the content provider with API documentation that will allow the content provider to easily control their simulation with Javascript (Registered Trade Mark®) commands. The following steps are then taken in order to integrate the simulation into the content provider's web page.
In step 1, the operator provides the content provider with a folder of files containing any 3D assets and the physics library files. (Note: Any files that contain operator source code may be obfuscated using the tool https://obfuscator.io/.) In step 2, the content provider places these files in a folder on their web server 119. In step 3, the operator will host a CDN (only part of which, computing device 109, is shown, that contains the simulation API as well as the simulation code. This must be loaded onto the client device in order to run the simulation. This allows the operator to easily make updates to the code. In step 4, the content provider will integrate the simulation into their web page with code similar to the following code segment 1, shown below:
The content provider is thereafter ready to begin simulations through their web server to their customers (operating a client device).
From a technical perspective, the method according to the present invention enables robotics simulations that were not previously possible to run in a native web browser. This is due in part to the port into Javascript® of Nvidia's® PhysX® 4.1 physics library. Until now, the most powerful in-browser engine has been Ammo.js which, as of March 2020, is a Javascript port of a 2013 physics engine. The use of PhysX 4.1 allows for new simulation possibilities, such as tank physics, 4WD vehicle physics, joint articulation physics (for simulating robotics such as Robot Arms), wheel slip physics and more. The Javascript port of PhysX also offers significant improvements in friction and performance with high numbers of simulated objects. Previously, development with in-browser physics engines was a difficult task that required large amounts of complex code. For example, a simple demo displaying a number of pyramids (as shown below in
In order to port PhysX, in one embodiment Emscripten was used. Emscripten (www.emscripten.org) is a source-to-source compiler used to compile C and C++ code into JavaScript.
The method and in particular the API component of the present invention makes it possible to add simulated bodies to a 3D scene with minimal amounts of code required. In order to add a simulated entity, whether that is a simple cube or a 4WD vehicle, it often only requires a single line of code which contains the configuration options for the simulated entity. For example, to add a cube to the simulation using the API according to the present invention, the following code segment 2 would be required:
In order to add a cube without the API according to the present invention, the code would look something like code segment 3 below:
It can be seen therefore that the computation overhead and time to process the code will be significantly reduced. Another advantageous aspect of the API forming part of the present invention is that the API allows for automatic linking between the rendering and physics engine and provides simple API commands to do so. An example to add a 3D model with physics collisions using the API is illustrated in Code Segment 4 below:
In addition to the foregoing, the present invention also makes it possible to run multithreaded physics in a web browser which helps with performance when simulating a large number of objects at once. The simulator runs physics in a Javascript web worker by default, and there is no extra configuration required for convenience. In the present implementation of the invention, phy.js is being run in the web worker. Using the following code content from the code library:
a worker will be run using phy.min.js if the isExport Boolean is true, otherwise it will run phy.js in the Worker. It is possible to enable physics multithreading using the port of PhysX according to the invention. Multithreading is only supported on Google Chrome browser at the moment however it is envisaged that extension to other browsers will be readily understood to the skilled addressee. The only change required to allow for multithreading is the following line of code, shown as code segment 5 below, illustrating the use of three threads, along with the alternative line of code for no multithreading:
These are functions that are built into the PhysX library. While this can increase performance on certain devices, there are some limitations of multithreading using the port of PhysX developed according to the present invention. First of all, the program memory cannot dynamically “grow”, so this means that the memory requirements are hardcoded at compilation time (For example: 128 MB). This is an Emscripten specific limitation. In order to circumvent the problem, it is possible to do a malloc/new (dynamic memory allocation), but it cannot exceed the hardcoded limit. This is troublesome because the operator must have a good understanding of the scene memory requirements, or allocate a high memory limit from the beginning (which can be a waste of resources). Secondly, the number of threads must also be hardcoded at compilation time (For example: 2 threads, etc.). This is because of PhysX itself and not Emscripten which is able to allocate threads dynamically. But this is problematic too, as the number of available threads depend on the user machine. For this problem, the following workaround may be used: as detecting the number of threads is possible from Javascript, if a version for each typical thread configuration (1,2,4,8) is complied, there will be the opportunity to start the version that would best suit the host platform. In this way, the resources are used more effectively and the code will be processed more efficiently.
Referring now to
It will be understood from the foregoing that there are numerous uses and benefits for the present invention. For example, it is envisaged that the 3D object may be a piece of equipment, the web server may be operated by a manufacturer/supplier of assembly line equipment and the client device may be operated by a potential customer. The potential customer can simulate the operation of the equipment to determine whether or not it will function as required and what clearances etc are required. Alternatively, the content provider may be a provider of online games and the client device may be operated by a gamer, operating a robot or other device in a challenge against the clock or other users. Alternatively still, the web server could be operated by a company or organisation for their design team (internal, external or both internal and external) and the design team may be able to access the simulation, evaluate the simulation and see if and how it could be improved prior to going into production. Similarly, the 3D object could be a vehicle such as an airplane, helicopter or spacecraft, or several vehicles simulated together such as cars, motorcycles, trucks, boats or combinations thereof, and the simulation could be used by a crash investigation team determining likely cause of failure and/or likely sequence of events in a crash or the like.
It will be further understood that various parts of the present invention are performed in hardware and other parts of the invention may be performed either in hardware and/or software. It will be understood that the method steps and various components of the present invention will be performed largely in software and therefore the present invention extends also to computer programs, on or in a carrier, comprising program instructions for causing a computer or a processor to carry out steps of the method or provide functional components for carrying out those steps. The computer program may be in source code format, object code format or a format intermediate source code and object code. The computer program may be stored on or in a carrier, in other words a computer program product, including any computer readable medium, including but not limited to a floppy disc, a CD, a DVD, a memory stick, a tape, a RAM, a ROM, a PROM, an EPROM or a hardware circuit. In certain circumstances, a transmissible carrier such as a carrier signal when transmitted either wirelessly and/or through wire and/or cable could carry the computer program in which cases the wire and/or cable constitute the carrier.
It will be further understood that the present invention may be performed on two, three or more devices with certain parts of the invention being performed by one device and other parts of the invention being performed by another device. The devices may be connected together over a communications network. The present invention and claims are intended to also cover those instances where the system and/or method is operated across two or more devices or pieces of apparatus located in one or more locations.
In this specification the terms “comprise, comprises, comprised and comprising” and the terms “include, included, includes and including” are all deemed interchangeable and should be afforded the widest possible interpretation.
The invention is not limited to the embodiments hereinbefore described but may be varied in both construction and detail within the scope of the appended claims.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2020/061530 | 4/24/2020 | WO |
