3D object capture and object reconstruction using edge cloud computing resources

Abstract

Described herein are methods and systems for three-dimensional (3D) object capture and object reconstruction using edge cloud computing resources. A sensor coupled to a mobile device captures (i) depth maps of a physical object, the depth maps including pose information, and (ii) color images of the object. An edge cloud device coupled to the mobile device via a 5G connection receives the depth maps and the color images. The edge cloud device generates a new 3D model of the object based on the depth maps and color images, when a 3D model of the object has not been generated. The edge cloud device updates an existing 3D model of the object based on the depth maps and color images, when a 3D model of the object has previously been generated. The edge cloud device transmits the new 3D model or the updated 3D model to the mobile device.

Description

TECHNICAL FIELD

This subject matter of this application relates generally to methods and apparatuses, including computer program products, for three-dimensional (3D) object capture and object reconstruction using edge cloud computing resources

BACKGROUND

With the start of the 5-G deployments, wireless/mobile computing systems and applications can now take advantage of ‘edge’ cloud computing and offload computationally intensive processing to the cloud. This has significant benefits in terms of being able to run computer vision applications that previously would have been too slow or would have consumed too much battery power, due to the inherent limitations of mobile devices. This is especially true in the case of 3D computer vision processing applications such as the 3D scanning technology described in:

the real-time object recognition and modeling techniques as described in U.S. Pat. No. 9,715,761, titled “Real-Time 3D Computer Vision Processing Engine for Object Recognition, Reconstruction, and Analysis;”

the dynamic 3D modeling techniques as described in U.S. patent application Ser. No. 14/849,172, titled “Real-Time Dynamic Three-Dimensional Adaptive Object Recognition and Model Reconstruction;”

the 3D model generation techniques as described in U.S. Pat. No. 9,710,960, titled “Closed-Form 3D Model Generation of Non-Rigid Complex Objects from Incomplete and Noisy Scans;”

the 3D photogrammetry techniques described in U.S. Pat. No. 10,192,347, titled “3D Photogrammetry;”

the sparse SLAM techniques described in U.S. patent application Ser. No. 15/638,278, titled “Sparse Simultaneous Localization and Mapping with Unified Tracking;”

the 2D and 3D video compression techniques described in U.S. Pat. No. 10,380,762, titled “Real-Time Remote Collaboration and Virtual Presence using Simultaneous Localization and Mapping to Construct a 3D Model and Update a Scene Based on Sparse Data;”

the 3D geometry reconstruction techniques described in U.S. patent application Ser. No. 16/118,894, titled “Enhancing Depth Sensor-Based 3D Geometry Reconstruction with Photogrammetry;”

the 3D tracking techniques described in U.S. patent application Ser. No. 16/123,256, titled “Combining Sparse Two-Dimensional (2D) and Dense Three-Dimensional (3D) Tracking;”

the 4D hologram generation and control techniques described in U.S. patent application Ser. No. 16/240,404, titled “4D Hologram: Real-Time Remote Avatar Creation and Animation Control;” and

the object scanning techniques described in U.S. patent application Ser. No. 16/421,822, titled “Keyframe-Based Object Scanning.”

Each of the above-referenced patents and patent applications is incorporated by reference herein in its entirety.

These types of 3D computer vision processing applications require much more computer processing power than typical 2D vision applications. And, existing non-5G wireless networks and corresponding mobile devices are not capable of providing the processing bandwidth and power necessary to efficiently and quickly perform these applications that would minimize or avoid excessive battery consumption and/or heat generation of the mobile devices.

SUMMARY

Therefore, the technology described herein advantageously utilizes edge cloud computing available in new 5G-based wireless networks for more reliable and robust implementation of real-time 3D computer vision capture technology using mobile devices—while also reducing excessive heat generation and battery consumption of said devices.

The invention, in one aspect, features a system for three-dimensional (3D) object capture and object reconstruction using edge cloud computing resources. The system comprises a sensor device, coupled to a mobile computing device, that captures (i) one or more depth maps of a physical object in a scene, the depth maps including related pose information of the physical object, and (ii) one or more color images of the physical object in the scene. The system comprises an edge cloud computing device, coupled to the mobile computing device via a 5G network connection, that receives the one or more depth maps and the one or more color images from the mobile computing device. The edge cloud computing device generates a new 3D model of the physical object in the scene based on the received one or more depth maps and one or more color images, when a 3D model of the physical object has not yet been generated. The edge cloud computing device updates an existing 3D model of the physical object in the scene based on the received one or more depth maps and one or more color images, when a 3D model of the physical object has previously been generated. The edge cloud computing device transmits the new 3D model or the updated 3D model to the mobile computing device.

The invention, in another aspect, features a computerized method of three-dimensional (3D) object capture and object reconstruction using edge cloud computing resources. A sensor device, coupled to a mobile computing device, captures (i) one or more depth maps of a physical object in a scene, the depth maps including related pose information of the physical object, and (ii) one or more color images of the physical object in the scene. An edge cloud computing device, coupled to the mobile computing device via a 5G network connection, receives the one or more depth maps and the one or more color images from the mobile computing device. The edge cloud computing device generates a new 3D model of the physical object in the scene based on the received one or more depth maps and one or more color images, when a 3D model of the physical object has not yet been generated. The edge cloud computing device updates an existing 3D model of the physical object in the scene based on the received one or more depth maps and one or more color images, when a 3D model of the physical object has previously been generated. The edge cloud computing device transmits the new 3D model or the updated 3D model to the mobile computing device.

Any of the above aspects can include one or more of the following features. In some embodiments, the mobile computing device crops the captured depth maps and color images prior to transmitting the captured depth maps and color images to the edge cloud computing device. In some embodiments, cropping the captured depth maps and color images comprises removing a portion of the captured depth maps and color images that corresponds to a background of the scene. In some embodiments, the edge cloud computing device performs the generating step, the updating step, and the transmitting step within less than 10 milliseconds after receiving the one or more depth maps and the one or more color images from the mobile computing device.

In some embodiments, updating an existing 3D model comprises tracking the physical object in the scene based upon the pose information received from the mobile computing device. In some embodiments, transmitting the updated 3D model to the mobile computing device comprises providing, to the mobile computing device, tracking information associated with the physical object in the scene based upon the tracking step. In some embodiments, the physical object is a non-rigid object.

In some embodiments, the edge cloud computing device performs one or more post-processing functions on the updated 3D model. In some embodiments, the one or more post-processing functions comprise a bundle adjustment process, a de-noising process, a mesh refinement process, a texture alignment process, a shadow removal process, or a blending process. In some embodiments, the edge cloud computing device generates a final 3D model from the updated 3D model after performing the one or more post-processing functions. In some embodiments, the edge cloud computing device transmits the final 3D model to the mobile computing device and/or to a cloud-based server computing device.

In some embodiments, a latency of the 5G network connection between the mobile computing device and the edge cloud computing device is less than 5 milliseconds.

Other aspects and advantages of the technology will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating the principles of the technology by way of example only.

BRIEF DESCRIPTION OF THE DRAWINGS

The advantages of the technology described above, together with further advantages, may be better understood by referring to the following description taken in conjunction with the accompanying drawings. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the technology.

FIG. 1 is a block diagram of a system for three-dimensional (3D) capture using edge cloud computing resources.

FIG. 2 is a flow diagram of a dynamic real-time 3D scanning workflow, using the system of FIG. 1.

FIG. 3 is a flow diagram of a computerized method of utilizing edge cloud resources for object scanning and 3D model generation.

FIG. 4 is a flow diagram of a computerized method of utilizing edge cloud resources for 3D model post-processing and refinement.

FIG. 5 is a flow diagram of a computerized method of utilizing edge cloud resources for non-rigid 3D reconstruction.

DETAILED DESCRIPTION

FIG. 1 is a block diagram of a system 100 for three-dimensional (3D) capture using edge cloud computing resources. The system 100 includes a mobile device 102 (e.g., a smartphone, tablet, or other type of portable computing device that connects to a wireless network), a 5G base station 104, an edge cloud micro data center 106 (e.g., one or more computing devices on the ‘edge’ of the 5G network), and a cloud computing environment 108. As shown in FIG. 1, an application on the mobile device 102 connects to, and interacts with, the edge cloud micro data center 106 (e.g., via 5G base station 104) instead of having to connect to (or in addition to connecting to) the cloud computing environment 108. Because latency between the mobile device 102 and the edge cloud 106 is very small (e.g., less than 5 milliseconds) relative to the latency between the mobile device 102 and the cloud computing environment 108 (e.g., 100 milliseconds or more), the application on the mobile device 102 can perform much faster, in some cases with almost no latency. This is especially important for real-time computer vision applications which require very fast response between the processing resource (i.e., the edge cloud 106) and the user device (i.e., the mobile device 102).

Specifically, using the edge cloud 106 instead of the cloud computing environment 108 has following benefits for real-time 3D object capture and object reconstruction technology:

1. Object tracking is faster—because the edge cloud 106 provides a much more powerful computing resource (including, in some embodiments, a dedicated or shared GPU) versus a processor available in the mobile device 102, 3D object tracking applications provided by the mobile device are faster and more robust. This is because the edge cloud 106 can leverage its processing power to track at higher frames per second as well as use more robust 2D and 3D features—thus resulting in more accurate pose information used for 3D reconstruction.

2. 3D model quality is better—because the edge cloud 106 can utilize denser point clouds and higher resolution RGB images, the computation of which requires more processing power than is available on current mobile devices. Due to the processing strength and speed of the edge cloud, the time required to refine a 3D mesh and texturing a 3D object is also improved.

3. The edge cloud 106 enables the utilization of, and improves the performance of, extremely processing intensive techniques such as dynamic fusion to scan flexible objects (i.e., people) in real-time. Such processing is not currently possible on typical mobile devices.

Therefore, the availability of the edge cloud 106 allows for:

1. Processing of more object types such as smaller and large objects, flexible objects, and scene reconstruction.

2. Easier use of 3D object capture and tracking applications because tracking is more robust, and applications can handle faster movements of the camera and of the objects in a scene.

3. Higher quality 3D models, with greater details and photorealistic textures.

FIG. 2 is a flow diagram of a dynamic capture real-time 3D scanning workflow, using the system 100 of FIG. 1. Dynamic capture refers to the scanning process where the mesh construction is being generated in real-time to provide feedback to the user on how much of the object or scene has been scanned so far. Dynamic capture also includes showing the quality of the mesh during the scanning process for any corrective action by the user. The main parts for the dynamic capture are tracking 202 and 3D reconstruction 204 (mesh)+ photogrammetry 206 (texture). This process allows capture of a high-quality 3D model using a mobile device 102. Specific descriptions of the dynamic capture processes mentioned above are found in U.S. Pat. No. 9,715,761, titled “Real-Time 3D Computer Vision Processing Engine for Object Recognition, Reconstruction, and Analysis,” as well as U.S. patent application Ser. No. 16/118,894, titled “Enhancing Depth Sensor-Based 3D Geometry Reconstruction with Photogrammetry,” which are both incorporated herein by reference.

Generally, all the dynamic capture functions can run natively on the mobile device 102 without any cloud-based processing, which takes about one minute to scan and process. However, due to the processing power, heat, and battery consumption constraints of most mobile devices, many of these dynamic capture functions run at lower resolution and frame rates. As a result, the overall quality and reliability of the functions is limited.

One option is to use generic cloud-based processing (i.e., by connecting the mobile device 102 to a cloud computing environment 108). However, this is typically not feasible because of the more than 100 millisecond latency (in one direction) from the device 102 to the cloud computing environment 108 in typical wireless networks. Therefore, when the images are captured from the mobile device 102, it takes too long for the images to be processed and the resulting 3D model to be sent back to the mobile device 102. Because dynamic capture generally requires real-time interaction, the desired framework would have a minimum latency delay (i.e., less than a few milliseconds). Therefore, utilization of the edge cloud 106 overcomes this technical drawback.

FIG. 3 is a flow diagram of a computerized method 300 of utilizing edge cloud 106 resources for object scanning and 3D model generation. FIG. 4 is a flow diagram of a computerized method of utilizing edge cloud 106 resources for 3D model post-processing and refinement.

As shown in FIG. 3, once the mobile device 102 captures (302) the depth map and RGB images, the mobile device 102 sends the depth map and RGB images to the edge cloud 106 for object tracking and 3D model generation. After only a few milliseconds of processing, the edge cloud 106 provides real-time feedback to the mobile device 102 regarding the latest ‘global model’ (304) that has been scanned so far, using the steps as described in U.S. patent application Ser. No. 14/849,172, titled “Real-time Dynamic Three-Dimensional Adaptive Object Recognition and Model Reconstruction,” (incorporated herein by reference), which describes the process of extracting 2D+3D features, tracking to capture the pose information, and performing 3D reconstruction. Some of the scan functions such as 2D+3D feature detection, tracking, and mapping can be done on-device, instead of the edge cloud, if sufficient amount of processing power is available on the mobile device. Therefore, it can be appreciated that the systems and methods described herein can be adapted in order to flexibly allocate how much of the functionality or processing can be done on the mobile device versus the edge cloud to save bandwidth or server cost.

After the scanning process, the rest of the post-processing steps (shown in FIG. 4) U.S. patent application Ser. No. 14/849,172, titled “Real-time Dynamic Three-Dimensional Adaptive Object Recognition and Model Reconstruction,” as well as U.S. Pat. No. 10,192,347, titled “3D Photogrammetry” (incorporated herein by reference) clean up the mesh and add texture. The post-processing step(s) can be done entirely in the edge cloud 106 to generate the final 3D model.

In doing so, it is estimated that the overall scan time is reduced significantly, from one minute to a few seconds, battery consumption of the mobile device 102 is reduced by 80% or more, thermal heating problems on the mobile device 102 are eliminated, 1080p or higher RGB textures are added—making the entire process more reliable and easier to use.

FIG. 5 is a flow diagram of a computerized method 500 of utilizing edge cloud 106 resources for non-rigid 3D reconstruction, using the system 100 of FIG. 1. FIG. 5 shows how a ‘non-rigid’ scanning process can leverage the edge cloud 106 resources to scan and track flexible objects, such as people. Similar to 3D capture, nonrigid 3D reconstruction builds a 3D model based on the depth map and color (RGB) images. However, instead of calculating a rigid pose, which is six degrees of freedom (6DOF), nonrigid 3D reconstruction calculates a non-rigid pose, which can be several thousand degrees of freedom. To achieve reliable and fast reconstruction, nonrigid 3D reconstruction heavily relies on GPU parallel computing, which makes it impossible to run on current-generation mobile devices.

In view of the above, the methods and systems described herein provides certain technological advantages, including but not limited to:

1) Leveraging 5G edge cloud architecture for real-time 3D capture;

2) Real-time feedback to a mobile device user of the 3D scan progress and quality;

3) Real-time tracking of objects and people that are being scanned; and

4) Reduction in the amount of bandwidth used between the mobile device 102 and edge cloud 106 by ‘segmenting’ the depth and RGB images intelligently based on the object pose.

The above-described techniques can be implemented in digital and/or analog electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The implementation can be as a computer program product, i.e., a computer program tangibly embodied in a machine-readable storage device, for execution by, or to control the operation of, a data processing apparatus, e.g., a programmable processor, a computer, and/or multiple computers. A computer program can be written in any form of computer or programming language, including source code, compiled code, interpreted code and/or machine code, and the computer program can be deployed in any form, including as a stand-alone program or as a subroutine, element, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one or more sites.

Method steps can be performed by one or more specialized processors executing a computer program to perform functions by operating on input data and/or generating output data. Method steps can also be performed by, and an apparatus can be implemented as, special purpose logic circuitry, e.g., a FPGA (field programmable gate array), a FPAA (field-programmable analog array), a CPLD (complex programmable logic device), a PSoC (Programmable System-on-Chip), ASIP (application-specific instruction-set processor), or an ASIC (application-specific integrated circuit), or the like. Subroutines can refer to portions of the stored computer program and/or the processor, and/or the special circuitry that implement one or more functions.

Processors suitable for the execution of a computer program include, by way of example, special purpose microprocessors. Generally, a processor receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and/or data. Memory devices, such as a cache, can be used to temporarily store data. Memory devices can also be used for long-term data storage. Generally, a computer also includes, or is operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. A computer can also be operatively coupled to a communications network in order to receive instructions and/or data from the network and/or to transfer instructions and/or data to the network. Computer-readable storage mediums suitable for embodying computer program instructions and data include all forms of volatile and non-volatile memory, including by way of example semiconductor memory devices, e.g., DRAM, SRAM, EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and optical disks, e.g., CD, DVD, HD-DVD, and Blu-ray disks. The processor and the memory can be supplemented by and/or incorporated in special purpose logic circuitry.

To provide for interaction with a user, the above described techniques can be implemented on a computer in communication with a display device, e.g., a CRT (cathode ray tube), plasma, or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse, a trackball, a touchpad, or a motion sensor, by which the user can provide input to the computer (e.g., interact with a user interface element). Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, and/or tactile input.

The above described techniques can be implemented in a distributed computing system that includes a back-end component. The back-end component can, for example, be a data server, a middleware component, and/or an application server. The above described techniques can be implemented in a distributed computing system that includes a front-end component. The front-end component can, for example, be a client computer having a graphical user interface, a Web browser through which a user can interact with an example implementation, and/or other graphical user interfaces for a transmitting device. The above described techniques can be implemented in a distributed computing system that includes any combination of such back-end, middleware, or front-end components.

The components of the computing system can be interconnected by transmission medium, which can include any form or medium of digital or analog data communication (e.g., a communication network). Transmission medium can include one or more packet-based networks and/or one or more circuit-based networks in any configuration. Packet-based networks can include, for example, the Internet, a carrier internet protocol (IP) network (e.g., local area network (LAN), wide area network (WAN), campus area network (CAN), metropolitan area network (MAN), home area network (HAN)), a private IP network, an IP private branch exchange (IPBX), a wireless network (e.g., radio access network (RAN), Bluetooth, Wi-Fi, WiMAX, general packet radio service (GPRS) network, HiperLAN), and/or other packet-based networks. Circuit-based networks can include, for example, the public switched telephone network (PSTN), a legacy private branch exchange (PBX), a wireless network (e.g., RAN, code-division multiple access (CDMA) network, time division multiple access (TDMA) network, global system for mobile communications (GSM) network), and/or other circuit-based networks.

Information transfer over transmission medium can be based on one or more communication protocols. Communication protocols can include, for example, Ethernet protocol, Internet Protocol (IP), Voice over IP (VOIP), a Peer-to-Peer (P2P) protocol, Hypertext Transfer Protocol (HTTP), Session Initiation Protocol (SIP), H.323, Media Gateway Control Protocol (MGCP), Signaling System #7 (SS7), a Global System for Mobile Communications (GSM) protocol, a Push-to-Talk (PTT) protocol, a PTT over Cellular (POC) protocol, Universal Mobile Telecommunications System (UMTS), 3GPP Long Term Evolution (LTE) and/or other communication protocols.

Devices of the computing system can include, for example, a computer, a computer with a browser device, a telephone, an IP phone, a mobile device (e.g., cellular phone, personal digital assistant (PDA) device, smart phone, tablet, laptop computer, electronic mail device), and/or other communication devices. The browser device includes, for example, a computer (e.g., desktop computer and/or laptop computer) with a World Wide Web browser (e.g., Chrome™ from Google, Inc., Microsoft® Internet Explorer® available from Microsoft Corporation, and/or Mozilla® Firefox available from Mozilla Corporation). Mobile computing device include, for example, a Blackberry® from Research in Motion, an iPhone® from Apple Corporation, and/or an Android™-based device. IP phones include, for example, a Cisco® Unified IP Phone 7985G and/or a Cisco® Unified Wireless Phone 7920 available from Cisco Systems, Inc.

Comprise, include, and/or plural forms of each are open ended and include the listed parts and can include additional parts that are not listed. And/or is open ended and includes one or more of the listed parts and combinations of the listed parts.

One skilled in the art will realize the technology may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The foregoing embodiments are therefore to be considered in all respects illustrative rather than limiting of the technology described herein.

Claims

1. A system for three-dimensional (3D) object capture and object reconstruction using edge cloud computing resources, the system comprising: a sensor device, coupled to a mobile computing device, that captures (i) one or more depth maps of a physical object in a scene, the depth maps including related pose information of the physical object, and (ii) one or more color images of the physical object in the scene;an edge cloud computing device, coupled to the mobile computing device via a 5G network connection, that: receives the one or more depth maps and the one or more color images from the mobile computing device;generates a new 3D model of the physical object in the scene based on the received one or more depth maps and one or more color images, when a 3D model of the physical object has not yet been generated;updates an existing 3D model of the physical object in the scene based on the received one or more depth maps and one or more color images, when a 3D model of the physical object has previously been generated, including tracking the physical object in the scene based upon the pose information received from the mobile computing device; andtransmits the new 3D model or the updated 3D model to the mobile computing device.

2. The system of claim 1, wherein the mobile computing device crops the captured depth maps and color images prior to transmitting the captured depth maps and color images to the edge cloud computing device.

3. The system of claim 2, wherein cropping the captured depth maps and color images comprises removing a portion of the captured depth maps and color images that corresponds to a background of the scene.

4. The system of claim 1, wherein the edge cloud computing device performs the generating step, the updating step, and the transmitting step within less than 10 milliseconds after receiving the one or more depth maps and the one or more color images from the mobile computing device.

5. The system of claim 1, wherein transmitting the updated 3D model to the mobile computing device comprises providing, to the mobile computing device, tracking information associated with the physical object in the scene based upon the tracking step.

6. The system of claim 1, wherein the physical object is a non-rigid object.

7. The system of claim 1, wherein the edge cloud computing device performs one or more post-processing functions on the updated 3D model.

8. The system of claim 7, wherein the one or more post-processing functions comprise a bundle adjustment process, a de-noising process, a mesh refinement process, a texture alignment process, a shadow removal process, or a blending process.

9. The system of claim 7, wherein the edge cloud computing device generates a final 3D model from the updated 3D model after performing the one or more post-processing functions.

10. The system of claim 9, wherein the edge cloud computing device transmits the final 3D model to the mobile computing device and/or to a cloud-based server computing device.

11. The system of claim 1, wherein a latency of the 5G network connection between the mobile computing device and the edge cloud computing device is less than 5 milliseconds.

12. A computerized method of three-dimensional (3D) object capture and object reconstruction using edge cloud computing resources, the method comprising: capturing, by a sensor device coupled to a mobile computing device, (i) one or more depth maps of a physical object in a scene, the depth maps including related pose information of the physical object, and (ii) one or more color images of the physical object in the scene;receiving, by an edge cloud computing device coupled to the mobile computing device via a 5G network connection, the one or more depth maps and the one or more color images from the mobile computing device;generating, by the edge cloud computing device, a new 3D model of the physical object in the scene based on the received one or more depth maps and one or more color images, when a 3D model of the physical object has not yet been generated;updating, by the edge cloud computing device, an existing 3D model of the physical object in the scene based on the received one or more depth maps and one or more color images, when a 3D model of the physical object has previously been generated, including tracking the physical object in the scene based upon the pose information received from the mobile computing device; andtransmitting, by the edge cloud computing device, the new 3D model or the updated 3D model to the mobile computing device.

13. The method of claim 12, wherein the mobile computing device crops the captured depth maps and color images prior to transmitting the captured depth maps and color images to the edge cloud computing device.

14. The method of claim 13, wherein cropping the captured depth maps and color images comprises removing a portion of the captured depth maps and color images that corresponds to a background of the scene.

15. The method of claim 12, wherein the edge cloud computing device performs the generating step, the updating step, and the transmitting step within less than 10 milliseconds after receiving the one or more depth maps and the one or more color images from the mobile computing device.

16. The method of claim 12, wherein transmitting the updated 3D model to the mobile computing device comprises providing, to the mobile computing device, tracking information associated with the physical object in the scene based upon the tracking step.

17. The method of claim 12, wherein the physical object is a non-rigid object.

18. The method of claim 12, wherein the edge cloud computing device performs one or more post-processing functions on the updated 3D model.

19. The method of claim 18, wherein the one or more post-processing functions comprise a bundle adjustment process, a de-noising process, a mesh refinement process, a texture alignment process, a shadow removal process, or a blending process.

20. The method of claim 18, wherein the edge cloud computing device generates a final 3D model from the updated 3D model after performing the one or more post-processing functions.

21. The method of claim 20, wherein the edge cloud computing device transmits the final 3D model to the mobile computing device and/or to a cloud-based server computing device.

22. The method of claim 12, wherein a latency of the 5G network connection between the mobile computing device and the edge cloud computing device is less than 5 milliseconds.