Patent Application
20240151530
The following U.S. patent applications (including this one) are being filed concurrently, and the entire disclosure of the other applications are incorporated by reference into this application for all purposes:
This disclosure relates in general to surveying systems and augmented reality. Surveying systems determine positions of points relative to each other and/or to the Earth. Surveying can be used in many applications by land surveyors, construction professionals, and civil engineers. Surveying often uses specialized and/or expensive equipment, such as laser levels, surveying rods, total stations, laser scanners, and GNSS (Global Navigation Satellite System) receivers.
In augmented reality, one or more virtual objects (e.g., computer-generated graphics) can be presented to a user in relation to real-world objects. Augmented reality can include a see-through display with a virtual object shown to a user on the see-through display. An example of an augmented-reality system is the Microsoft HoloLens. Another type of augmented reality is overlaying a virtual object on an image of the real world. For example, a smartphone camera is used to acquire an image of objects in the real world. The smart phone then overlays a graphic on the image of the real world while presenting the image on a screen of the smart phone. Artificial reality is sometimes used to refer to both augmented reality and virtual reality.
This disclosure relates to providing metadata about points, and, without limitation, to using a robotic device to mark points and an augmented-reality device to provide metadata related to the marked points.
In certain embodiments, a method for construction layout using robotic marking and augmented-reality metadata comprises receiving data from a building plan, wherein the building plan includes a plurality of elements; marking out points in a construction site related to the plurality of elements in the building plan, wherein marking out points is performed by a robotic device, and points are marked in the construction site within a pre-established accuracy using a robotic total station with the robotic device, marks are made using a permanent marker or spray paint, metadata of points are not marked using the robotic device, the robotic device operates in a global navigation satellite system (GNSS) restricted environment, accuracy of marking points in the construction site is equal to or better than 5 centimeters, and/or a number of points marked is equal to or greater than 500; transmitting data from the building plan to an augmented-reality device, wherein data from the building plan includes metadata about the points; aligning the internal map of the augmented-reality device to the construction site by comparing locations of points in the internal map to data from the building plan; and/or presenting metadata about the points to a user of the augmented-reality device as the user looks at the points in the construction site. In some embodiments, the augmented-reality device is a first augmented-reality device; the method further comprises transmitting data from the building plan to a second augmented-reality device; and/or the first augmented-reality device receives different metadata than the second augmented-reality device.
In certain embodiments, a system for construction layout using robotic marking and augmented-reality metadata comprises a robotic device and an augmented-reality device. The robotic device comprising instructions to mark out points in a construction site within a pre-established accuracy related to a plurality of elements in a building plan. The augmented-reality device comprising instructions to: receive data of the building plan, wherein data of the building plan includes metadata about the points; align an internal map of the augmented-reality device to the construction site; and/or present metadata about the points to a user of the augmented-reality device as the user looks at points in the construction site. In some embodiments, aligning the internal map of the augmented-reality device to the construction site comprises comparing locations of points in the internal map to data from the building plan; instructions of the robotic device do not have the robotic device mark metadata near points; a number of points marked is equal to or greater than 200; the pre-established accuracy is equal to or less than 5 centimeters; the robotic device is a projector; and/or the augmented-reality device is a first augmented-reality device, and the system further comprises a second augmented-reality device configured to receive data of the building plan.
In certain embodiments, a method for construction layout using robotic marking and augmented-reality metadata comprises receiving data from a building plan, wherein the building plan includes a plurality of elements; marking out points in a construction site related to the plurality of elements in the building plan, wherein marking out points is performed by a robotic device, and/or points are marked in the construction site within a pre-established accuracy; transmitting data from the building plan to an augmented-reality device, wherein data from the building plan includes metadata about the points; aligning an internal map of the augmented-reality device to the construction site; and/or presenting metadata about the points to a user of the augmented-reality device as the user looks at the points in the construction site. In some embodiments, aligning the internal map of the augmented-reality device to the construction site comprises comparing locations of points in the internal map to data from the building plan; points are marked without marking metadata near points; transmitting data from the building plan to the augmented-reality device is performed using one or more wireless connections a number of points marked is equal to or greater than 200; the pre-established accuracy is equal to or less than 5 centimeter; the robotic device is a quadruped; at least one of the points is marked on a ceiling; the augmented-reality device is a first augmented-reality device, and the method further comprises transmitting data from the building plan to a second augmented-reality device; and/or the first augmented-reality device receives different metadata than the second augmented-reality device.
Further areas of applicability of the present disclosure will become apparent from the detailed description provided hereinafter. It should be understood that the detailed description and specific examples, while indicating various embodiments, are intended for purposes of illustration only and are not intended to necessarily limit the scope of the disclosure.
The present disclosure is described in conjunction with the appended figures.
In the appended figures, similar components and/or features may have the same reference label. Further, various components of the same type may be distinguished by following the reference label by a dash and a second label that distinguishes among the similar components. If only the first reference label is used in the specification, the description is applicable to any one of the similar components having the same first reference label irrespective of the second reference label.
The ensuing description provides preferred exemplary embodiment(s) only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the ensuing description of the preferred exemplary embodiment(s) will provide those skilled in the art with an enabling description for implementing a preferred exemplary embodiment. It is understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.
This disclosure relates to using precision marking with augmented-reality overlays to provide metadata related to the markings. U.S. patent application Ser. No. 16/924,648, filed on Jul. 9, 2020, and U.S. patent application Ser. No. 17/308,431, filed on May 5, 2021, which are incorporated by reference for all purposes, provide examples of using augmented reality in a construction environment.
Layout is a common workflow in the construction industry that involves translating digital designs onto the real world via high-precision instruments. This enables workers to begin fabrication work to tolerances stipulated by the design. The Layout workflow often involves laying out hundreds if not thousands of points in any given day to prepare the work to be done by subsequent workers. Layout typically uses a crew of several people to keep ahead of the work to be done. Historically, layout has been done with chains and tape measures. In modern times, these workflows have largely been performed using high precision instruments, such as robotic total stations and GNSS receivers.
A recent trend in the construction industry is to further automate layout-type tasks. For example, a number of technology providers are creating robots that, positioned with external precision sensors (such as a total station), move throughout the job site and mark floors with translated data from the design. An advantage of this approach is that it largely removes the use of humans doing the work, as this is generally tedious and repetitive work that is prone to costly errors.
In both human layout and robot layout, data being transferred from the design includes both positional data (often represented as a point or ‘X’) and metadata associated with a position. For example, metadata can include the point number, the type of material being installed, description of inserts, tolerance, etc. Positional data typically has explicit precision requirements, thus the need for tape measures, total stations, etc. Metadata has much less stringent precision requirements, so long as it's clear what point the metadata refers to.
Existing layout methods can be limited by a number of factors. The environment where layout is being done is often dirty and/or has other obstructions that can limit the ability to transcribe and persist data on the site. It is typically easy to recognize the location of a signal point or ‘X’, but can be difficult to read metadata associated with it over time. In many cases, the time it takes to add the additional metadata (either by a human or robot) is enough to deter its transcription at all. When metadata is transcribed, it is often limited to a single color (e.g., a single marker or can of spray paint), which can make it difficult to delineate between different points, types of data, etc. In some cases, the design limits the ability to mark significant amounts of data on a work surface (e.g., a concrete slab that will remain unfinished in the final design or a material that is difficult to write on). Oftentimes, the metadata associated with points changes in the time period between layout and fabrication, thus leaving the data that was marked as outdated. Existing processes can be very time intensive because of all of the metadata that must be marked. Sometimes, because of all the issues with existing processes, workers just don't mark the metadata at all. Not marking metadata can lead to bigger problems when other people come to do work and find marks on the ground with no context.
By moving the metadata to a digital visualization, the need to mark a surface can be significantly reduced. In some embodiments, only those markings that require high precision are physically marked on the surface by a human or robot. These could be physical markings or projected markings. Other data that is associated with the marked point (e.g., data that has less precision requirement) can be maintained digitally and overlaid through augmented reality. This could be in the form of a phone or tablet (e.g., pass-through AR) or a head-mounted display (e.g., Trimble XR10). By reducing the need for physical marking of associated metadata, previous system limitations described above can be reduced or removed. In some embodiments, the only mark used is a point (e.g., a dot, circle, ‘X’, ‘+’, etc.). This can remove the concern about space limitations and/or dirty environments where metadata markings might not persist. In some embodiments, marks include points, lines, and/or non-letter symbols. In some embodiments, marked metadata associated with a point is no more than 7, 5, 3, 2, 1, or 0 characters, numbers, or symbols. Some worksites have very large amounts of data (e.g., more than 100, 200, 500, 1000, 10,000 points to layout). Some worksites use multiple layout teams and/or robots. By reducing or eliminating marking of metadata, the speed at which points can be marked is significantly increased, as much of the physical marking is no longer performed.
In robotic scenarios, marking only the point can also reduce downtime and/or cost of operating the system as it reduces the ink required and improves the number of points that can be marked on a single battery charge. When viewed digitally, metadata (and point overlays) can be visualized in various colors or icons. The associated metadata to a point can be updated digitally in real time as designs change, as long as the positional information remains constant. Even in a situation where the positional information has also changed, an update can be shared with the user (e.g., “Warning—this laid-out point is no longer valid, please refer to new point over there→”).
For positioning, computer vision of the AR device can be used to identify the points marked (e.g., on the ground, on a wall, or on a ceiling). The AR device can reference those points back to a digital model to automatically determine and/or persist six degrees of freedom (6DOF) of the AR device in the environment. By leveraging these points, the AR device is able to overlay data at a much higher precision than it would have otherwise (e.g., because of drift errors in the AR device). In this way, the AR device and the layout system work in symbiosis. The AR device is reducing the requirements on the layout system by the layout system not marking metadata, and the points laid out by the layout system are providing higher accuracy positioning to the AR device.
Positioning data can be shared with other devices, people, or workflows. For example, existing points could be leveraged to extrapolate other information at high precision without the use of external precision instruments. In another example, another trade could use the points that have been marked to precisely align their own devices without using their own high-precision positioning (e.g., for visualization, layout, etc.).
Layout users can use robots to do their work autonomously or semi-autonomously. The act of positioning and marking is taken off their hands. Layout users may still be involved to assist the robot if the robot encounters errors, loses line-of-sight, etc. Because the metadata is moved digitally, the robot can move faster and do more points in less time (as well other benefits listed above). The positioning of the robot and/or the marks it makes can be used to position a mixed-reality (MR) or augmented-reality (AR) device, for quality assurance/control, for a fabricator, etc. Coupling an MR device with a robot for layout can increase the output/capabilities of the system: the MR device can take over or augment much of the marking tasks for the robot, and the robot can take over or augment the positioning task of the MR device.
Referring first to
The instructions of the robot 104 do not have the robot 104 mark metadata near the points 108 (e.g., metadata of points is not physically drawn next to points 108 by the robot). In some embodiments, a number of points 108 marked can be equal to or greater than 50, 100, 200, 500, 1000, or 10,000 and/or equal to or less than 100,000, 15,000, 10,000, 1,000, and/or 500. In some embodiments, the pre-established accuracy is equal to or better than 5, 2, 1, or 0.5 centimeters. In some embodiments, a global navigation satellite system (GNSS) receiver is used by the robot 104 for precision positioning. In the embodiment shown, a total station 112 is used to provide precision measurement data to the robot 104, so the robot 104 can mark the points 108 within the pre-established accuracy within a GNSS-restricted environment.
In some embodiments, instructions for marking out points comes from a building plan, wherein the building plan includes a plurality of elements. Points corresponding to the plurality of elements are marked in the construction site by the robot 104. Marks of points 108 can be made in a variety of ways, including using a permanent marker and/or spray paint.
In some embodiments, metadata can include a picture or drawing of an item to be used. In some embodiments, the picture or drawing is superimposed on an object held by the user to verify the correct component. For example, the user could hold up an anchor bolt, and a drawing of the anchor bolt to be used is superimposed on the anchor bolt held by the user. As the system matches that the anchor bolt held by the user matches the metadata, the drawing of the anchor bolt presented on a display of the augmented-reality device can turn green and/or an audio cue played to indicate to the user that the user has the correct object. The drawing presented on the display can then be removed from the display of the augmented-reality device (e.g., based on a user input or timeout) after verification. In some embodiments, verification is performed automatically (e.g., based on palm identification, when a user holds an object in a palm of the user's hand). In some embodiments, a user can enter a verification menu using a menu selection in the AR device.
In step 608, points are marked out in an environment related to the plan. For example, points are marked out in a construction site related to the plurality of elements in the building plan. In some embodiments, marking out points is performed by a robotic device (e.g., the robotic device makes the marks), and/or points are marked in the construction site within a pre-established accuracy (e.g., as shown in
In step 612, data from the plan is transmitted to an augmented-reality device. For example, data from the building plan is transmitted to the augmented-reality device 204 in
In step 616, an internal map of the augmented-reality device is aligned with the environment. For example, an internal map of the augmented-reality device 204 is aligned to the construction site in
In step 620, metadata about the points is presented to a user of the augmented-reality device. For example, metadata 304 about a type of fastener at the second point 108-2 is presented on a display of the augmented-reality device 204 in
In some embodiments, aligning the internal map of the augmented-reality device to the construction site comprises comparing locations of points in the internal map to data from the building plan. Points can be marked without marking metadata near the points, or marking very little metadata. Transmitting data from the building plan to the augmented-reality device can be performed using one or more wireless connections. For example, data from the building plan can be transmitted from a laptop to a wireless router (e.g., a WiFi router), and from the wireless router to the augmented-reality device. In some embodiments, data from the building plan can be preloaded onto a device. In some embodiments, the robot is wheeled, quadrupedal, or humanoid. Points can be marked on the floor, wall, or ceiling of the construction environment. The process can further include transmitting data from the building plan to a second, third, fourth, or more augmented-reality devices (e.g., to support a plurality of users at a construction site); and/or data from the building plan can be different data sent to different augmented-reality devices (e.g., for different trades and/or different tasks, though each device could use position data of points for aligning with the construction site).
In some configurations, data of the building plan can be a 3D map and/or a list of positions of points. Aligning of the AR device to the environment can use more than points or might not use points for alignment. For example, the AR device could be aligned to the environment using a total station as described in U.S. patent application Ser. No. 16/924,648, filed on Jul. 9, 2020, which is incorporated by reference for all purposes.
Storage subsystem 704 can be implemented using a local storage and/or removable storage medium, e.g., using disk, flash memory (e.g., secure digital card, universal serial bus flash drive), or any other non-transitory storage medium, or a combination of media, and can include volatile and/or non-volatile storage media. Local storage can include random access memory (RAM), including dynamic RAM (DRAM), static RAM (SRAM), or battery backed up RAM. In some embodiments, storage subsystem 704 can store one or more applications and/or operating system programs to be executed by processing subsystem 702, including programs to implement some or all operations described above that would be performed using a computer. For example, storage subsystem 704 can store one or more code modules 710 for implementing one or more method steps described above.
A firmware and/or software implementation may be implemented with modules (e.g., procedures, functions, and so on). A machine-readable medium tangibly embodying instructions may be used in implementing methodologies described herein. Code modules 710 (e.g., instructions stored in memory) may be implemented within a processor or external to the processor. As used herein, the term “memory” refers to a type of long term, short term, volatile, nonvolatile, or other storage medium and is not to be limited to any particular type of memory or number of memories or type of media upon which memory is stored.
Moreover, the term “storage medium” or “storage device” may represent one or more memories for storing data, including read only memory (ROM), RAM, magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and/or other machine-readable mediums for storing information. The term “machine-readable medium” includes, but is not limited to, portable or fixed storage devices, optical storage devices, wireless channels, and/or various other storage mediums capable of storing instruction(s) and/or data.
Furthermore, embodiments may be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and/or any combination thereof. When implemented in software, firmware, middleware, scripting language, and/or microcode, program code or code segments to perform tasks may be stored in a machine-readable medium such as a storage medium. A code segment (e.g., code module 710) or machine-executable instruction may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a script, a class, or a combination of instructions, data structures, and/or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, and/or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted by suitable means including memory sharing, message passing, token passing, network transmission, etc.
Implementation of the techniques, blocks, steps and means described above may be done in various ways. For example, these techniques, blocks, steps and means may be implemented in hardware, software, or a combination thereof. For a hardware implementation, the processing units may be implemented within one or more ASICs, DSPs, DSPDs, PLDs, FPGAs, processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described above, and/or a combination thereof.
Each code module 710 may comprise sets of instructions (codes) embodied on a computer-readable medium that directs a processor of a computing device 700 to perform corresponding actions. The instructions may be configured to run in sequential order, in parallel (such as under different processing threads), or in a combination thereof. After loading a code module 710 on a general purpose computer system, the general purpose computer is transformed into a special purpose computer system.
Computer programs incorporating various features described herein (e.g., in one or more code modules 710) may be encoded and stored on various computer readable storage media. Computer readable media encoded with the program code may be packaged with a compatible electronic device, or the program code may be provided separately from electronic devices (e.g., via Internet download or as a separately packaged computer-readable storage medium). Storage subsystem 704 can also store information useful for establishing network connections using the communication interface 708.
User interface 706 can include input devices (e.g., touch pad, touch screen, scroll wheel, click wheel, dial, button, switch, keypad, microphone, etc.), as well as output devices (e.g., video screen, indicator lights, speakers, headphone jacks, virtual- or augmented-reality display, etc.), together with supporting electronics (e.g., digital-to-analog or analog-to-digital converters, signal processors, etc.). A user can operate input devices of user interface 706 to invoke the functionality of computing device 700 and can view and/or hear output from computing device 700 via output devices of user interface 706. For some embodiments, the user interface 706 might not be present (e.g., for a process using an ASIC).
Processing subsystem 702 can be implemented as one or more processors (e.g., integrated circuits, one or more single-core or multi-core microprocessors, microcontrollers, central processing unit, graphics processing unit, etc.). In operation, processing subsystem 702 can control the operation of computing device 700. In some embodiments, processing subsystem 702 can execute a variety of programs in response to program code and can maintain multiple concurrently executing programs or processes. At a given time, some or all of a program code to be executed can reside in processing subsystem 702 and/or in storage media, such as storage subsystem 704. Through programming, processing subsystem 702 can provide various functionality for computing device 700. Processing subsystem 702 can also execute other programs to control other functions of computing device 700, including programs that may be stored in storage subsystem 704.
Communication interface 708 can provide voice and/or data communication capability for computing device 700. In some embodiments, communication interface 708 can include radio frequency (RF) transceiver components for accessing wireless data networks (e.g., Wi-Fi network; 3G, 4G/LTE; etc.), mobile communication technologies, components for short-range wireless communication (e.g., using Bluetooth communication standards, NFC, etc.), other components, or combinations of technologies. In some embodiments, communication interface 708 can provide wired connectivity (e.g., universal serial bus, Ethernet, universal asynchronous receiver/transmitter, etc.) in addition to, or in lieu of, a wireless interface. Communication interface 708 can be implemented using a combination of hardware (e.g., driver circuits, antennas, modulators/demodulators, encoders/decoders, and other analog and/or digital signal processing circuits) and software components. In some embodiments, communication interface 708 can support multiple communication channels concurrently. In some embodiments, the communication interface 708 is not used.
It will be appreciated that computing device 700 is illustrative and that variations and modifications are possible. A computing device can have various functionality not specifically described (e.g., voice communication via cellular telephone networks) and can include components appropriate to such functionality.
Further, while the computing device 700 is described with reference to particular blocks, it is to be understood that these blocks are defined for convenience of description and are not intended to imply a particular physical arrangement of component parts. For example, the processing subsystem 702, the storage subsystem 704, the user interface 706, and/or the communication interface 708 can be in one device or distributed among multiple devices.
Further, the blocks need not correspond to physically distinct components. Blocks can be configured to perform various operations, e.g., by programming a processor or providing appropriate control circuitry, and various blocks might or might not be reconfigurable depending on how an initial configuration is obtained. Embodiments of the present invention can be realized in a variety of apparatus including electronic devices implemented using a combination of circuitry and software. Electronic devices described herein can be implemented using computing device 700.
Various features described herein, e.g., methods, apparatus, computer-readable media and the like, can be realized using a combination of dedicated components, programmable processors, and/or other programmable devices. Processes described herein can be implemented on the same processor or different processors. Where components are described as being configured to perform certain operations, such configuration can be accomplished, e.g., by designing electronic circuits to perform the operation, by programming programmable electronic circuits (such as microprocessors) to perform the operation, or a combination thereof. Further, while the embodiments described above may make reference to specific hardware and software components, those skilled in the art will appreciate that different combinations of hardware and/or software components may also be used and that particular operations described as being implemented in hardware might be implemented in software or vice versa.
Specific details are given in the above description to provide an understanding of the embodiments. However, it is understood that the embodiments may be practiced without these specific details. In some instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.
While the principles of the disclosure have been described above in connection with specific apparatus and methods, it is to be understood that this description is made only by way of example and not as limitation on the scope of the disclosure. Embodiments were chosen and described in order to explain the principles of the invention and practical applications to enable others skilled in the art to utilize the invention in various embodiments and with various modifications, as are suited to a particular use contemplated. It will be appreciated that the description is intended to cover modifications and equivalents.
Also, it is noted that the embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed, but could have additional steps not included in the figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.
A recitation of “a”, “an”, or “the” is intended to mean “one or more” unless specifically indicated to the contrary. Patents, patent applications, publications, and descriptions mentioned here are incorporated by reference in their entirety for all purposes. None is admitted to be prior art.
