SYSTEM FOR VIRTUAL DISPLAY AND METHOD OF USE

Abstract

A preferred system and method for projecting a business information model at a construction site includes a network, a system administrator connected to the network, a database connected to the system administrator, a set of registration markers positioned in the construction site, and a set of user devices connected to the network. The system includes a hard hat, a set of headsets mounted to the hard hat, a set of display units movably connected to the set of headsets, a set of cameras connected to the set of headsets, and a wearable computer connected to the set of headsets and to the network. The cameras capture an image of the set of registration markers. A position of the user device is determined from the image and an orientation is determined from motion sensors. A BIM is downloaded and projected to a removable visor based on the position and orientation.

Description

FIELD OF THE INVENTION

The present invention relates to systems and methods managing distribution and use of information at a construction site. In particular, the present invention relates to systems and methods for displaying and using a business information model and other construction information at a construction site. The present invention also relates to virtual display systems, such as heads up displays.

BACKGROUND OF THE INVENTION

A major goal of any contractor in the building industry is timely job completion. Hence, efficiency is a paramount concern.

Scheduling of construction projects often requires numerous subtasks. The subtasks are often interdependent. These subtasks must necessarily be completed in order to maximize efficiency. For example, electrical conduit and foundation pads must be in place before installation of electrical equipment can begin. If an error is made in any interdependent subtask, it must be corrected before other tasks can proceed. Hence, correcting errors made in interdependent subtasks is expensive and time consuming because it often delays project completion.

Furthermore, heavy equipment such as cranes and elevators are scheduled to be on site at specific times when they are needed. If errors in subtasks are made, then the equipment must be either stored or rescheduled quickly, leading to increased construction costs and delay in project completion.

In a similar way, delivery of certain engineering, mechanical and scheduling information is critical to timely project completion. For example, engineering change orders, site drawings, schematics, photographs, tool type and location, physical equipment specifications and diagrams and repair manuals and parts lists for heavy equipment all are required to be easily available at a construction site for maximum efficiency. Other critical construction information includes queuing times and scheduling times for skilled personnel, tools and equipment. Any delay in receiving such critical information can effect timely project completion.

In order to be useful, construction information is generally accessed in the field at a construction site by paper drawings or in some cases, on a laptop computer. However, neither paper drawings nor laptop computers display the information to scale. Viewing information in this manner is often difficult to do and can lead to dangerous and costly mistakes.

Modern construction projects have attempted to remedy many of the inefficiencies caused by lack of timely information delivery and errors in interdependent subtasks by employing a consolidated building information model (BIM). The BIM is a set of computer graphics files that, when viewed on a CAD system, provide the current displays of wire frame models of structures in the completed construction project. The CAD display is layered in a manner that allows all separate views and accurate representations of all structures, physical equipment, wiring and plumbing. While the BIM has helped coordination of tasks and schedules, it is still not completely satisfactory because it is not easily accessible in the field. Further, the BIM does not address schedules or query times.

The prior art has attempted solutions to solve some of these problems with limited success. For example, U.S. Publication No. 2014/0184643 to Friend discloses a system and method of dynamically coordinating machines and personnel about a physical worksite using augmented content on an operator display device. To receive the augmented content, which is generated by an off-board management system, the operator display device is associated with a transmitter/receiver attached to a machine, such as an excavator or bulldozer. A position of the machine or personnel is determined by a GPS system or a laser scanning system. The operator display device includes a visor or goggles with transparent lenses, a scaled-down controller that includes a processor or other electronics to communicate with a personnel transmitter/receiver carried by a person and a controller that processes information signals received from the off-board management system and project them on the lenses of the operator display device. The augmented content is projected in the person's field of view as an overlay superimposed on the surrounding environment to show restricted area for personnel, routes of travel for machinery, and areas designated for excavation. However, the operator display device of Friend cannot determine its position without a construction site. Further, it does not display or interact with a BIM model.

U.S. Publication No. 2014/0210856 to Finn et al. discloses a system and method that integrates augmented reality technology with land surveying. A 3D digital model of internal elements of a building is generated using a 3D laser scanner upon installation of the internal elements, such as electrical and plumbing before wall panels are installed. The 3D digital model is associated with a set of markers that are placed on a finished wall in the building. The markers are used to project the generated 3D model on a mobile device, such as a smartphone, in view of a user. However, the system in Finn requires the 3D model to be generated once the internal systems are already installed, sometimes incorrectly, just prior to installing wall paneling. Therefore, the system in Finn cannot be used to prevent incorrect installation of building elements leading to costly construction overruns.

U.S. Publication No. 2014/0268064 to Kahle et al. discloses a system and method for projecting an image on a surface in a building under construction. The system includes a projector mounted on a moveable support for supporting a worker at a work position in the building. The projector projects the image on a surface above the moveable support in response to an image signal defining the image to be projected. The projected image indicates the location of connectors, anchors, and holes to be affixed to, or cut through, the surface and features behind the surface. A positioning system for determining the two dimensional position of the projector includes a laser measuring system that projects a rotating beam of laser light that sweeps across the moveable support to determine the distance and heading of the moveable support. However, the system in Kahle is prone to error because the laser measuring system is easily misaligned in the construction environment, thereby providing an incorrect position to the projector. Further, the system must be attached to the moveable support and cannot be transported easily between construction sites.

Therefore, there is a need in the art for a portable augmented reality system that provides access to virtual information accurately, in real time, at a construction site to prevent mistakes, thereby increasing the usability of the information and improving safety, time use and cost efficiency.

SUMMARY

A system and method for projecting information including, as an example, segments of a business information model at a construction site includes a network, a system administrator connected to the network, a database connected to the system administrator, a set of registration markers positioned in the construction site, and a set of user devices connected to the network. Each user device includes a hard hat, a set of headsets mounted to the hard hat, a set of display units movably connected to the set of headsets, a set of registration cameras connected to the set of headsets and directed towards the set of registration markers, and a wearable computer connected to the set of headsets and to the network.

The wearable computer is programmed with a set of instructions to carry out the method which includes the steps of receiving the business information model, receiving a position image of the set of registration markers, receiving a set of motion data, determining a position of the user device and an orientation of the user device based on the position image and the set of motion data, rendering the business information model based on the position, the orientation, and the position image as a rendered business information model, and displaying the rendered business information model as a stereoscopic image to the user.

The described embodiments herein disclose significantly more than an abstract idea including technical advancements in the fields of construction management and data processing, and a transformation of data which is directly related to real world objects and situations. The disclosed embodiments enable a computer and integrated optics and dedicated electrical components to operate more efficiently and improve the optical display of the BIM and other information and construction management technology in general.

BRIEF DESCRIPTION OF THE DRAWINGS

In the detailed description presented below, reference is made to the accompanying drawings.

FIG. 1 is a schematic of a BIM distribution system of a preferred embodiment.

FIG. 2 is a schematic diagram of a user device of a preferred embodiment.

FIG. 3A is a side view of a user device of a preferred embodiment.

FIG. 3B is a side view of a camera matrix of a preferred embodiment.

FIG. 4A is a front view of a user device of a preferred embodiment.

FIG. 4B is an isometric view of a user device of a preferred embodiment.

FIG. 4C is an isometric view of a display unit of a preferred embodiment.

FIG. 4D is an exploded isometric view of a connection for a display unit of a preferred embodiment.

FIG. 5 is a top view of a camera matrix of a preferred embodiment.

FIG. 6 is a top view schematic of a projector unit and ray diagram of a preferred embodiment.

FIG. 7A is a top view schematic of a projector unit and a virtual image of a preferred embodiment.

FIG. 7B is a point of view of an augmented image of a preferred embodiment.

FIG. 8 is a plan view of a user device and a registration system of a preferred embodiment.

FIG. 9A is a schematic of a registration marker of a preferred embodiment.

FIG. 9B is a schematic of a registration marker of a preferred embodiment.

FIG. 10 is a schematic of a data flow for an augmented reality application of a preferred embodiment.

FIG. 11 is a command input menu for a user device of a preferred embodiment.

FIG. 12 is a flow chart of a state machine method of a preferred embodiment.

FIG. 13 is a flow chart of a method for registering a marker of a preferred embodiment.

FIG. 14A is a flow chart of a method for calibrating a position of a user device of a preferred embodiment.

FIG. 14B is a top view schematic of a camera position with respect to a registration marker of a preferred embodiment.

FIG. 14C is a side view schematic of a camera position with respect to a marker of a preferred embodiment.

FIG. 14D is a captured skewed image of a registration marker of a preferred embodiment.

FIG. 14E is a flow chart of a method for deskewing an image of a preferred embodiment.

FIG. 14F is a deskewed image of a preferred embodiment.

FIG. 14G is a side view schematic of a camera and a registration marker of a preferred embodiment.

FIG. 15 is a flow chart of a runtime process of a preferred embodiment.

FIG. 16 is a flow chart for method of determining a position of a user device of a preferred embodiment.

FIG. 17 is a flow chart for a method of rendering a stereoscopic overlay for a user device of a preferred embodiment.

FIG. 18 is a flow chart of a method for updating a business information model of a preferred embodiment.

FIG. 19 is a flow chart of a method for updating a business information model of a preferred embodiment.

DETAILED DESCRIPTION

It will be appreciated by those skilled in the art that aspects of the present disclosure may be illustrated and described in any of a number of patentable classes or contexts including any new and useful process or machine or any new and useful improvement.

Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, C#, .NET, Objective C, Ruby, Python SQL, or other modern and commercially available programming languages.

Referring to FIG. 1, system 100 includes network 101, system administrator 102 connected to network 101, and a set of user devices 104 each of which is connected to network 101. System administrator 102 is further connected to BIM database 103 for storage of relevant data. For example, receiving data may include a business information model, engineering change orders, textual data, equipment manuals and operation instructions, images, photos, text messages, videos, emails, graphics, documents, 2-dimensional and 3-dimensional drawings, and sketches.

In a preferred embodiment, each of user devices 104 communicates with system administrator 102 to access BIM database 103 to project a BIM as will be further described below.

It will be appreciated by those skilled in the art that any type of three-dimensional rendering may be employed in the disclosed embodiment and that a BIM is just one example of such a three-dimensional rendering.

Referring to FIG. 2, user device 200 includes wearable computer 201 connected to headset 202. Wearable computer 201 is further connected to replaceable battery 203, microphone 204, control input device 206, and speaker 205.

Wearable computer 201 includes processor 207 and memory 209 connected to processor 207, and network interface 208 connected to processor 207. Augmented reality application 210, BIM 211, and a set of videos, images, and data 212 are stored in memory 209. In one embodiment, control input device 206 is connected to wearable computer 201. In preferred embodiment control input device 206, is a remote control having a navigation pad and a selection button. Any type of control input device known in the art may be employed.

Headset 202 is further connected to display unit 213 and a set of cameras 214. Headset 202 includes processor 215, a set of sensors 216 connected to processor 215, and memory 217 connected to processor 215.

Referring to FIG. 3A, a preferred implementation of user device 200 is described. Hard hat 302 is worn by user 301. Hard hat 302 has pocket 305 integrally formed in it. In a preferred embodiment, the hard hat includes a “pocket and clip” arrangement used to secure the headset as shown and described in U.S. Pat. No. 8,191,292 to Cummings, et al., which is incorporated herein by reference. Various types of head covers or helmets may be employed to support the headset. Headset 303 detachably mounts to hard hat 302 by flexible mounts 304 attached to case 350 and pocket 305. Headset 303 is further connected to wearable computer 313 via connection 314. Wearable computer 313 is preferably a portable computing device, such as a laptop or tablet computer, worn as a backpack by user 301. Connection 314 provides a data and power connection from wearable computer 313 to headset 303. Headset 303 includes processor 310, memory 311 connected to processor 310, and sensors 312 connected to processor 310. Mounting arm 306 is slidably inserted into headset 303 to allow forward and backward movement. In a preferred embodiment, the mounting arm is biased by a mechanical coil spring which enables it to retract into case 350. Display arm 307 is pivotably connected to mounting arm 306 for pivotal movement about axis 324. Display unit 308 is attached to display arm 307. Display unit 308 includes projector 325, camera 326, and display light guide 309. Camera 326 has field of view 328. In a preferred embodiment, field of view 328 is 90°. In other embodiments, other suitable field of view ranges may be employed. Display arm 307 is further connected to headset 303 with data and power connection 327.

User 301 wears communication device 315. Communication device 315 includes earpiece speaker 316 and microphone 317. Communication device 315 is preferably connected to wearable computer 313 via a wireless connection such as a Bluetooth connection. In other embodiments, other wireless or wired connections are employed. Communication device 315 enables voice activation and voice control of an augmented reality application stored in the wearable computer 313 by user 301.

In one embodiment, camera matrix 318 is detachably connected to headset 303. Camera matrix 318 includes halo 319 and halo 321, each of which is detachably connected to headset 303. A set of base cameras 320 is connected to halo 319 and in communication with headset 303. A set of angled cameras 322 is connected to halo 321 and in communication with headset 303.

Referring to FIG. 3B in another embodiment, camera matrix 318 is attached inside surface 329 of hard hat 302. In this embodiment, halos 319 and 321 are attached to inside surface 329 of hard hat 302 with a suitable adhesive or fastener. Hole 334 is integrally formed in hard hat 302 adjacent to headset 303 for connection to camera matrix 318. In a preferred embodiment, connector 335 is a USB 3.0 connector, connected to a processor of headset 303 and positioned in hole 334 to connect to camera matrix 318. Other suitable data connections may be employed. Set of base cameras 320 is connected to halo 319, each of which is positioned in a hole of set of holes 330. Set of holes 330 is integrally formed in hard hat 302. In one embodiment, side cameras 333 of set of base cameras 320 are attached to headset 303 outside of hard hat 302. In another embodiment, side cameras 333 are eliminated. Set of angled cameras 322 is connected to halo 321, each of which is positioned in a hole of set of holes 331. Set of holes 331 is integrally formed in hard hat 302.

In another preferred embodiment, the cameras are each mounted securely to the inside surface of the hard hat and are positioned to view the outside world through the holes.

In a preferred embodiment, a BIM is downloaded from a system administrator server into a memory resident in a wearable computer 313. The BIM is transmitted from wearable computer 313 through headset 303 and projector 325 for viewing adjacent eye 323 of user 301 to augment the vision of user 301, as will be further described below. The user can select different layers of the BIM to view via voice control. For example, the BIM includes an electrical layer, which shows the location of electrical conduit, connection points, and equipment. As the user moves, headset 303 and wearable computer 313 tracks the location of user 301 and the position and orientation of the user's head using camera 326 and/or camera matrix 318.

In one embodiment, a set of data is downloaded, selected, and displayed to user 301. In one embodiment, the position and orientation of the user's head is not tracked in a display mode. Rather, the data is displayed without regard to the position of the user or hard hat. Any type of data content may be selected, formatted, scaled and displayed, including images, photos, text messages, videos, emails, graphics, documents, drawings, and sketches.

In a preferred embodiment, processor 310 is a 2.8 GHz octa-core Snapdragon 810 processor available from QUALCOMM® Technologies, Inc. Other suitable processors known in the art may be employed.

In a preferred embodiment, sensors 312 is a 9-axis motion tracking system-in-package package sensor, model no. MP11-9150 available from InverSense®, Inc. In this embodiment, the 9-axis sensor combines a 3-axis gyroscope, a 3-axis accelerometer, an on-board digital motion processor, and a 3-axis digital compass. In other embodiments, other suitable sensors and/or suitable combinations of sensors may be employed.

In a preferred embodiment, memory 311 is a 2 GB LPDDR3 RAM. Other suitable memory known in the art may be employed.

In a preferred embodiment, each of base cameras 320 and angled cameras 322 is a 16 megapixel smartphone camera capable of recording video at 30 fps that includes a CMOS image sensor, part no. 5K3M2 available from Samsung Semiconductor. Other suitable cameras and/or image sensors known in the art may be employed.

Referring to FIG. 4A, user device 400 includes display unit 402 which includes camera 404 on the temporal side of eye 406 of user 401 and display light guide 405. Camera 404 has field of view 413. Display light guide 405 is positioned in the field of view and adjacent to eye 406. Display unit 402 is movably connected to headset 403, which is detachably mounted to hard hat 407. Display unit 409 is movably connected to headset 408, which is detachably mounted to hard hat 407. Display unit 409 includes camera 410 on the temporal side of eye 412 of user 401. Camera 410 has field of view 414. Display light guide 411 is in the field of view and adjacent to eye 412. Display units 402 and 409 and headsets 403 and 408 are the same as previously described. Display units 402 and 409 provide a stereoscopic augmented view to user 401.

In a preferred embodiment, each of cameras 404 and 410 is a 16 megapixel smartphone camera capable of recording video at 30 fps that includes a CMOS image sensor, part no. 5K3M2 available from Samsung Semiconductor. Other suitable cameras and/or image sensors known in the art may be employed.

Referring to FIGS. 4B, 4C, and 4D in one embodiment, hard hat 407 includes pocket 450 integrally formed in hard hat 407 and adjacent to pocket 449. Glasses 451 includes display arm 452 adjustably engaged with connector 453 and display arm 469 adjustably engaged with connector 470. Mounting arm 454 is adjustably engaged with connector 453. Mounting arm 471 is adjustably engaged with connector 470. Data connection 468 connects display unit 409 to headset 408 detachably mounted to pocket 449, as previously described. Display arm 452 includes a set of ridges 462 integrally formed on it. Mounting arm 454 includes flexible mounts 455 to detachably mount glasses 451 to hard hat 407. Mounting arm 454 further includes a set of ridges 463 integrally formed on it. Connector 453 has mount portion 456 and display portion 457. Mount portion 456 includes channel 458 integrally formed in it. Channel 458 has ridge 459 integrally formed on it. Mounting arm 454 slidingly engages with channel 458. Set of ridges 463 engages with ridge 459 to enable adjustable positional movement along directions 464 and 465. Display portion 457 includes channel 460 integrally formed in it. Channel 460 includes ridge 461 integrally formed on it. Display arm 452 slidingly engages with channel 460. Set of ridges 462 engages with ridge 461 to enable adjustable positional movement along directions 466 and 467.

Likewise, display arm 469 includes a set of ridges 472 integrally formed on it. Mounting arm 471 includes flexible mounts 476 to detachably mount glasses 451 to a pocket in hard hat 407. Mounting arm 471 further includes a set of ridges 477 integrally formed on it. Connector 470 has mount portion 478 and display portion 479. Mount portion 478 includes channel 475 integrally formed in it. Channel 475 has ridge 480 integrally formed on it. Mounting arm 471 slidingly engages with channel 475. Set of ridges 477 engages with ridge 480 to enable adjustable positional movement along directions 464 and 465. Display portion 479 includes channel 473 integrally formed in it. Channel 473 includes ridge 474 integrally formed on it. Display arm 469 slidingly engages with channel 473. Set of ridges 472 engages with ridge 474 to enable adjustable positional movement along directions 466 and 467. Glasses 451 includes display light guides 405 and 411 and display units 402 and 409, as previously described. Display unit 402 is connected to headset 403 with a data connection.

In a preferred embodiment, channel 458 is generally perpendicular to channel 460 and vice versa. Other arrangements may be employed.

In a preferred embodiment, channel 475 is generally perpendicular to channel 473 and vice versa. Other arrangements may be employed.

In a preferred embodiment, each of display arms 452 and 469, connectors 453 and 470, and mounting arms 454 and 471 is made of an injection molded plastic. Other suitable materials known in the art may be employed.

In one embodiment, mount portion 456 and display portion 457 are separate pieces attached to each other with a suitable adhesive or epoxy. In another embodiment, mount portion 456 and display portion 457 are integrally formed portions of a single piece adjacent to each other. Other attachment means known in the art may be employed.

In one embodiment, mount portion 478 and display portion 479 are separate pieces attached to each other with a suitable adhesive or epoxy. In another embodiment, mount portion 478 and display portion 479 are integrally formed portions of a single piece adjacent to each other. Other attachment means known in the art may be employed.

Referring to FIG. 5, camera matrix 318 will be further described. Camera matrix 318 includes halo 501 and halo 502 connected to halo 501. Each of cameras 503, 504, 505, and 506 is connected to halo 501. Camera 503 has field of view 507. Camera 504 has field of view 508. Camera 505 has field of view 509. Camera 506 has field of view 510. Each of cameras 511, 512, 513, and 514 is connected to halo 502. Camera 511 has field of view 515. Camera 512 has field of view 516. Camera 513 has field of view 517. Camera 514 has field of view 518.

In a preferred embodiment, each of cameras 503, 504, 505, and 506 is positioned approximately 90° with respect to each other around halo 501. Other angular intervals may be employed.

In a preferred embodiment, each of cameras 511, 512, 513, and 514 is positioned approximately 90° with respect to each other around halo 502. Other angular intervals may be employed.

In a preferred embodiment, each of field of views 507, 508, 509, and 510 is approximately 90°. Other field of view ranges may be employed.

In a preferred embodiment, each of field of views 515, 516, 517, and 518 is approximately 90°. Other field of view ranges may be employed.

In a preferred embodiment, camera matrix 318 provides a 360° view of the surroundings of a user. In other embodiments, other numbers of cameras, angular positions, and field of view ranges may be employed to provide a 360° view.

Referring to FIG. 6, each of display units 402 and 409 will be further described as display unit 600. Display unit 600 will be further described with respect to a right eye of a user. It will be appreciated by those skilled in the art that the arrangement of display unit 600 is simply reversed for implementation on a left eye. Display unit 600 includes light guide 601, projector 602 attached to light guide 601, and camera 603 connected to and adjacent to light guide 601 and projector 602. Camera 603 is connected to headset 615 and includes lens 604. Projector 602 includes light source 605. Light source 605 is connected to headset 615. Collimating lens 606 is positioned adjacent to light source 605. Light guide 601 includes input surface 607 and output surface 608, each of which is attached to the interior of light guide 601. Each of input surface 607 and output surface 608 is positioned at angles ω and γ, respectively from front surface 613 to provide total internal reflection (“TIR”) for light guide 601, thereby projecting an image in field of view 610 of user eye 609.

In a preferred embodiment, angles ω and γ, are 30° and 45°, respectively. Any angles may be employed to provide TIR for light guide 601.

In use, light source 605 displays an image received from headset 615. The image is represented by rays 611 and 612. Rays 611 and 612 are transmitted through collimating lens 606 and reflected off of input surface 607 for TIR. Rays 611 and 612 are further reflected off of front surface 613 and rear surface 614 and output surface 608 in field of view 610 of user eye 609.

In a preferred embodiment, light source 605 is an organic light emitting diode (“OLED”) display such as the WUXGA OLED-XL Microdisplay, part no. EMA-100801-01, available from eMagin Corporation. In another embodiment, light source 605 is a light emitting diode (“LED”) display. Other suitable light sources and displays known in the art may be employed.

In a preferred embodiment, light guide 601 is made of acrylic. In another embodiment, light guide 601 is made of poly (methyl methacrylate) (“PMMA”). Other suitable materials known in the art may be employed.

In a preferred embodiment, input surface 607 is a flat mirror and output surface 608 is a partially-reflective mirror, such as a half-silvered mirror. In other embodiments, other combinations for input surface 607 and output surface 608 may employed and are summarized in Table 1 below.

TABLE 1
Combinations for Input and Output Surfaces
	Output Surface
			Diffraction
	Single	Multiple	grating	Diffraction
	Reflective	Reflective	(varying	Grating	Holographic
Input Surface	Surface	Surfaces	index)	(lines)	Element
Single Reflective	x	x	x	x	x
Surface
Multiple Reflective	x	x	x	x	x
Surfaces
Combination	x	x	x	x	x
Refractive/Reflective
Element
Diffraction Grating	x	x	x	x	x
(varying index)
Diffraction Grating	x	x	x	x	x
(lines)
Holographic	x	x	x	x	x
Element

Referring to FIG. 7A in use, lens 705 of camera 704 is automatically focused on real object 706 at a distance d from camera 704 and sent to headset 710 as image 707. Headset 710 and wearable computer 711 determine distance d and the position of display unit 701 with respect to real object 706. Wearable computer 711 generates virtual image 708 based on distance d. Projector 703 projects virtual image 708 into light guide 701, as previously described. Virtual image 708 is displayed as virtual object 709 to appear at distance d in view of user eye 702. Virtual object 709 is magnified to coincide with the size and position of real object 706 to create a perceived depth of focus d. In one embodiment, d is less than d. In another embodiment, d is equal to d. In one embodiment, d is a fixed distance from camera 704 for all real objects.

Referring to FIG. 7B, point of view 712 is the view a user sees while wearing a headset and display unit. Point of view 712 includes floor 713 and adjoining wall 714. Registration marker 715 is attached to floor 713. Registration marker 716 is attached to wall 714. Real object 718 is being lowered into position. According to the BIM, the correct location for real object 718 is outlined by virtual object 717. In this way, a user easily determines if real object 718 is properly positioned and can quickly make adjustments to ensure real object 718 is properly positioned.

In one embodiment, a set of data 719 is displayed. The set of data 719 includes image 720 and text 721. Any type of data including images, photos, text messages, videos, emails, graphics, documents, drawings, schematics, diagrams, and hand-drawn sketches may be employed. For example, image 720 is an installation diagram of real object 718 and text 721 is a set of installation instructions for real object 718.

Each of the positions and sizes of image 720 and text 721 is optionally changed by the user.

In one embodiment, set of data 719 is displayed simultaneously with virtual object 717. In another embodiment, set of data 719 is displayed without virtual object 717 in a display mode, as will be further described below.

Referring to FIG. 8, construction site 800 includes floor 801 and adjoining walls 802 and 803. Registration system 804 includes registration markers 805, 806, and 807 positioned at precise locations on floor 801, wall 802 and wall 803, respectively and serve as a set of reference points for user device 808 worn by user 809.

Each of the positions of registration markers 805, 806, and 807 is associated with a position in a BIM. Survey location 810 is precisely positioned at a known location at construction site 800 and saved in the BIM. Reference marker 811 is a master reference point based on the location of the survey location 810. Each of registration markers 805, 806, and 807 is positioned from reference marker 811 to ensure proper location of floor 801 and walls 802 and 803. At least one of registration markers 805, 806, 807, and 811 will be in view of a camera of user device 808 worn by user 809 and at any given time. The camera captures an image of at least one of registration markers 805, 806, 807, and 811. A wearable computer of user device 808 decodes the captured image to determine a real location of at least one of registration markers 805, 806, 807, and 811. The wearable computer determines a corresponding virtual location in the BIM.

For example, user 809 is standing in construction site 800 wearing user device 808 and looking down at location 812 where object 813 is to be installed. Registration marker 805 is in view of user device 808. The projected BIM shows the correct installation position 814 in view of user 809 as if the user were standing inside the BIM. As user 809 tilts his or her head up to look at wall 802 the movement of the user's head is detected by user device 808 and registration marker 806 is in view of user device 808. Based on the position of registration marker 806, the BIM is moved and rotated in real time to align with the user's field of vision and provide an in-person view of the BIM to user 809. Crane 815 lowers object 813 towards location 812. Based on the projected BIM, object 813 should be installed at installation position 814. User 809 uses the projected BIM to properly lower the object 813 and precisely install object 813 at proper installation position 814, thereby saving time and money in the form of overrun construction costs.

If a mistake is found, user 809 captures still images using the camera for upload to the system administrator or records or streams video back to the system administrator. In this way, the party responsible for the mistake can be easily and quickly identified.

Referring to FIGS. 9A and 9B, registration marker 901 includes shape 903 and code 904. Registration marker 902 includes shape 905 and code 906. Any polygon may be employed for shapes 903 and 905.

In a preferred embodiment, each of codes 904 and 906 is a two-dimensional bar code. In this embodiment, each of codes 904 and 906 includes a set of marker information, including a set of dimensions of shapes 903 and 905, and a set of x, y, z coordinates position at which registration markers 901 and 902 are placed, and a description of each shape and location. Any type of code may be employed.

Shapes 903 and 905 enable detection of codes 904 and 906, respectively, at an offset angle. For example, shape 903 is an equilateral triangle and shape 905 is a rectangle. If a camera capturing an image of shapes 903 and 905 is positioned at an offset angle, shapes 903 and 905 will appear as a scalene triangle and a parallelogram, respectively, in a skewed image.

Referring to FIG. 10, data flow 1000 for augmented reality application 1001 for a user device will be described. BIM 1002 is input into augmented reality application 1001. Application commands 1003 provide input control for the processes of augmented reality application 1001. Images 1004 are received and sent by augmented reality application 1001. For example, a set of cameras captures a set of registration images. The set of marker images is used to determine the position of the user. In another example, images 1004 are still or video images captured by a set of cameras adjacent to the eyes of the user and saved to memory for later upload or streamed to a server. Point of view image 1005 is captured by the set of headset cameras adjacent to the eyes of a user. Set of data 1007 is input into augmented reality application 1001.

In a preferred embodiment, the position of the user is determined from the set of code images 1004 by augmented reality application 1001. Augmented reality application 1001 orients BIM 1002 according to the determined position of the user. Commands 1003 determine which layers of BIM 1002 are displayed. Augmented reality application 1001 overlays the selected layers of BIM 1002 at the determined position to generate stereoscopic image overlay 1006 for display.

In one embodiment, commands 1003 determine a subset of set of data 1007 to display and the size and position of the subset of the set of data. Augmented reality application 1001 overlays the selected subset of data 1007 according to the selected size and position of the set of data 1007 for display.

Referring to FIG. 11, commands menu 1100 includes standby/run toggle 1101, BIM layer selection 1102, reload BIM 1103, save overlaid image 1104, and calibrate 1105. Standby/run toggle 1101 toggles the augmented reality application to a standby mode or a run mode. BIM layer selection 1102 enables the user to select any layer of the BIM to view. For example, the layers include, but are not limited to, structural, electrical, plumbing, data, and HVAC. Reload BIM button 1103 downloads the BIM into memory. Save overlaid image 1104 captures a “screen capture” of the point of view and the overlaid BIM from the perspective of the user. Calibrate 1105 executes a calibration process, as will be further described below. Position and orientation toggle 1106 toggles the position and orientation functions on and off to selectively run in a display mode. Select data 1107 enables the user to select which data to display and the size and the position of the selected data. Selection of 1101, 1102, 1103, 1104, 1105, 1106, and 1107 is accomplished via voice controls.

Referring to FIG. 12, state machine method 1200 for an augmented reality application will now be described. State machine method 1200 begins at step 1201 in a power off mode. Once the system is enabled in step 1202 by initiating power, state machine method 1200 proceeds to a standby mode at step 1203. Once a “run” command is received, state machine method 1200 proceeds to step 1204. At step 1204, a position and an orientation function of the augmented reality application is toggled on or off If toggled off, then the augmented reality application runs in a display mode at step 1205 and optionally displays a set of data selectable by the user. The augmented reality application runs in the display mode until the user toggles the position and the orientation function on at step 1204. If turned on, then state machine method 1200 proceeds to step 1206.

At step 1206, state machine method 1200 turns on a set of cameras and begins to search for a registration marker in a loss of “marker lock” mode. At step 1207, a position and orientation of a user device is determined from the registration marker, as will be further described below. If the position and orientation of the user device cannot be determined, then state machine method 1200 returns to step 1206 to search for a registration marker. If the position and orientation of the user device is determined, then state machine method 1200 proceeds to step 1208. At step 1208, the augmented reality application runs in a “marker lock” mode, that is the position and orientation of the user device can repeatedly be determined within a predetermined time. In this step, a runtime loop for the augmented reality application is initiated and a BIM is displayed, as will be further described below. In a preferred embodiment, the predetermined time is 30 seconds. Other times may be employed.

In one embodiment, the set of data is displayed when the augmented reality application runs in the “marker lock” mode.

At step 1209, a consistency is determined. In this step, if the position and orientation of the user device can be repeatedly determined within the predetermined time, then state machine method 1200 returns to step 1208. In this step, if the BIM is properly displayed, i.e., is rotated and aligned with the user point of view, then stated machine method 1200 returns to step 1208. If the position and orientation of the user device cannot be repeatedly determined within the predetermined time or the BIM is not properly displayed, i.e., is not rotated and aligned with the user point of view, then state machine method 1200 proceeds to step 1210. At step 1210, a message is displayed to the user indicating a position and orientation consistency problem and state machine method 1200 begins a calibration process at step 1211, as will be further described below.

Referring to FIG. 13, method 1300 for registering a registration marker for a BIM will be described. The registration marker includes a shape and a code, as previously described. At step 1301, a position of the registration marker is calibrated. In this step, a surveyor or a user positions the registration marker in a desired location. For example, the registration marker is placed in the middle of a wall or a column or a stud. Any desired location may be employed. Measurements are taken to ensure the registration marker is placed in the desired location. At step 1302, a set of location coordinates of the placed registration marker is stored in the code and in the BIM. At step 1303, a set of dimensions for the shape of the registration marker is stored in the code and in the BIM. At step 1304, a description of the registration marker is stored in the code and the BIM. Method 1300 is repeated for each registration marker.

Referring to FIG. 14A, method 1400 for calibrating a position of a user device will be described. At step 1401, a camera of the user device is pointed at a registration marker so that the registration marker is within a field of view of the camera.

Referring to FIGS. 14B and 14C, an offset position of user device 1409 with respect to registration marker 1411 shown in a top view and a side view, respectively, will now be described. User device 1409 has camera 1410. Camera 1410 has camera axis 1414. Registration marker 1411 is in view 1413 of camera 1410. Registration marker 1411 has marker axis 1412. Camera 1410 and user device 1409 is positioned offset with respect to registration marker 1411. Position angle α is the angle between marker axis 1412 and camera axis 1414 in the x-z plane of coordinates 1415. Position angle β is the angle between marker axis 1412 and camera axis 1414 in the y-z plane of coordinates 1415. In one embodiment, registration marker 1411 is rotated about the z-axis of coordinates 1415.

Because of the offset position of user device 1409 and camera 1410 as defined by position angles α and β, the image of registration marker 1411 is skewed.

Returning to FIG. 14A, at step 1402, an image of the registration marker is captured by the camera.

Referring to FIG. 14D, skewed image 1416 includes skewed registration marker 1442. Skewed registration marker 1442 includes skewed shape 1417 and skewed code 1418. Skewed registration mark 1442 is in the x-y plane defined by x-axis 1419 and y-axis 1420. Z-axis 1421 traverses perpendicularly through skewed registration mark 1442. As can be seen in FIG. 14D, skewed registration mark 1442 appears as a parallelogram. In this example, skewed registration mark 1442 is rotated approximately 30° about each of x-axis 1419, y-axis 1420, and z-axis 1421.

Returning to FIG. 14A, at step 1403, a set of edges in the image of the registration marker is located. In this step, Gaussian smoothing is first applied to the image to reduce noise in the image. In a preferred embodiment, Canny edge detection is then employed to locate the set of edges. In other embodiments, other edge detection means may be employed. In one embodiment, edge thinning is applied to the set of edges to remove any unwanted points. In a preferred embodiment, the set of edges is a boundary of the shape of the registration marker.

At step 1404, the image is deskewed in order to determine a set of position angles with respect to the registration marker, as will be further described below.

At step 1405 the code is read to determine the set of dimensions of the shape of the registration marker, including an actual height and an actual width. At step 1406, a distance from the camera to the registration marker is determined.

At step 1407, an absolute position of the user is calculated based on the position angles and the distance from the registration marker.

Referring to FIG. 14E, step 1404 will be further described as method 1422 for deskewing an image. Method 1422 begins at step 1423. At step 1424, a set of reference lines for the set of edges of a registration marker is determined. In a preferred embodiment, the set of references lines is determined by the Hough transform. Other suitable methods known in the art may be employed.

At step 1425, a pair angle is calculated between each pair of intersecting reference lines to generate a set of pair angles. At step 1426, a skew angle is calculated from set of pair angles by averaging the set of pair angles. At step 1427, the image is rotated about an axis by the skew angle. The skew angle is the position angle with respect to each axis, as previously described. At step 1428, whether or not the image has been deskewed for all axes is determined. If not, method 1422 advances to the next axis at step 1429 and returns to step 1424. If so, method 1422 ends at step 1430.

Referring to FIG. 14F, deskewed image 1443 includes deskewed registration marker 1444 along x-axis 1419, y-axis 1420, and z-axis 1421. Deskewed registration marker 1444 includes deskewed shape 1431 and deskewed code 1432. Deskewed registration marker 1444 is approximately perpendicular to z-axis 1421. Deskewed code 1432 can now be read. Deskewed image 1443 has height 1433 and width 1434. Deskewed registration marker 1444 has height 1435 and width 1436.

In a preferred embodiment, each of heights 1433 and 1435 and widths 1434 and 1436 is measured by counting the number of pixels for deskewed registration marker 1444 and deskewed image 1443.

Referring to FIG. 14G, step 1406 will now be further described. Camera 1437 has field of view 1438 spanning an angle 0, which varies depending on the type of camera employed. Registration marker 1439 is in plane 1440. Plane 1440 is distance 1441 from camera 1437. Height 1442 of registration marker 1439 is retrieved from a code contained in registration marker 1439. Distance 1441 is calculated by:

$\begin{array}{cc}d=\frac{h}{x\tan \theta },& \mathrm{Eq}.1\end{array}$

where d is distance 1441, h is height 1442, θ is angle θ of field of view 1438, and xis a height percentage of the height of the deskewed registration marker in the deskewed image to the height of the deskewed image. For example, if the height of the deskewed registration marker is 60% of the height of the deskewed image, then x=0.6.

Referring to FIG. 15, runtime process 1500 for an augmented reality application will now be described. Runtime process 1500 starts at step 1501. At step 1502, a BIM is retrieved. In this step, the BIM is downloaded from a system administration server and saved into memory of a user device. At step 1503, an image is captured from a set of cameras. At step 1504, a position and an orientation of the user device is determined, as will be further described below. At step 1505, a stereoscopic overlay of the BIM is rendered according to the position and the orientation of the user device, as will be further described below. At step 1506, the rendered stereoscopic overlay is output to a display unit of the user device for display to the user. In a preferred embodiment, the rendered stereoscopic overlay is rendered at least 24 fps.

In one embodiment, a set of data is retrieved at step 1507. In this step, the set of data is downloaded from the system administrator and saved into the memory of the user device. In one embodiment, the position and the orientation function is deactivated. In another embodiment, the position and the orientation function remain activated. At step 1508, a subset of the set of data is selected for display including the size and the position of the selected set of data. At step 1509, the selected subset of data is displayed on the display unit.

At step 1510, a determination is made as to whether an end command has been received. If not, runtime process returns to step 1503. If so, runtime process 1500 ends at step 1511.

Referring to FIG. 16, step 1504 will be further described as method 1600 for determining a position and an orientation of a user device. Method 1600 begins at step 1601. At step 1602, a set of registration markers is identified and decoded to determine the position of the user device. In a preferred embodiment, method 1400 is employed. At step 1603, a set of motion detection data is received from a set of sensors in the user device to determine movement of the user device. At step 1604, the set of motion detection data and the position of the user device are combined to determine an x, y, z position of the user device in reality and in the BIM and a roll, pitch, and yaw or detection of the user device in reality and the BIM. In this step, the user device determines which camera captured the image of the registration marker, i.e., a temporal camera or a camera of the camera matrix. If the camera of the camera matrix captures the images, then a difference angle is calculated between an axis of the camera of the camera matrix and an axis of the temporal camera. The orientation is calculated from the set of position angles and the difference angles. The set of motion detection data received is the roll, pitch, and yaw orientation movement of the head of the user. Method 1600 ends at step 1605.

Referring to FIG. 17, step 1505 will be further described as method 1700 for rendering a stereoscopic overlay according to the position and the orientation of the user device for a user device. Method 1700 begins at step 1701. At step 1702, a BIM is rotated and magnified based on the position and the orientation of the user device. At step 1703, the BIM is “clipped” based on a set of barriers in the BIM, i.e., the nearest set of walls. For example, if the user is standing the middle of a room, the BIM is “clipped” to only show the room of the BIM in which the user is standing. Otherwise, the entire BIM of the entire building would be shown to the user. At step 1704, a layer selection of the BIM is determined from the command menu. At step 1705, the selected layers of the “clipped” BIM is rendered as a stereoscopic image, i.e., the BIM image is rendered as a pair of BIM images, a left BIM image for a left display unit and a right BIM image for a right display unit of the user device. Method 1700 ends at step 1706.

In a preferred embodiment, the left BIM image and the right BIM image are shifted with respect to each other, in a range of approximately 2.5 to 3 inches to compensate for the average distance between the pupils of human eyes.

Referring to FIG. 18, method 1800 for updating a BIM will now be described. Method 1800 begins at step 1801. At step 1802, a virtual location of a virtual object in the BIM is determined by viewing the virtual location on a display unit of a user device. At step 1803, an actual location of a real object associated with the virtual object is determined. At step 1804, a tolerance for the real object location is determined by any measuring means. In a preferred embodiment, the tolerance is determined by a set of building codes. At step 1805, the actual location is compared to the virtual location to determine whether the actual location is within the tolerance. If so, then method 1800 ends at step 1809. If the actual location is not within the tolerance, then method 1800 proceeds to step 1806. At step 1806, an image is captured of the actual location and the virtual location as seen through the display by the user. At step 1807, the captured image is uploaded to a system administrator server. At step 1808, the captured image is saved in the BIM as a “mistakes” layer. The “mistakes” layer is then a selectable layer in the BIM once a user reloads the BIM to the user device from the system administrator server. Method 1800 ends at step 1809.

Referring to FIG. 19 in another embodiment, method 1900 for updating a BIM will now be described. Method 1900 begins at step 1901. At step 1902, a streaming session between a user device and a system administrator server is initiated and a video is captured and streamed in real time to the system administrator server. The video includes the point of view of the user captured by a camera with the overlaid BIM. At step 1903, a virtual location of a virtual object in the BIM is determined by viewing the virtual location on the display. At step 1904, an actual location of a real object associated with the virtual object is determined. At step 1905, a tolerance for the real object location is determined by any measuring means. In a preferred embodiment, the tolerance is determined by a set of building codes. At step 1906, the actual location is compared to the virtual location to determine whether the actual location is within the tolerance. If so, then method 1900 ends at step 1908. If the actual location is not within the tolerance, then method 1900 proceeds to step 1907. At step 1907, the video is saved in the BIM in a “mistakes” layer as a selectable element, such as an icon or link. The “mistakes” layer is then a selectable layer in the BIM once a user reloads the BIM to a wearable computer from the system administrator server. The user selects the selectable element to stream and view the video. Method 1900 ends at step 1908.

It will be appreciated by those skilled in the art that the described embodiments disclose significantly more than an abstract idea including technical advancements in the field of data processing and a transformation of data which is directly related to real world objects and situations in that the disclosed embodiments enable a computer to operate more efficiently and make improvements to construction management technology. Specifically, the disclosed embodiments eliminate the remanufacture of construction components and rescheduling of equipment. Further, the disclosed embodiments eliminate the reliance and use of external positioning systems, such as GPS or laser-based systems.

It will be appreciated by those skilled in the art that changes could be made to the embodiments described above without departing from the broad inventive concept. It is understood, therefore, that this disclosure is not limited to the particular embodiments herein, but it is intended to cover modifications within the spirit and scope of the present disclosure as defined by the appended claims.

Claims

1. A system for projecting a building information model at a construction site comprising: a set of registration markers positioned at the construction site; and,a user device comprising: a set of headset computation modules;a set of display units operatively connected to the set of headset computation modules;a set of registration cameras operatively connected to the set of headset computation modules and directed towards the set of registration markers;a computer connected to the set of headset computation modules and programmed to carry out the steps of: receiving the building information model;receiving a position image of at least one registration marker of the set of registration markers from the set of registration cameras;determining a tolerance for a location of a real object by measuring the real object location; and,comparing the real object location to a virtual location to determine whether the real object location is within the tolerance.

2. The system of claim 1, wherein the computer is further programmed to carry out the steps of: when the real object location and the virtual location are not within the tolerance: capturing a captured image of the real object location and the virtual location as seen through a display unit;uploading the captured image to a system administrator server;saving the captured image in the building information model in a mistakes layer; and,making the mistakes layer a selectable layer in the building information model once a user reloads the building information model to the user device from the system administrator server.

3. The system of claim 2, wherein the computer is further programmed to carry out the steps of: determining the virtual location of a virtual object in the building information model by viewing the virtual location on a display unit of a user device;associating the real object location of the real object with the virtual object; and,determining the tolerance by using a set of building codes.

4. The system of claim 3, wherein the computer is further programmed to carry out the steps of: determining a position and an orientation of the user device based on the position image;rendering the building information model, based on the position and the orientation, as a rendered building information model; and,displaying the building information model as a stereoscopic image.

5. The system of claim 4, wherein the computer is further programmed to carry out the steps of: rotating and magnifying the building information model based on the position and the orientation of the user device;clipping the building information model based on a set of barriers in the building information model, wherein the set of barriers includes a set of walls;when the user device is located in a room of a building associated with the building information model, clipping the building information model to only show the room of the building information model in which the user device is located; and,when the user device is not located in the room, displaying the building information model of the building in its entirety.

6. The system of claim 5, wherein the computer is further programmed to carry out the steps of: determining a layer selection of the building information model from a command menu;rendering a set of selected layers of the clipped building information model as a stereoscopic image that includes a left building information model image for a left display unit and a right building information model image for a right display unit of the user device; and,shifting the left building information model image and the right building information model image with respect to each other in a range of about 2.5 to about 3 inches.

7. The system of claim 6, wherein the computer is further programmed to carry out the step of: initiating a streaming session between the user device and the system administrator server.

8. The system of claim 7, wherein the computer is further programmed to carry out the step of: capturing and streaming a video in real time to the system administrator server.

9. The system of claim 8, wherein the computer is further programmed to carry out the step of: including in the video a point of view of the user device captured by a registration camera with the building information model.

10. The system of claim 9, wherein the computer is further programmed to carry out the step of: upon reloading the building information model to the user device from the system administrator server, making the mistakes layer a selectable layer in the building information model.

11. A method for displaying a building information model with a user device comprising a set of headset computation modules, a set of display units operatively connected to the set of headset computation modules, a set of registration cameras operatively connected to the set of headset computation modules and directed towards the set of registration markers, and a computer connected to the set of headset computation modules and programmed to carry out the steps of the method, the method comprising the steps of: receiving the building information model;receiving a position image of at least one registration marker of the set of registration markers from the set of registration cameras;determining a tolerance for a real object location; and,comparing the real object location to a virtual location to determine whether the real object location is within the tolerance.

12. The method of claim 11, wherein the method further comprises the steps of: when the real object location and the virtual location are not within the tolerance: capturing a first image of the real object location and the virtual location as seen through a display unit;uploading the first image to a system administrator server;saving the first image in the building information model in a mistakes layer; and,making the mistakes layer a selectable layer in the building information model once a user reloads the building information model to the user device from the system administrator server.

13. The system of claim 12, wherein the method further comprises the steps of: determining the virtual location of a virtual object in the building information model by viewing the virtual location on a display unit of a user device;associating the real object location with the virtual object; and,determining the tolerance by using a set of building codes.

14. The system of claim 13, wherein the method further comprises the steps of: determining a position and an orientation of the user device based on the position image;rendering the building information model, based on the position and the orientation, as a rendered building information model; and,displaying the building information model as a stereoscopic image.

15. The system of claim 14, wherein the method further comprises the steps of: rotating and magnifying the building information model based on the position and the orientation of the user device;clipping the building information model based on a set of barriers in the building information model, wherein the set of barriers includes a set of walls;when the user device is located in a room of a building associated with the building information model, clipping the building information model to only show the room of the building information model in which the user device is located; and,when the user device is not located in the room, displaying the building information model of the building in its entirety.

16. The system of claim 15, wherein the method further comprises the steps of: determining a layer selection of the building information model from a command menu;rendering a set of selected layers of the clipped building information model as a stereoscopic image that includes a left building information model image and a right building information model image; and,compensating for a distance between pupils of human eyes by shifting the left building information model image and the right building information model image with respect to each other.

17. The system of claim 16, wherein the method further comprises the step of: initiating a streaming session between the user device and the system administrator server.

18. The system of claim 17, wherein the method further comprises the step of: capturing and streaming a video in real time to the system administrator server.

19. The system of claim 18, wherein the method further comprises the step of: including in the video a point of view of the user device captured by a registration camera with the building information model.

20. The system of claim 19, wherein the method further comprises the step of: upon reloading the building information model to the user device from the system administrator server, making the mistakes layer a selectable layer in the building information model.