Patent Application
20230325549
This application relates in general to 3D modeling and in particular to a system and method for automatic floorplan generation.
Floorplans of a desired space are useful for navigating in a building, planning furniture placement, planning a remodel, planning the placement of pipes, wires, ducts, sensors and thermostats, and modeling the building for heating, ventilation, and air conditioning (HVAC) applications. However, producing a floorplan can be a time-consuming and expensive process that requires expert skills, including the measuring of distances and angles, and entering data into a CAD program, for example. Furthermore, this process may need to be performed more than once because the floorplan of a building can change over a period of time, such as when walls are moved, added, or removed, for example.
Accordingly, a system and method for automatically generating a floorplan is needed. Preferably, the floorplans are easily generated by users having little or no training.
Automatic floor plan generation should be able to be used by anyone capable of moving around in a building and looking at its walls, floors, and ceilings and require little or no training. A resulting automatic floorplan can show walls, chairs, tables, file cabinets, and other furniture and is automatically aligned with horizontal and vertical axes. If desired, a user can annotate the positions of sensors, thermostats, vents, doors, windows, and other objects and these will appear in the floorplan as well. A first style of floorplan can resemble a pen-and-ink diagram that shows features at different distances from the floor, and a second style of floorplan resembles a drafting diagram. Clutter in the space represented by the floor plan can be removed by recognizing walls in 3D space before projection to 2D. The resulting floorplan can include the positions of sensors, thermostats, vents, doors, windows, and other objects.
An embodiment provides a system and method for automatic floorplan generation. Mesh triangles are gathered for a space having one or more walls and a floor. A request for a floorplan for the space is received. A facing direction of the floor is determined from the mesh triangles. The mesh triangles are rotated until the floor is horizontal. A primary wall facing direction is also determined from the mesh triangles. The mesh triangles are rotated so the primary wall facing direction is parallel to a major axis or other desired direction. The floorplan is generated based on the floor and walls.
Still other embodiments will become readily apparent to those skilled in the art from the following detailed description, wherein are described embodiments by way of illustrating the best mode contemplated. As will be realized, other and different embodiments are possible and the embodiments' several details are capable of modifications in various obvious respects, all without departing from their spirit and the scope. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not as restrictive.
Currently, obtaining a floorplan takes time and money due to requiring expert skill and the arduous task of measuring distances and angles within a space for which the floorplan is generated. Some floorplans can take days or longer to generate. To obtain floorplans at a lower cost and in a lesser amount of time, automatic floorplan generation can be utilized. Individuals, regardless of skill level and floorplan knowledge, can use the automatic floorplan generation to create drafting style floorplans and pen-and-ink floorplans.
Automatic floorplan generation can utilize an augmented reality headset to obtain data regarding a space for which the floorplan is to be generated.
The altitude module 17 utilizes an algorithm to determine an approximate altitude of a ceiling of the space from a histogram of triangle altitudes, and the drafter module 18 draws a floorplan of the space for output or display to the user, such as via a computing device 23 over the Internetwork 14. The computing device 23 can include a desktop computer or mobile computing device, such as a cellular phone, laptop, tablet, or other types of mobile computing devices.
In a further embodiment, the mesh triangle data and annotations can be processed to generate a floorplan via the augmented head set and provided to a different device for output. At a minimum, the headset should include a processor and wireless transceiver.
The mesh data utilized for generating the floorplan is obtained by moving around a space.
As the user walks around the space, the user can annotate the mesh triangles using eye gaze and voice commands to indicate the positions of objects, such as sensors, thermostats, vents, doors, windows, and other important objects. For example, the user can speak “that is a thermostat” when one is identified in the building space. A marker can then be added to the mesh data collected by the augmented reality headset at a location at which the user is gazing, for representation on a display. Further, sensors generate Bluetooth low energy signals that provide a MAC address of the respective sensor. When close enough, the HoloLens can find a sensor with a strong signal and obtain the MAC address for that sensor to get identification information for that sensor. The synthetic objects are displayed in the augmented reality view at these indicated positions. The annotations of the objects by the user can be collected (step 32), transmitted, and stored with or separate from the mesh triangles.
When sufficient mesh triangles and synthetic objects have been gathered, the user requests the automatic generation of a floorplan. The request is received (step 33), such as by a server, for processing. An algorithm can be used to discover a position (step 34) of a floor in the space using the mesh triangle orientation and dominant surface normal directions of the mesh triangles, as determined by k-means clustering in spherical coordinates. Subsequently, the floor of the mesh is rotated (step 34) to be precisely horizontal. The algorithm can also discover the dominant directions of the walls, if any, using another application of k-means clustering in spherical coordinates. The dominant walls are rotated (step 35) to align with Euclidian coordinates. The annotations are also rotated (step 36) based on the above rotation transformations, which are recorded and applied to positions of the synthetic objects, so they retain their relationship to the mesh.
If a drafting-style floorplan is desired, flat walls can be computed (step 37). Specifically, the walls can be recognized via an algorithm that applies a modified DBScan algorithm to the mesh triangles to find wall segments, discards wall segments that do not match a wall pattern, and replaces the mesh triangles, in each remaining wall segment, with rectangular blocks. If walls are not necessary to include in the floorplan, computing the walls can be skipped.
The mesh triangles can be sliced (step 38) to produce a set of line segments. In particular, the algorithm can determine an approximate altitude of a ceiling of the space from a histogram of triangle altitudes. A set of altitudes (y values) are chosen in the range from the floor to the ceiling. For example, this distance can be divided into n equal layers, for an integer n. At each altitude, a plane is constructed parallel to the floor at that altitude. The intersection of that plane is computed with the triangle mesh, producing a set of line segments.
Subsequently, the floorplan is drawn (step 39) by setting the y values of all line segments equal to 0, resulting in line segments in two dimensions. Specifically, in any given slice, all of the segments will have the same y value and that y value is the altitude of the slice. All the resulting line segments from all slices are drawn to form a single two-dimensional illustration, which is the floorplan. Finally, the synthetic objects are drawn (step 40). The y values of all synthetic objects are set equal to 0 and drawn onto the floorplan. Once generated, the floorplans can be provided to the user or a third party via a link, an attachment, or web access.
The resulting floorplan is automatically aligned with the horizontal and vertical axes and can show walls, chairs, tables, file cabinets, and other furniture. If desired, the user can annotate the positions of sensors, thermostats, vents, doors, windows, and other objects for display in the floorplan as well. A first style of floorplan can resemble a pen-and-ink diagram that shows features at different distances from the floor. A second style of floorplan can resemble a drafting diagram that removes clutter by recognizing walls in a 3D space before projection to 2D. Any user can generate the floorplans as long as an augmented reality headset and access to the floorplan generation algorithms are available.
Augmented reality headsets, such as the HoloLens 2, come with hardware and software that constructs 3D triangles that represent the surrounding environment. These triangles can be displayed to the user as the user walks around in a space and looks in different directions.
After gathering the mesh triangle data, the user can annotate objects in the space by indicating positions of important objects in the building interior, such as thermostats and sensors. The annotation can occur via eye gaze direction. With this method, the user looks at an object, such as a sensor, and gives some voice commands that tell the system: 1. The type of the object (e.g., a sensor) 2. The position of the object (the eye gaze point) 3. Any additional information (e.g., the type of sensor) and then tells the system to add a synthetic object at that position.
Once the mesh triangle data and annotations of the objects are obtained, the user can request that a floorplan be generated. The request can be generated by giving voice commands to the augmented reality headset, by typing commands in a terminal window, or by interacting with a graphical user interface. Other processes for generating the request are also possible.
After receiving the request, the automatic floorplan generation system takes the triangle mesh and synthetic objects that the user has created and makes them available to the processing steps. Processing can be done on the augmented reality headset itself or the data can be moved to another computer for processing.
To produce an accurate floorplan during processing, the automatic floorplan system can determine a facing direction and location of the floor. In one embodiment, the augmented reality headset already determines the rough direction of gravity from its built-in sensors. However, the direction may not be very accurate, so additional computation is needed. The floor direction can be determined by setting the rough gravity direction grough to be the negative y axis in which the mesh data is expressed. As the HoloLens creates the triangle mesh, higher y values are assigned to triangles that are located higher in the room and lower y values are assigned to triangles lower in the room. Accordingly, positive y direction is up, while the negative y direction is down. The final floor direction can be determined using grough and a true gravity or dominant direction, gtrue, as described below.
Often, one story of a building will be larger horizontally than vertically, the ceiling will be at a constant altitude, and the floor will be at a constant altitude. In such cases, the following steps can be performed: compute a bounding box of the entire mesh that fits the mesh tightly; find the minimum dimension of the bounding box, which will generally be the vertical dimension; and compute the surface normal of the top face of the bounding box that points towards the interior of the box. This is a good approximation of the true gravity direction, known as gtrue. Next, the mesh is rotated by the angle between gtrue and grough using a cross product of gtrue and grough as the axis of rotation. The floor will now be represented as horizontal.
In a further embodiment, spherical k-means can be used to determine gtrue. For each triangle in the mesh, consider the surface normal vector that represents its facing direction. Each triangle has two such vectors and the one that faces toward the observer (away from the interior of a wall, for example) is selected. Discard any triangles where the facing direction is far from the positive y direction, such as more than 30 degrees from that direction. However, other angles are possible for determining a far direction from the positive y direction.
The remaining triangles are the ones that are most likely to be part of the floor. A spherical coordinates k-means algorithm can be used to find the dominant direction of this collection of triangles. Having found this direction, the triangles that are relatively far from the dominant direction can be discarded and the k-means algorithm is run again. The process of finding the dominant direction, discarding triangles, and running the k-means algorithm can be performed until the dominant direction converges. The dominant direction can be represented by gtrue. The mesh is then rotated by the angle between gtrue and grough using the cross product of gtrue and grough as the axis of rotation. The floor will now be represented as horizontal.
Now that the mesh is rotated for representing the floor as horizontal, the altitude of the floor along the y direction can be determined. To do this, the centroid of each mesh triangle whose facing direction is within a small angle of the positive y axis is determined or obtained. For each such centroid, consider its y coordinate. A histogram of all of these y coordinates, where each bucket represents a small distance, such as 2 inches, can be generated. The buckets can be considered in pairs, such as (0, 1), (1, 2), (2, 3) and so on. Pairs of buckets with the highest number of points in each bucket are identified.
If working on a single story of the space, two large bucket pairs, one near the bottom of the centroids and one near the top of the centroids can be identified. The bucket pair at the bottom of the centroids can represent the floor and the bucket pair near the top of the centroids can represent the ceiling. In case the building has some rooms with sunken floors and some with raised ceilings, a gap of several feet (e.g, the expected floor to ceiling height of a room) between the low spikes and the high spikes in the histogram can be identified. This allows the y values of the floor levels and the ceiling levels to be found. For the automatic floorplan generation, it is generally safe to take the highest of the floor levels and the lowest of the ceiling levels. The buckets can be considered in pairs or individually; however, in pairs, spikes are less likely to be missed in the histogram if the triangles happen to split fairly evenly across two neighboring buckets.
Next, the mesh model can be rotated so that the primary wall directions are aligned with the axes of Euclidian coordinates. The primary wall directions are first located and then, the model of the space in the floorplan is rotated. The primary wall directions can be located by first, optionally, removing from consideration any mesh triangles with surface normal that face nearly in the positive y or negative y directions. These are most likely ceiling or floor triangles. While this step is not necessary, it does reduce the number of triangles that need to be processed in subsequent steps and may speed up the processing. Next, the surface normals of the remaining triangles in spherical coordinates are identified. The surface normal is a 3D vector that points in a direction perpendicular to the plane of the triangle. The negative of a surface normal is also a surface normal, but one that points in the opposite direction. The HoloLens is consistent in the way its triangle vertices are ordered (clockwise or counter-clockwise), so the surface normal vectors can be chosen consistently. For example, all floor triangle surface normals point up, while all ceiling triangle surface normals point down.
Each normal is then expressed in spherical coordinates. For example, the vectors in Euclidian coordinates [x, y, z] or vectors near Euclidian coordinates [x, y, z] are converted to spherical coordinates [theta, phi]. Spherical coordinates k-means clustering, such as described in Straub, Julian, Jason Chang, Oren Freifeld, and John Fisher III. “A Dirichlet process mixture model for spherical data.” In Artificial Intelligence and Statistics, pp. 930-938. PMLR, 2015, can be used to find the dominant wall directions. Assuming the building is designed with mainly perpendicular walls, there will be four dominant wall directions, so set k=4 for our k-means clustering. If working with a model that still has its floor and ceiling triangles, k can be set to k=6. The additional two primary directions are the floor and ceiling directions.
A guide 81 can be provided adjacent to the heat map 80. The guide can display color and corresponding values to indicate a number of mesh triangles in each bucket of directions. Each rectangle in the heat map 80 represents a cluster of triangles that have surface normals that point in nearly the same direction. For instance, if a globe of planet Earth is visualized, a rectangle represents a small range of longitudes and a small range of latitudes. If the globe is placed so its center touches the wall of a house, that wall will face towards one of the longitude, latitude points on the globe, which are the spherical coordinates point for that wall. In one example, a cluster might represent all of the walls that are on the West side of their respective rooms.
As one can see in
The k-means algorithm can be modified, such that after computing spherical coordinates k-means in the usual way, all triangles in each cluster whose facing direction is more than a threshold θ1 from the cluster center are identified and discarded. Then, the k-means clustering can be run again and updated cluster centers are computed. All triangles that are more than θ2 from each cluster center where θ2<θ1 can be discarded. This process can be repeated several times until a desired accuracy is achieved. For example, a sequence of angles θ2: [60, 50, 40, 30, 20, 10, 5, 3] can be used, where the angles are in degrees. Once the modified spherical coordinates k-means algorithm completes, there will be 4 (or 6) cluster centers. Larger numbers can be used if the building has more wall directions, as is the case for buildings with diagonal walls, octagonal rooms, and similar more complicated geometries.
Once the primary wall directions are chosen, the cluster, which represents a set of walls that face in the same direction, with the largest number of triangles is selected and the direction is identified as θwall. The primary walls can be the walls that face West, North, East, or South for example. In that case, non-primary walls would face other directions like Southeast. The mesh is rotated by the angle between θwall and the x axis. After this operation, the primary walls will be pointing along the x axis.
Specifically, the mesh model can be rotated for alignment along one or more coordinate axes.
If the user of the augmented reality application has annotated the building model, such as by adding marker objects to the model, such as the positions of sensors, doors, windows, vents, or other objects, these objects should appear in the correct positions in the final floorplan model. To determine these positions, the same transformation is applied to the markers for the objects as was applied to the mesh itself.
To rotate the markers for the objects, the floor leveling transformation can be represented as a matrix and the wall rotation can be represented as another matrix. The matrices are constructed so that simple matrix multiplication allows multiple transformations to be composed. If the floor leveling transformation matrix is A, and the wall rotation transformation matrix is B, the full transformation C=AB is computed and C is applied to the marker objects before displaying the markers to the user.
To compute flat walls and separate the walls from furniture and other building contents wall directions are determined and a DBScan can be performed. For the wall directions, divide the triangles into those with similar facing directions, such as north, south, east and west, based on the surface normals of the triangles. For each wall direction, a modified DBScan algorithm can be performed and a centroid point of each triangle is computed. For each centroid point during DBScan, a number of other centroid points that are near enough to be considered neighbors is determined. However, instead of looking for neighbors in a sphere around the point, neighbors in a rectangular block centered on the point should be identified. The blocks should be tall enough to reach from the floor to the ceiling in the y direction, a little less wide than a door in the direction parallel to the proposed wall (e.g., 1.5 feet), and a few inches in the wall direction (to allow for some deviations from being perfectly flat). After running the DBScan, the mesh triangles can be grouped into wall segments.
Those wall segments that are not good candidate walls can be discarded. For example, walls that are too small, that don't come near enough to the floor, or that don't come near enough to the ceiling should be discarded. Subsequently, for each wall segment, a plane that has the same facing direction as the wall direction and is a good fit to the triangle centroids in that wall segment is identified. Given that the points are tightly collected in this direction, simply having the plane go through any centroid works surprisingly well. However, the point can also be chosen more carefully, such as finding a point at a median position in the wall direction.
For each remaining wall segment, rectangles can be constructed that lie in the fitted plane, are as wide as the wall segment triangles extend in width, and as tall as the wall segment triangles extend in height. For purposes of floorplan construction, the original mesh triangles can be discarded and replaced with the new rectangles to serve as a de-cluttered mesh. If the subsequent steps are based on libraries that expect a triangle mesh, two adjacent right triangles are used in place of each rectangle.
For example, during computation of the flat walls, a triangle mesh of an office building can be displayed.
Now that there is a mesh that is nicely aligned to the coordinate axes, the mesh can be sliced at several altitudes in order to produce a pen-and-ink floorplan. As described above, a histogram of y values allows the height of the floor or floors and the height of the ceiling or ceilings to be identified. A floor height, such as for the highest floor, is determined as the floor height yfloor. A ceiling height, such as for the lowest ceiling is determined as height yceiling. Then, a series of y values is selected between yfloor and yceiling, such as an even spacing of y values, where each yi represents a single slice:
y
i
=y
floor+(yceiling−yfloor)*(i/n)
for each i such that 0≤i≤n. For slicing the mesh, the altitude of mesh is divided into n levels. Here, i represents each slice at a different altitude level. The plane which lies at an ith level is selected and a slice is computed to create a stack of slices. For each such yi, the intersection of the mesh with the plane y=yi is computed producing the stack of slices.
A two-dimensional floorplan can be produced by ignoring the y coordinates of the resulting line segments and plotting the resulting (x, z) coordinates as a two-dimensional image.
Shapes or colors can be used as markers to represent synthetic objects placed by the user to show the positions of sensors and other objects. As described above, the objects have been transformed by the matrix C, so they appear in the correct positions relative to the mesh. For example, in
While the invention has been particularly shown and described as referenced to the embodiments thereof, those skilled in the art will understand that the foregoing and other changes in form and detail may be made therein without departing from the spirit and scope of the invention.
| Number | Date | Country | |
|---|---|---|---|
| 63328428 | Apr 2022 | US |
