Patent Application
20230161463
The following disclosure relates generally to using automated tools and associated techniques to analyze and use images acquired in a defined area as part of generating mapping information for the area, such as to generate a floor plan for a building using images acquired at the building, as well as subsequently using the generated mapping information in one or more manners.
In various fields and circumstances, such as architectural analysis, property inspection, real estate acquisition and development, remodeling and improvement services, general contracting and other circumstances, it may be desirable to view information about the interior of a house, office, or other building without having to physically travel to and enter the building, including to determine actual as-built information about the building rather than design information from before the building is constructed. However, it can be difficult to effectively capture, represent and use such building interior information, including to display visual information captured within building interiors to users at remote locations (e.g., to enable a user to fully understand the layout and other details of the interior, including to control the display in a user-selected manner). In addition, while a floor plan of a building may provide some information about layout and other details of a building interior, such use of floor plans has some drawbacks in certain situations, including that floor plans can be difficult to construct and maintain, to accurately scale and populate with information about room interiors, to visualize and otherwise use, etc.
The present disclosure describes techniques for using one or more computing devices to perform automated operations related to, as part of generating mapping information of a defined area for subsequent use in one or more further automated manners, performing analyses and/or other uses of images acquired in the defined area. In at least some embodiments, the defined area includes an interior of a multi-room building (e.g., a house, office, etc.), the images include panorama images acquired at the building (e.g., 360° panorama images acquired at various acquisition locations within rooms of the building), and the generated information includes a floor plan of the building, such as a 2D (two-dimensional) overhead view (e.g., an orthographic top view) of a schematic floor map that is generated using information from the images—in at least some such embodiments, the generating of the mapping information is further performed without having or using depth information acquired from depth-sensing equipment about distances from the images' acquisition locations to walls or other objects in the surrounding building interior. The generated floor plan and/or other generated mapping-related information may be subsequently used in one or more manners in various embodiments, such as for controlling navigation of mobile devices (e.g., autonomous vehicles), for display on one or more client devices in corresponding GUIs (graphical user interfaces), etc. Additional details are included below regarding the automated operations of the computing device(s) involved in the generation and use of the mapping information, and some or all of the techniques described herein may, in at least some embodiments, be performed at least in part via automated operations of a Mapping Information Generation Manager (“MIGM”) system involved in the generating of the mapping information, as discussed further below.
In at least some embodiments and situations, some or all of the images acquired for a building are 360° panorama images that are each acquired at one of multiple acquisition locations in or around the building, such as with each panorama image covering 360 degrees horizontally around a vertical axis (e.g., by using an image acquisition device with a spherical camera having one or more fisheye lenses to capture a panorama image that extends 360 degrees horizontally, such as in a single moment, or by otherwise generating 360° panorama images, such as by horizontally rotating a camera at an acquisition location that captures video or a sequence of constituent images during the rotating). In addition, in at least some such embodiments, such panorama images may be provided and used in a spherical format having an equirectangular projection in which straight vertical data (e.g., the sides of a typical rectangular door frame) in the room remains straight in the image and in which straight horizontal data (e.g., the top of a typical rectangular door frame) in the room remains straight in the image if it is shown at a horizontal midline of the image but is increasingly curved in the image in a convex manner relative to the horizontal midline as the distance increases in the image from the horizontal midline. It will be appreciated that a 360° spherical panorama image may in some situations be represented in a spherical coordinate system and cover up to 360° around a vertical axis, such that a user viewing such a panorama image may move the viewing direction within the panorama image to different orientations to cause different subset images (or “views”) to be rendered within the panorama image (including, if the panorama image is represented in a spherical coordinate system, to convert the image being rendered into a planar coordinate system, such as for a perspective image view before it is displayed). Furthermore, acquisition metadata regarding the capture of such panorama images may be obtained and used in various manners, such as data acquired from IMU (inertial measurement unit) sensors or other sensors of a mobile image acquisition device as it is carried by a user or otherwise moved, and/or other data from other associated sensors (e.g., depth data from one or more depth sensors at an image acquisition location to measure distances to walls of the room or other objects in the room surrounding the acquisition location). In addition, images acquired for a building may further include one or more non-spherical images acquired in one or more rooms in at least some embodiments, such as perspective images in a rectilinear format in which horizontal and vertical straight lines in the room remain straight in the perspective images. Additional details are included below regarding automated operations of device(s) implementing an Image Capture and Analysis (ICA) system involved in acquiring images and optionally acquisition metadata, as well as in optionally performing preprocessing of the images before later use (e.g., to render 360° spherical panorama images in an equirectangular format).
The automated operations of the computing device(s) to provide the described techniques may in some embodiments and situations include operations of the MIGM system to interact with one or more MIGM system operator users who assist with the generating of the mapping information using acquired images (e.g., acquired 360° panorama images), such as by displaying one or more GUIs that show information related to the images and/or that show associated mapping information being generated, and by receiving and using input submitted by the user(s) via the GUI(s) as part of the mapping information generation. As one non-exclusive example, one or more MIGM system operator users may, in at least some embodiments, manipulate displayed information in the GUI about two or more rooms in order to identify and/or confirm interconnections between the rooms via passages into and/or out of the rooms, such as doors and other openings in walls of the rooms (e.g., inter-room wall openings such as doors, stairs and other non-door wall openings between rooms; other wall openings that are not between two rooms, such as exterior windows and exterior doors; etc.)—in addition, in at least such embodiments, such user manipulations via the GUI may further modify and otherwise control how rooms are interconnected, such as to specify a width of walls between rooms, to control alignment of room shapes relative to each other, etc., and/or may otherwise specify information about rooms or about a floor plan being generated. In some embodiments, such displayed information in the GUI may include displayed panorama images of one or more of the rooms in one or more distinct sections or ‘panes’ of the GUI, with additional displayed information overlaid on some or all of those displayed panorama images to show information about one or more other rooms (e.g., an outline of some or all borders of a second room that is overlaid on a panorama image of a first room in a location within the image at which that second room would be situated if connected to the first room via specified connected inter-room openings of the two rooms). In addition, in some embodiments, such displayed information may include a displayed floor plan pane of the GUI that shows room shapes of two or more rooms in locations relative to each other that reflect the rooms being connected via specified inter-room openings of the rooms (e.g., a 2D overhead view outline of the walls and wall openings for the room, with the connected inter-room openings being located adjacent to or on top of each other, and optionally to have walls of the two rooms that are within a defined threshold amount of being parallel being adjusted to be parallel). In such embodiments with multiple panes each showing different information (e.g., a first pane showing a first panorama image of a first room with a first inter-room opening; a second pane showing a second panorama image of a second room with a second inter-room opening to potentially connect to the first room via a connection between the first and second inter-room openings, such as to show that the first and second inter-room openings are two sides of the same wall opening between the first and second rooms; a third pane showing a floor plan view with room shapes of at least the first and second rooms, and possibly other connected rooms; and optionally one or more additional panes showing additional panorama images of additional rooms to potentially connect to one or more of the first and second rooms), the displayed information between the panes may be coordinated in the GUI, such as to simultaneously update corresponding information in other panes as a user manipulates information in one of the panes (e.g., to change relative locations of the first and second rooms as the user adjusts location of at least one of the rooms in one of the panes). In this manner, the generation of a floor plan for the building and optionally other associated mapping information may include using the inter-room passage information and other information to determine relative global positions of the associated room shapes to each other in a common coordinate system or other common frame of reference (e.g., without knowing the actual measurements of the rooms)—in addition, if distance scaling information is available for one or more of the images, corresponding distance measurements may be determined, such as to allow room sizes and other distances to be determined and further used for the generated floor plan. Additional details are included below related to such GUIs and associated user interactions techniques for use in generating floor plans.
In addition, the automated operations of the computing device(s) to provide the described techniques may in some embodiments and situations further include operations of the MIGM system to construct a room connection graph that includes information about possible and/or actual connections between rooms of a building via inter-room wall openings, and/or to use such a room connection graph to assist in generating a floor plan for the building. In at least some embodiments, the room connection graph includes a node for each of the rooms, with each node having information about each inter-room wall opening passage into and out of the room (as well as various other information about the room). For each such inter-room wall opening passage of a room, one or more links are added in the room connection graph to one or more other room nodes in order to represent possible or actual connections from that inter-room wall opening passage to other inter-room wall opening passages of the one or more other rooms represented by the other room node(s). For example, in some embodiments, an initial version of the room connection graph is constructed that is fully connected to include links for each possible connection between two inter-room wall opening passages of two rooms, with the connections that are determined to be possible being limited in at least some such embodiments by factors such as the types of passages (e.g., so that a door opening in one room only has potential connections to door openings of other rooms; a stair opening in one room only has potential connections to stair openings of other rooms; etc.) and/or the sizes of the passages (e.g., so that two inter-room wall openings only have a potential connection if they are of the same size, such as the same width and/or the same height, or have two different sizes that differ by at most a defined threshold size amount). In such embodiments, the links representing possible connections for a particular inter-room wall opening of a particular room may be used during floor plan generation to identify candidates for other rooms and inter-room wall openings to which that particular inter-room wall opening and particular room may be connected, with such candidate rooms and/or candidate inter-room connections optionally being ranked or otherwise assessed by the MIGM system to assist in the floor plan generation, such as to initially present a highest ranked candidate room and inter-room wall opening as a suggestion to an MIGM system operator user for use with the particular inter-room wall opening of the particular room—furthermore, once a particular inter-room connection between two particular inter-room wall openings of two rooms is identified during floor plan generation (whether automatically by the MIGM system and/or manually by one or more MIGM system operator users, such as for such a user to confirm an automated suggestion by the MIGM system), the links corresponding to other possible connections for those two particular inter-room wall openings may be removed from the room connection graph, such that the final room connection graph after that floor plan generation process is completed reflects the final actual inter-room connections shown in the generated floor plan. In embodiments in which possible inter-room connections are ranked or otherwise assessed, the assessment may be done in various ways, such as using one or more cost metrics that assess aspects related to the possible inter-room connection and its two rooms. Additional details are included below related to such a room connection graph and techniques for assessing possible inter-room connections.
The automated operations of the computing device(s) to provide the described techniques may in some embodiments and situations further include operations of the MIGM system to assess images and/or their associated acquisition metadata in order to generate information about room layouts of rooms of a building for use during generation of a floor plan of the building. In at least some embodiments, such room layout information for a room includes a shape of the room (e.g., a 2D overhead view of a rectangular shape or other shape of walls of the room) and/or locations of inter-room wall openings in the room, optionally along with additional information such as types of inter-room wall openings (e.g., a door or stair or other inter-room wall opening), sizes of inter-room wall openings (e.g., width and/or height), types of the rooms (e.g., kitchen, bathroom, bedroom, etc.), dimensions of the rooms (e.g., widths and/or heights of each of the walls), etc. Some or all such room layout information for a room may be determined from one or more images captured in the room in various manners in various embodiments, such as by applying machine learning techniques to automatically assess the image(s) (e.g., supplying the image(s) as input to one or more neural networks that have been trained using other images and associated room layout information to identify one or more such types of room layout information, and obtaining the corresponding room layout information as output from the trained neural networks), and/or by using information supplied by one or more users (e.g., MIGM system operator users) that assess the image(s) to determine some or all of the room layout information. In some embodiments in which acquisition metadata for an image captured at an acquisition location in a room includes depth data from one or more depth sensors at the acquisition location to surrounding walls or other objects of the room, such depth information may be used to determine some or all such room layout information, whether by using such depth information together with other of the described image assessment techniques or instead using only such depth information. Thus, such assessment techniques of one or more images acquired in a room may provide various types of room information in various embodiments and situations, including to identify structural and other visual features of the room, such as to identify one or more of the following: borders between adjacent walls; borders between walls and a floor; borders between walls and a ceiling; windows and/or sky-lights; passages into and/or out of the room, such as doors and stairs and other wall openings; other structures (e.g., represented as cuboid shapes), such as countertops, bath tubs, sinks, fireplaces, and furniture; etc. Additional details are included below related to such generation and use of room layout information for rooms based on assessment of images and/or their associated acquisition metadata.
Additional details are included below regarding further automated operations of computing device(s) implementing an MIGM system as part of performing additional automated analyses of information about the buildings and/or information received from MIGM system operator user(s), as well as in interacting with the MIGM system operator user(s). In some embodiments, one or more types of additional processing may be further performed, such as to determine additional mapping-related information for a generated floor plan or to otherwise associate additional information with a generated floor plan. As one example, one or more types of additional information about a building may be received and associated with the floor plan (e.g., with particular locations in the floor plan), such as additional images, textual and/or audio annotations or other descriptions of particular rooms or other locations, other audio information, such as recordings of ambient noise; overall dimension information, etc. As another example, in at least some embodiments, additional processing of images is performed to determine estimated distance information of one or more types, such as to measure sizes in images of objects of known size, and use such information to estimate room width, length and/or height dimensions—such estimated size information for one or more rooms may be associated with the floor plan, stored and optionally displayed, and if the size information is generated for all rooms within a sufficient degree of accuracy, a more detailed floor plan of the building may further be generated, such as with sufficient detail to allow blueprints or other architectural plans to be generated. In addition, if estimated size information includes height information from floors to ceilings, a 3D (three-dimensional) model (e.g., with full height information represented) and/or 2.5D (two-and-a-half dimensional) model (e.g., with partial representations of height shown) of some or all of the 2D (two-dimensional) floor plan may be created (optionally with information from in-room images projected on the walls of the models), associated with the floor plan, stored and optionally displayed. Other types of additional information may be generated or retrieved and used in some embodiments, such as to determine a geographical alignment for a building (e.g., with respect to true north or magnetic north) and/or geographical location for a building (e.g., with respect to latitude and longitude, or GPS coordinates; for a street address; etc.), and to optionally include corresponding information on its generated floor plan and/or other generated mapping-related information, and/or to optionally further align the floor plan or other generated mapping-related information with other associated external information (e.g., satellite or other external images of the building, including street-level images to provide a ‘street view’ of the building and/or panorama images acquired at one or more locations in a yard or other area around a building; information for an area in which the building is located, such as nearby street maps and/or points of interest; etc.). Other information about the building may also be retrieved from, for example, one or more external sources (e.g., online databases, ‘crowd-sourced’ information provided by one or more end users, etc.), and associated with and linked to the floor plan and/or to particular locations within the floor plan—such additional information may further include, for example, exterior dimensions and/or shape of the building, additional images and/or annotation information acquired corresponding to particular locations within the building (optionally for locations different from acquisition locations of the acquired panorama or other images), etc. Such generated floor plans and optionally additional associated information may further be used in various manners, as discussed elsewhere herein.
The described techniques provide various benefits in various embodiments, including to allow floor plans of multi-room buildings and other structures to be generated from images acquired in the buildings or other structures via automated operations of one or more computing systems (including in some embodiments to perform automated operations to interact with one or more users to obtain one or more types of user-supplied input that is used for further automated analysis), including in some embodiments without having or using acquired depth information from depth sensors about distances from images' acquisition locations to walls or other objects in a surrounding building or other structure. Furthermore, such automated techniques allow such a floor plan to be generated much more quickly than previously existing techniques, and in at least some embodiments with greater accuracy, based at least in part on using information acquired from the actual building environment (rather than from plans on how the building should theoretically be constructed), including based on using 360° spherical panorama images in an equirectangular format that display an entire room and allow efficient user identification of elements of interest in the room, as well as enabling the capture of changes to structural elements that occur after a building is initially constructed. Such described techniques further provide benefits in allowing improved automated navigation of a building by mobile devices (e.g., semi-autonomous or fully-autonomous vehicles), including to significantly reduce their computing power used and time used to attempt to otherwise learn a building's layout. In addition, in some embodiments the described techniques may be used to provide an improved GUI in which an end user may more accurately and quickly obtain information about a building's interior (e.g., for use in navigating that interior, such as via a virtual tour), including in response to search requests, as part of providing personalized information to the end user, as part of providing value estimates and/or other information about a building to an end user, etc. Various other benefits are also provided by the described techniques, some of which are further described elsewhere herein.
For illustrative purposes, some embodiments are described below in which specific types of information are acquired, used and/or presented in specific ways for specific types of structures and by using specific types of devices—however, it will be understood that the described techniques may be used in other manners in other embodiments, and that the invention is thus not limited to the exemplary details provided. As one non-exclusive example, while floor plans may be generated for houses that do not include detailed measurements for particular rooms or for the overall houses, it will be appreciated that other types of floor plans or other mapping information may be similarly generated in other embodiments, including to generate 3D models, and to do so for buildings (or other structures or layouts) separate from houses. As another example, while floor plans for houses or other buildings may be used for display to assist viewers in navigating the buildings, generated mapping information may be used in other manners in other embodiments. In addition, the term “building” refers herein to any partially or fully enclosed structure, typically but not necessarily encompassing one or more rooms that visually or otherwise divide the interior space of the structure—non-limiting examples of such buildings include houses, apartment buildings or individual apartments therein, condominiums, office buildings, commercial buildings or other wholesale and retail structures (e.g., shopping malls, department stores, warehouses, etc.), etc. The term “acquire” or “capture” as used herein with reference to a building interior, acquisition location, or other location (unless context clearly indicates otherwise) may refer to any recording, storage, or logging of media, sensor data, and/or other information related to spatial and/or visual characteristics of the building interior or subsets thereof, such as by a recording device and/or by another device that receives information from the recording device. In addition, various details are provided in the drawings and text for exemplary purposes, but are not intended to limit the scope of the invention. For example, sizes and relative positions of elements in the drawings are not necessarily drawn to scale, with some details omitted and/or provided with greater prominence (e.g., via size and positioning) to enhance legibility and/or clarity. Furthermore, identical reference numbers may be used in the drawings to identify similar elements or acts.
An MIGM (Mapping Information Generation Manager) system 140 is further executing on one or more server computing systems 180 to generate and provide building floor plans 145 and/or other mapping-related information (not shown) based on use of the panorama images 165 and optionally additional associated information, as well as by using supporting information supplied by MIGM system operator users via computing devices 105 over intervening computer network(s) 170—additional details related to the automated operation of the MIGM system are included elsewhere herein, including with respect to
Various components of the mobile image acquisition device 185 are also illustrated in
In the example of
One or more end users (not shown) of one or more map viewer client computing devices 175 may further interact over computer networks 170 with the MIGM system 140 (and optionally the ICA system 160), such as to obtain, display and interact with a generated floor plan. In addition, while not illustrated in
In the depicted computing environment of
In operation, the mobile image acquisition device 185 arrives at a first acquisition location 210A within a first room of the building interior (in this example, in a living room accessible via an external door 190-1), and captures a view of a portion of the building interior that is visible from that acquisition location 210A (e.g., some or all of the first room, and optionally small portions of one or more other adjacent or nearby rooms, such as through doors, halls, stairs or other connecting passages from the first room). The view capture may be performed in various manners as discussed herein, and may include a number of objects or other features (e.g., structural details) that may be visible in images captured from the acquisition location—in the example of
After the first acquisition location 210A has been captured, the mobile device 185 may move or be moved to a next acquisition location (such as acquisition location 210B), optionally recording images and/or video and/or other data from the hardware components (e.g., from one or more IMUs, from the camera, etc.) during movement between the acquisition locations. At the next acquisition location, the mobile device may similarly capture a 360° panorama image from that acquisition location. This process may repeat for some or all rooms of the building and in some cases external to the building, as illustrated for acquisition locations 210C-210M in this example. The acquired panorama images for each acquisition location may be further analyzed, including in some embodiments to render or otherwise place each panorama image in an equirectangular format, whether at the time of image capture or later.
Various details are provided with respect to
In particular,
In the example of
More generally, in at least some embodiments, one or more metrics that measure the degree of match (or lack of match) of a potential connection between two wall openings of two rooms may be based on one or more of a variety of factors, such as the following non-exclusive list: an alignment of walls of the two rooms if the two rooms are connected via the two wall openings (e.g., to decrease the weight given to a candidate as the alignment improves); an amount of overlap between panorama images of the two rooms if one room's panorama image is overlaid on the other room's panorama image at a position reflecting the two rooms being connected via the two wall openings (e.g., to decrease the weight given to a candidate as the similarity or other fit increases between the two images' visual information, such as based on alignment or other agreement of one or more of lines, colors, brightness/darkness, etc.); an amount of wall space (if any) between the walls of the two rooms if the two rooms are connected via the two wall openings (e.g., to decrease the weight given to a candidate as the amount of space between wall(s) of the candidate room and wall(s) of the other existing rooms at a location of the inter-room connection approaches an expected or otherwise specified amount of space); a closeness in order and/or time of capturing the panorama images for the two rooms as part of capturing panorama images throughout some or all of the rooms of the building (e.g., to decrease the weight given to a candidate as the difference decreases between the order and/or time of capturing the two panorama images); etc. Furthermore, the score from such metrics may either increase or decrease as the degree of match increases, depending on the embodiment and situation, with the term ‘cost’ metric generally referred to herein as having a decreasing score (or ‘cost’) as the degree of match increases, unless indicated otherwise.
In at least some embodiments, multiple cost metrics are further used together, and a candidate room connection's ranking or other assessment score is based on combining cost information from multiple cost metrics, such as to use a weighted sum of the multiple cost metrics, or to instead use machine learning techniques to train a neural network or other model to provide an aggregate cost score based on a combination of information from the multiple cost metrics. Non-exclusive examples of multiple cost metrics that may be used in this manner include the following:
(1) a shape-intersection-area cost, such as to measure how much a candidate room's room shape location is overlapping with the existing partial floor plan. As one example, this cost can be measured with the following equation (a), using ‘target shape’ to refer to the room shape of the candidate room, ‘reference shape’ to refer to the room shape(s) of the one or more rooms in the current partial floor plan, and ‘shape intersection’ to refer to the overlap of the candidate room's room shape with the existing room shape(s):
Cost=step function (area of shape intersection/area of target shape)+step function (area of shape intersection/area of reference shape) (a)
In at least some embodiments, some degree of room shape overlap is permitted (e.g., in large open spaces), with this cost metric considering wall openings, and optionally using ray casting to decide if an overlapping area is behind an opening or not (with overlapping areas behind openings not being counted as overlaps).
(2) a panorama capture order cost, such as to measure a similarity in order and/or time between when two panorama images for two rooms are captured, and to produce a higher score if a candidate/target room's image capture increases in order and/or time from that of the image capture(s) of the current room(s) in the partial floor plan to which the candidate room may be connected.
(3) a shape alignment cost, such as to measure how well the walls of the candidate/target room's room shape (or corresponding panorama image) aligns with the room shape(s) (or corresponding panorama image(s)) of the current room(s) in the partial floor plan to which the candidate room may be connected, and to optionally include both negative costs (or ‘rewards) and positive costs. For example, if a candidate/target room's room shape has a wall that blocks an inter-room wall opening of a current room of the partial floor plan (or vice versa), a specified (positive) cost will be included, while other walls separate from the location of the inter-room connection that are parallel or perpendicular may in some embodiments can a specified negative cost (reward) to be included.
(4) a snapping feature similarity cost, such as to measure the similarity of two inter-room wall openings to be connected (e.g., by their size and/or their type, including in some embodiments to classify doors into sub-categories such as entry door, inter-room door, closet door, etc.).
(5) a shape orientation cost, such as to measure how well, if a potential connection between two inter-room wall openings is made, the orientation of the candidate/target room's room shape is aligned with the room shape(s) of the current room(s) of the partial floor plan.
(6) a room object spatial reprojection cost, such as to measure a degree to which wall feature positions match expected positions when an image reprojection is performed. For example, given known wall features for a room shape (e.g., wall features automatically determined with an object detection pipeline, manually annotated by one or more users, etc.), and an algorithm for object localization within an image (e.g., the ‘Faster R-CNN’ algorithm, or the faster region-based convolutional neural network algorithm), bounding boxes may be generated from panorama images for wall openings (e.g., doors, windows and other wall openings), and a wall feature position may be computed from a candidate/target room's room shape in a panorama image for a current room to be connected to the candidate/target room via inter-room wall openings of the rooms—such a reprojected wall feature bounding box may then be compared with bounding boxes generated for the panorama image for the current room for a corresponding wall feature (e.g., a wall opening). A corresponding cost value can be calculated in some embodiments with the following equation (b):
Cost=Area of Bounding Box Intersection/Area of Bounding Box from Panorama Image For Current Room (b)
(7) an image camera spatial relation based cost, such as to measure a camera relative angle between a panorama image for a candidate/target room and a panorama image for a current room of the existing floor plan to which the candidate/target room is to be connected. For example, a panorama visual connection (PVC) angle may be generated to represent the angle between 2 panorama images' camera positions in space, such as to compute the PVC angle using feature matching between two such panorama images to solve the homograph matrix between them, or using a convolutional neural network-based image patch matching algorithm, or using SLAM (simultaneous localization and mapping) and compass information. For a candidate/target room and a current room of an existing partial floor plan, if they are connected, the camera relative angle can be computed between panorama images for the two rooms, and can be compared to the PVC angle, with the costs increasing as the angles diverge.
(8) an image content similarity cost, such as to measure a degree of similarity between 2 panorama images that have visual overlaps. For example, convolutional neural networks (e.g., a Siamese neural network architecture) can be used to compare 2 panorama images to determine if they have any visual overlaps and to compute where any such image overlaps occur in image space. Such a convolutional neural network may thus be used to determine the visual overlaps between two panorama images of a candidate/target room and current room of an existing partial floor plan if connected via two inter-room wall openings, the visual overlaps between two such panorama images may be further inferred using ray casting, and the content similarity between the panorama images may be measured based on the consistency between the inferred visual overlap and the determined visual overlap.
(9) a cost related to room shape alignment, such as to reduce the cost as 2 shapes become more aligned and a line segment composing a room shape polygon becomes aligned.
(10) a cost based on a topological relation related to room type (e.g., adjacency of room types), such as to reduce the cost for room types that have a higher likelihood of being close to each other (e.g., a bathroom typically being close to a bedroom and/or a hallway, and typically being less close to a patio).
(11) a cost based on a shape alignment between a shape of combined rooms and an exterior contour of the structure. For example, an exterior contour of a house and/or a location or alignment of windows and doors can be determined based on analysis of exterior images of the structure (e.g., images taken from the group, from the air, from satellites, etc.), and a cost for combining rooms may be reduced as the shape of the combined rooms matches the exterior contour and/or other location/alignment information.
In addition, with respect to using machine learning techniques in some embodiments to train a neural network or other model to provide an aggregate cost score based on a combination of information from the multiple cost metrics, at least some such embodiments may concatenate the features or other information for each cost item corresponding to an application of one of the multiple cost metrics, and supply them as input to a trained machine learning network (e.g., a trained decision tree) to produce a final aggregate cost score.
In particular, in the example embodiment displayed in
As the number of rooms in the partial floor plan increases, additional information may be available to improve rankings of candidate room connection options, as reflected in aspects 266a-266g illustrated in pane 255h for the benefit of the reader (and optionally displayed to the MIGM system operator user in some embodiments, such as if the user requests details about likelihood rankings or other assessment scores of one or more candidate room connection options). For example, with respect to option 255h1 in which the doorway 190-3 of the hallway is interconnected with a doorway of a displayed room shape 242b, aspect 266a corresponds to a potential connection of the two doorways, which in this example appears to match doorway openings of the same size—furthermore, the west and north walls of the room shape 242b appear to be generally aligned with the east wall of the living room and south wall of the hallway, respectively, and the south wall of the room shape 242b appears to generally be aligned with the south wall of the living room, and an initial width 264h is shown corresponding to a wall depth between the west wall of the room shape 242b and the east wall of the living room, and a similar wall depth is shown between the north wall of the room shape 242b and the south wall of the hallway. In this example, the room shape 242b is labeled as corresponding to a “bedroom 1” room, such as if that annotation information is available with room layout information for that room, while in other embodiments such room labels may be added by the MIGM system operator user (whether at a time of connecting that room into the partial floor plan or another time, such as after all of the room shapes have been interconnected) or may not be displayed at this time.
Returning to
In a manner similar to that of
The MIGM system may continue to perform automated operations to iteratively connect additional rooms to the existing partial floor plan, with one example of a resulting completed floor plan being illustrated and discussed with respect to
While not illustrated in
After all of the room shape interconnection and other layout information has been specified for the house, whether automatically by the MIGM system and/or by using information supplied by one or more MIGM system operator users, the final results may be used to generate a 2D floor plan of the house, optionally after final optimizations have been performed and visual aspects of the final floor plan have been added—such final optimizations may include, for example, one or more of ensuring consistent visual aspects (e.g., line widths, colors, text styles, etc.), placing textual room labels at preferred locations on the final floor plan, adding missing spaces such as small closets or other additional areas not included on the defined room shape layouts (e.g., areas that did not have any images taken from within them, resulting in empty spaces within an exterior of the building that are not identified in the defined room shape layouts), merging multiple overlapping and/or adjacent walls, correcting any geometric anomalies, etc. In at least some embodiments, the described techniques may include performing at least some such updates in an automated manner, and optionally providing corresponding GUI tools for one or more users to make final manual adjustments (e.g., GUI tools similar to those of a drawing or painting program) to a floor plan for the house that is generated.
Various details have been provided with respect to
In addition, a variety of additional automated functionalities (and in at least some embodiments, associated GUI functionality for use by one or more MIGM system operator users) may be provided by and/or used via the MIGM system in at least some embodiments. As one example, in some embodiments functionality may be provided to combine multiple panorama images that are taken in a single room into a single panorama image, such as by localizing information for one of the panorama images into the space of the other panorama image—for example, both panorama images may be displayed to a user who selects one or more common points in both images (e.g., a common plane with infinite points in both images), with the MIGM system determining the corresponding locations of the visual information of the two panorama images based on the indicated common point(s). After such a combination panorama image is created, it may be further used in a similar manner to that of other panorama images, as discussed in greater detail elsewhere herein. In addition, in some embodiments one or more additional supplemental panorama images are used in combination with a single primary panorama image for each room, such as to generate a supplemental panorama image at the location of each of one or more inter-room connections in order to assist in determining the connection between those rooms (e.g., alignment and other layout of the room shapes of the rooms), such as by using information in the supplemental panorama image to match to corresponding features in the panorama images for each of the connecting rooms. Moreover, in some embodiments additional functionality may be provided via the MIGM system to perform a global optimization of a generated floor plan, such as to identify final alignments of walls and other room shape information.
In addition, in some embodiments additional functionality may be provided via the MIGM system to refine transformations of room shapes, such as by providing an optimization that uses alignment of line segments and a top-down view or by using direct image overlaps (e.g., via rendering). Moreover, in some embodiments additional functionality may be provided via the MIGM system to perform or assist with a selection of a first room shape to begin generation of a floor plan, such as based on an automated analysis of information about that room (e.g., relative to that of other rooms of the building), and/or based on information supplied by an MIGM system operator user (e.g., after information about some or all room shapes is displayed or otherwise provided to the user)—as one example, a room may be selected to be used as a starting room based on one or more factors, such as having the most inter-room wall openings, the least inter-room wall openings, a wall opening corresponding to an exterior door (e.g., the entry to the building), the order of panorama image capture (e.g., to use the room corresponding to the first panorama image capture or the last panorama image captured), etc.
Furthermore, in some embodiments additional functionality may be provided via the MIGM system to align or otherwise connect multiple floors or other levels of the building, such as via connecting stairway's or other connecting passages. Such additional functionality may include, for example, aligning the multiple floors of a house into a single coordinate system so that they can all be rendered as a 3D model (e.g., in a rendering system), and/or aligning the multiple floors of a house in 2D so that they can be overlaid in a top-down orthographic projection (e.g., in a CAD system or in architectural blueprints). As one non-exclusive example, one way to implement connections between rooms on two separate floors is to use panorama images that show the stairway connecting the two floors, such as a panorama image at one or both of the bottom and top of the stairs (e.g., for a straight stairway that directly connects the floors, without any stair landing), and to interconnect the wall openings of the rooms at the top and bottom of the stairs in a manner similar to other wall opening connections (such as by including a horizontal distance between the two wall openings corresponding to a measured or estimated length of the stairs, and optionally including vertical information between the two wall openings if available), and with the sub-floor plans for two such floors being rotated in a consistent manner and at corresponding positions in 3D space. Estimates of the height difference and horizontal distance between two such wall openings at the ends of a stairway may be determined, for example, if a height of a stairway step is known (e.g., the height of a riser and the tread above it) and/or if a panorama image is available in which both the stairway foot and head ends are visible (e.g., from the top or bottom of a straight stairway; from each stairway landing of a non-straight stairway, such as by treating each such landing as an ‘intermediate’ floor that is connected with the other floors in a manner analogous to that of connecting two floors for a straight stairway; etc.) that enables determination of horizon line(s) in the panorama image corresponding to the stairway foot and/or head. In some embodiments and situations, a height and/or depth of a step could be measured during panorama image capture, whether manually or automatically using an object of known size on one of the steps. In addition, the quantity of steps may be automatically determined using image processing in some embodiments, with that information combined with step depth information to determine a horizontal length of the stairway (optionally accounting for the nosing/overhang of the stair tread over the riser) and/or with step height information to determine a vertical height of the stairway. In this manner, such embodiments may perform the connection of rooms on two floors using relatively sparse geometric features with clear semantic meanings (e.g., the lines in one or more panorama images representing the foot and head of a stairway) in a minimal number of captured panorama images, rather than using dense data with numerous panorama images (e.g., to provide dense visual connection information) and/or other dense measurements taken along a stairway.
In addition, in at least some embodiments additional information may be used as part of generating a floor plan for a building that is obtained outside of the building, such as one or more panorama images acquired outside of the building (e.g., in which some or all of the building is visible), one or more panorama images acquired of outbuildings or other structures on the same property, satellite and/or drone images from overhead, images from a street adjacent to the building, information from property records or other sources about dimensions of the exterior of the building, etc. As one example, one or more exterior panorama images may be used to identify a shape of an exterior wall of the building, the quantity and/or locations of one or more windows in the exterior wall, identification of one or more floors of the building that are visible from that exterior, etc., such as from an automated analysis of the panorama images and/or based on manual annotations of the panorama images by one or more MIGM system operator users, and with such information subsequently used to eliminate/select and/or to rank possible room connections according to how they fit with the information acquired from the exterior panorama image(s). As another example, one or more exterior panorama images may be treated as being part of one or more exterior rooms that surround or are otherwise associated with the building, with the exterior rooms being modeled (e.g., with room shapes) and connected to and used with other interior rooms of the building in a floor plan and/or in other manners. It will be appreciated that a variety of other types of functionality may similarly be provided in at least some embodiments.
The server computing system(s) 300 and executing MIGM system 340, and server computing system(s) 380 and executing ICA system 389, may communicate with each other and with other computing systems and devices in this illustrated embodiment via one or more networks 399 (e.g., the Internet, one or more cellular telephone networks, etc.), such as to interact with user client computing devices 390 (e.g., used by system operator users of the MIGM or ICA systems to interact with those respective systems; used by end users to view floor plans, and optionally associated images and/or other related information; etc.), and/or mobile image acquisition devices 360 (e.g., used to acquire panorama images and optionally other information for buildings or other environments to be modeled), and/or optionally other navigable devices 395 that receive and use floor plans and optionally other generated information for navigation purposes (e.g., for use by semi-autonomous or fully autonomous vehicles or other devices). In other embodiments, some of the described functionality may be combined in less computing systems, such as to combine the MIGM system 340 and the image acquisition functionality of device(s) 360 in a single system or device, to combine the ICA system 389 and the image acquisition functionality of device(s) 360 in a single system or device, to combine the MIGM system 340 and the ICA system 389 in a single system or device, to combine the MIGM system 340 and the ICA system 389 and the image acquisition functionality of device(s) 360 in a single system or device, etc.
In the illustrated embodiment, an embodiment of the MIGM system 340 executes in memory 330 of the server computing system(s) 300 in order to perform at least some of the described techniques, such as by using the processor(s) 305 to execute software instructions of the system 340 in a manner that configures the processor(s) 305 and computing system 300 to perform automated operations that implement those described techniques. The illustrated embodiment of the MIGM system may include one or more components, not shown, to each perform portions of the functionality of the MIGM system, and the memory may further optionally execute one or more other programs 335—as one specific example, a copy of the ICA system may execute as one of the other programs 335 in at least some embodiments, such as instead of or in addition to the ICA system 389 on the server computing system(s) 380. The MIGM system 340 may further, during its operation, store and/or retrieve various types of data on storage 320 (e.g., in one or more databases or other data structures), such as one or more of the following: acquired 360° panorama image information 324, such as from ICA system 389 (e.g., for analysis to produce room layout information; for use to assist in generating floor plans; to provide to users of client computing devices 390 for display; etc.); generated or received information 325 about room layouts for rooms of one or more buildings (e.g., room shapes and locations of doors and windows and other wall openings in walls of the rooms); generated floor plans and other associated mapping information 326 for one or more buildings (e.g., generated and saved 2.5D and/or 3D models, building and room dimensions for use with associated floor plans, additional images and/or annotation information, etc.); optionally various types of user information 322; and/or various types of additional optional information 328 (e.g., various analytical information related to presentation or other use of one or more building interiors or other environments captured by an ICA system and/or modeled by the MIGM system).
In addition, an embodiment of the ICA system 389 executes in memory 387 of the server computing system(s) 380 in the illustrated embodiment in order to perform some of the described techniques, such as by using the processor(s) 381 to execute software instructions of the system 389 in a manner that configures the processor(s) 381 and computing system 380 to perform automated operations that implement those described techniques. The illustrated embodiment of the ICA system may include one or more components, not shown, to each perform portions of the functionality of the ICA system, and the memory may further optionally execute one or more other programs (not shown). The ICA system 389 may further, during its operation, store and/or retrieve various types of data on storage 385 (e.g., in one or more databases or other data structures), such as information 386 about acquired panorama images and optionally associated acquisition metadata, and optionally other types of information that are not shown in this example (e.g., about ICA system operator users, additional images and/or annotation information, dimension/size information for one or more images, etc.).
Some or all of the user client computing devices 390 (e.g., mobile devices), mobile image acquisition devices 360, optional other navigable devices 395 and other computing systems (not shown) may similarly include some or all of the same types of components illustrated for server computing system 300. As one non-limiting example, the mobile image acquisition devices 360 are each shown to include one or more hardware CPU(s) 361, I/O components 362, storage 365, and memory 367, with one or both of a browser and one or more client applications 368 (e.g., an application specific to the MIGM system and/or ICA system) executing within memory 367, such as to participate in communication with the MIGM system 340, ICA system 389 and/or other computing systems—the devices 360 each further include one or more imaging systems 364 and IMU hardware sensors 369 and optionally other components (e.g., a lighting system, a depth-sensing system, etc.), such as for use in acquisition of images and associated movement data of the device 360. While particular components are not illustrated for the other navigable devices 395 or other computing systems 390, it will be appreciated that they may include similar and/or additional components.
It will also be appreciated that computing systems 300 and 380 and the other systems and devices included within
It will also be appreciated that, while various items are illustrated as being stored in memory or on storage while being used, these items or portions of them may be transferred between memory and other storage devices for purposes of memory management and data integrity. Alternatively, in other embodiments some or all of the software components and/or systems may execute in memory on another device and communicate with the illustrated computing systems via inter-computer communication. Thus, in some embodiments, some or all of the described techniques may be performed by hardware means that include one or more processors and/or memory and/or storage when configured by one or more software programs (e.g., by the MIGM system 340 executing on server computing systems 300, by the ICA software 389 executing on server computing systems 380 and/or on devices 360, etc.) and/or data structures, such as by execution of software instructions of the one or more software programs and/or by storage of such software instructions and/or data structures, and such as to perform algorithms as described in the flow charts and other disclosure herein. Furthermore, in some embodiments, some or all of the systems and/or components may be implemented or provided in other manners, such as consisting of one or more means that are implemented partially or fully in firmware and/or hardware (e.g., rather than as a means implemented in whole or in part by software instructions that configure a particular CPU or other processor), including, but not limited to, one or more application-specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and/or embedded controllers), field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the components, systems and data structures may also be stored (e.g., as software instructions or structured data) on a non-transitory computer-readable storage medium, such as a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM or flash RAM), a network storage device, or a portable media article (e.g., a DVD disk, a CD disk, an optical disk, a flash memory device, etc.) to be read by an appropriate drive or via an appropriate connection. The systems, components and data structures may also in some embodiments be transmitted via generated data signals (e.g., as part of a carrier wave or other analog or digital propagated signal) on a variety of computer-readable transmission mediums, including wireless-based and wired/cable-based mediums, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). Such computer program products may also take other forms in other embodiments. Accordingly, embodiments of the present disclosure may be practiced with other computer system configurations.
The illustrated embodiment of the routine begins at block 405, where instructions or information are received. At block 410, the routine determines whether the received instructions or information indicate to acquire data representing a building interior, and if not continues to block 490. Otherwise, the routine proceeds to block 412 to receive an indication to begin the image acquisition process at a first acquisition location (e.g., from a user of a mobile image acquisition device that will perform the acquisition process). After block 412, the routine proceeds to block 415 in order to perform acquisition location image acquisition activities for acquiring a 360° panorama image for the acquisition location in the interior of the target building of interest, such as via one or more fisheye lenses and/or non-fisheye rectilinear lenses on the mobile device, and such as to provide horizontal coverage of at least 360° around a vertical axis. As one non-exclusive example, the mobile image acquisition device may be a rotating (scanning) panorama camera equipped with a fisheye lens (e.g., with 180° degrees of horizontal coverage) and/or other lens (e.g., with less than 180° degrees of horizontal coverage, such as a regular lens or wide-angle lens or ultrawide lens). The routine may also optionally obtain annotation and/or other information from the user regarding the acquisition location and/or the surrounding environment, such as for later use in presentation of information regarding that acquisition location and/or surrounding environment.
After block 415 is completed, the routine continues to block 420 to determine if there are more acquisition locations at which to acquire images, such as based on corresponding information provided by the user of the mobile device. If so, and when the user is ready to continue the process, the routine continues to block 422 to optionally initiate the capture of linking information (e.g., acceleration data) during movement of the mobile device along a travel path away from the current acquisition location and towards a next acquisition location within the building interior. The captured linking information may include additional sensor data (e.g., from one or more IMU, or inertial measurement units, on the mobile device or otherwise carried by the user) and/or additional visual information (e.g., images, video, etc.) recorded during such movement. Initiating the capture of such linking information may be performed in response to an explicit indication from a user of the mobile device or based on one or more automated analyses of information recorded from the mobile device. In addition, the routine may further optionally monitor the motion of the mobile device in some embodiments during movement to the next acquisition location, and provide one or more guidance cues to the user regarding the motion of the mobile device, quality of the sensor data and/or visual information being captured, associated lighting/environmental conditions, advisability of capturing a next acquisition location, and any other suitable aspects of capturing the linking information. Similarly, the routine may optionally obtain annotation and/or other information from the user regarding the travel path, such as for later use in presentation of information regarding that travel path or a resulting inter-panorama image connection link. In block 424, the routine determines that the mobile device has arrived at the next acquisition location (e.g., based on an indication from the user, based on the forward movement of the user stopping for at least a predefined amount of time, etc.), for use as the new current acquisition location, and returns to block 415 in order to perform the acquisition location image acquisition activities for the new current acquisition location.
If it is instead determined in block 420 that there are not any more acquisition locations at which to acquire image information for the current building or other structure, the routine proceeds to block 425 to optionally analyze the acquisition location information for the building or other structure, such as to identify possible additional coverage (and/or other information) to acquire within the building interior. For example, the ICA system may provide one or more notifications to the user regarding the information acquired during capture of the multiple acquisition locations and optionally corresponding linking information, such as if it determines that one or more segments of the captured information are of insufficient or undesirable quality, or do not appear to provide complete coverage of the building. After block 425, the routine continues to block 435 to optionally preprocess the acquired 360° panorama images before their subsequent use for generating related mapping information, such as to produce images of a particular type and/or in a particular format (e.g., to perform an equirectangular projection for each such image, with straight vertical data such as the sides of a typical rectangular door frame or a typical border between 2 adjacent walls remaining straight, and with straight horizontal data such as the top of a typical rectangular door frame or a border between a wall and a floor remaining straight at a horizontal midline of the image but being increasingly curved in the equirectangular projection image in a convex manner relative to the horizontal midline as the distance increases in the image from the horizontal midline). In block 477, the images and any associated generated or obtained information is stored for later use.
If it is instead determined in block 410 that the instructions or other information recited in block 405 are not to acquire images and other data representing a building interior, the routine continues instead to block 490 to perform any other indicated operations as appropriate, such as any housekeeping tasks, to configure parameters to be used in various operations of the system (e.g., based at least in part on information specified by a user of the system, such as a user of a mobile device who captures one or more building interiors, an operator user of the ICA system, etc.), to respond to requests for generated and stored information (e.g., to identify one or more groups of inter-connected linked panorama images each representing a building or part of a building that match one or more specified search criteria, one or more panorama images that match one or more specified search criteria, etc.), to generate and store inter-panorama image connections between panorama images for a building or other structure (e.g., for each panorama image, to determine directions within that panorama image toward one or more other acquisition locations of one or more other panorama images, such as to enable later display of an arrow or other visual representation with a panorama image for each such determined direction from the panorama image to enable an end-user to select one of the displayed visual representations to switch to a display of the other panorama image at the other acquisition location to which the selected visual representation corresponds), to obtain and store other information about users of the system, etc.
Following blocks 477 or 490, the routine proceeds to block 495 to determine whether to continue, such as until an explicit indication to terminate is received, or instead only if an explicit indication to continue is received. If it is determined to continue, the routine returns to block 405 to await additional instructions or information, and if not proceeds to step 499 and ends.
The illustrated embodiment of the routine begins at block 505, where information or instructions are received. The routine continues to block 510 to determine whether the instructions received in block 505 indicate to generate a floor plan for an indicated building, optionally along with other related mapping information or other associated information about the building, and if so the routine continues to perform blocks 512-564 to do so, and otherwise continues to block 565.
In block 512, the routine first obtains panorama images captured in multiple rooms of the indicated building and obtains room layout information for some or all of those rooms, such as a room shape and locations of wall openings (e.g., doors, stairs and other wall openings) for each room, and optionally additional related information such as room sizes and/or information about other visible features of the room. The panorama images and/or room layout information may be obtained in some embodiments by receiving it in block 505 or retrieving the information from storage, while in other embodiments the routine may analyze some or all panorama images for the building in order to dynamically determine some or all of the room layout information, as discussed in greater detail elsewhere herein.
After block 512, the routine continues to block 515, where it determines whether the instructions or other information received in block 505 indicate to determine a suggested floor plan for the entire building in an automated manner for confirmation by the user, and if so proceeds to block 550, and otherwise continues to block 520. If the routine proceeds to block 520, the routine determines an initial room of the indicated building's multiple rooms to currently use in beginning an interactive process for generating a floor plan layout for the building, and if the routine instead proceeds to block 550, the routine instead determines one or more initial current rooms of the building for use in an automated process for generating a suggested floor plan for the building.
After blocks 520 or 550, the routine continues to block 552, where it identifies recommendations of other candidate rooms (e.g., all possible other rooms) to connect to one or more inter-room wall openings of the one or more current rooms, such as to inter-connect doors, stairs, or other inter-room wall openings in the current room(s) and in the other candidate rooms. In some embodiments and situations, the identification of other recommended candidate rooms includes using a room connection graph that has nodes for each room and links between nodes that correspond to potential and/or actual inter-room connections, such as by using the links for potential inter-room connections to identify some or all such other candidate rooms—in such embodiments, if such a room connection graph does not exist, the routine may further generate the room connection graph using room layout information obtained in block 512 before its initial use, as discussed in greater detail elsewhere herein. After block 552, the illustrated embodiment of the routine continues to block 554 to rank each candidate room based on the likelihood of it being the actual room in the building that connects to the one or more inter-room wall openings of the current room(s), although in some other embodiments such ranking may not be performed—in order to perform the ranking, one or more defined cost metrics may each be used to evaluate one or more aspects of each possible connection with an associated cost, as discussed in greater detail elsewhere herein, including to aggregate multiple costs if multiple cost metrics are used to reach an overall cost ranking for each possible connection.
After block 554, the routine continues to block 556, where it determines whether the routine is currently performing an automated generation of a full suggested building floor plan, and if so continues to block 560 to select one or more of the highest ranked candidate rooms and to combine the selected room(s) with the current room(s) via the corresponding inter-room wall openings of the current and selected rooms, for use as a new combination of current rooms—in some embodiments, multiple alternative combinations of current rooms will be generated by combining different candidate rooms to the current room(s), such as to provide alternative possible floor plans. After block 560, the routine continues to block 562 to determine whether there are more rooms to connect to the current combination(s) of rooms, and if so returns to block 552 to continue the process, with subsequent operations of blocks 552 and 554 being performed separately for each combination of current rooms if there are multiple alternative combinations. If it is instead determined in block 562 that there are not more rooms to connect, the routine continues instead to block 564, where if there are multiple alternative combinations of rooms for the building, those multiple combinations are each assessed (e.g., by combining the ranking scores previously generated in block 554 for each candidate room that was added to a particular combination to generate a corresponding overall assessment score for each combination, and/or by newly using one or more cost metrics for an entire combination of connected rooms to generate a new overall assessment score), and otherwise the single combination of current rooms is selected and its corresponding assessment score optionally determined, with the combination(s) and any associated created assessment scores returned for further use as rankings of the one or more combinations as suggested floor plans for the building. Alternatively, if it is instead determined in block 556 that an automated process to complete a full suggested floor plan for the building is not being performed, the routine continues instead to block 558, where it returns indications of the current candidate rooms and their associated rankings for use as suggestions to connect to the current room(s).
After blocks 558 or 564, the routine continues to block 525, where it receives the rankings of one or more candidate rooms to add to the current room determined in block 520 (if the interactive floor plan generation process is being used) or it receives the rankings of one or more candidate room combinations that are suggested floor plans of the building (if the automated full floor plan generation process is being used), selects one or more of the highest ranked candidates, and displays information for the selected candidate(s) to the user in a GUI for confirmation or changes. In at least some embodiments and situations, the displaying of information includes displaying multiple GUI panes that have different but related information, such as to display a room layout view of a first GUI pane that shows a partial or full floor plan with interconnected room shapes for two or more rooms, and to optionally display one or more additional GUI panes that each include one or more panorama images for at least one of the rooms shown in the floor plan pane and that optionally have additional information overlaid on the displayed panorama image(s), as discussed in greater detail elsewhere herein. After block 525, the routine continues to block 530, where it receives one or more user instructions via interactions of the user with the GUI, and updates the GUI as appropriate based on those interactions, until the user confirms to use the current information in the GUI for proceeding (to use a current partial combination of rooms for further floor plan generation if the interactive floor plan generation process is being used, or to use a current full combination of rooms as the floor plan for the building if the automated full floor plan generation process is being used). In block 535, the routine then determines if there are more rooms to connect, such as if the interactive floor plan generation process is being used and all rooms have not been added to the current combination of rooms—if so, the routine continues to block 540 to select the current room combination as the current rooms to use, and then returns to block 552 to determine additional candidate rooms to connect to those current rooms. If it is instead determined in block 535 that there are not more rooms to connect, such as if the interactive floor plan generation process is being used and all rooms have been added to the current combination of rooms, or if the automated full floor plan generation process is used and a suggested floor plan has been confirmed by the user after any modifications, the routine continues to block 545.
In block 545, the routine optionally performs a global optimization of the generated floor plan layout for the indicated building, and then continues to block 588, where it stores or otherwise uses the floor plan generated for the indicated building and any other information generated in blocks 510-585, such as to provide the generated information for display on one or more client devices, provide that generated information to one or more other devices for use in automating navigation of those devices and/or associated vehicles or other entities, etc. The generated floor plan may include, for example, relative position and shape information for the various rooms without providing any actual dimension information for the individual rooms or building as a whole, and may further include multiple linked or associated sub-maps (e.g., to reflect different stories, levels, sections, etc.) of the building—it will be appreciated that if sufficiently detailed dimension information is obtained, a floor plan may be generated from the floor plan that includes dimension information for the rooms and the overall building. It will also be appreciated that while various operations are discussed herein for generating mapping information for a building, such operations made be performed in other orders in other embodiments, and that some such operations may not be performed in some embodiments.
While not illustrated in routine 500, one or more MIGM system users may perform various types of modifications to produce the final floor plan to be used, such as to produce a consistent style (e.g., line widths, colors, text styles, etc.), to add textual room labels if not previously specified and/or to place the textual room labels at preferred locations on the final floor plan, to add missing spaces such as small closets, to correct any geometric anomalies, to modify locations of images and/or other associated and linked information, etc. In addition, while not illustrated in routine 500, in other embodiments the routine may generate other types of mapping information for the building, whether instead of or in addition to a 2D schematic floor plan as discussed for this example embodiment of routine 500—non-exclusive examples of other mapping information include a 2.5D texture map in which 360° panorama images can optionally be re-projected on the geometry of the displayed texture map, a 3D structure that illustrates accurate height information as well as width and length (and in which 360° panorama images can optionally be re-projected on the geometry of the displayed 3D structure), etc. In addition, in some embodiments additional information may be generated and used, such as to determine a geographical alignment (e.g., with respect to true north or magnetic north) and/or geographical location (e.g., with respect to latitude and longitude, or GPS coordinates) for the building and corresponding parts of the generated floor plan, and to optionally further align with other external information (e.g., satellite or other external images, including street-level images to provide a ‘street view’ of the building; neighborhood information, such as nearby street maps and/or points of interest; etc.). Other information about the building may also be retrieved from, for example, one or more external sources (e.g., online databases, ‘crowd-sourced’ information provided by one or more end users, etc.), and associated with and linked to the floor plan and/or particular locations within the floor plan—such additional information may further include, for example, exterior dimensions and/or shape of the building, additional images and/or annotation information acquired corresponding to particular locations within the building (optionally for locations different from acquisition locations of the acquired panorama or other images), etc.
If it was determined in block 510 to not generate a floor plan for an indicated building, the routine continues instead to block 565 to determine whether the instructions or other information received in block 505 are to combine multiple panorama images for a room into a single overall panorama image for that room, such as by localizing one of the panorama images within the other. If so, the routine continues to block 567, where it receives two panorama images taken in the same room, and displays them to the user in the GUI. In block 569, the routine then receives one or more user manipulations via the GUI to indicate at least one common point in the two images (e.g., optionally a common plane having infinite points in each of the two images), and in block 573 uses the common point information to combine information from the two images into one overall panorama image, and stores the combined information as a panorama image to later use for the room.
After block 573, or if it was instead determined in block 565 that the instructions or other information received in block 505 are not to combine multiple panorama images for a room, the routine continues to block 575, where it determines whether the instructions or other information received in block 505 are to combine information for multiple rooms together into a single overall room (e.g., if the two or more separate rooms are partial rooms that are not fully separated from each other by complete walls). If so, the routine continues to block 576, where indications of the two or more rooms to combine are received, and information about the two or more rooms is displayed to the user in the GUI (e.g., by displaying their room shapes, associated panorama images, etc.). In block 578, the routine then receives one or more user manipulations via the GUI to indicate how to combine the layout information for the multiple rooms into a new single room, and stores the combination information to later use with the new single room.
After block 578, or if it was instead determined in block 575 that the instructions or other information received in block 505 are not to combine multiple rooms into one, the routine continues to block 580, where it determines whether the instructions or other information received in block 505 are to obtain and store user-specified information related to a building. If so, the routine continues to block 585, where it receives user-specified information about one or more rooms (e.g., dimensions, annotations, etc.), and stores the information for later use.
After block 585, or if it was instead determined in block 580 that the instructions or other information received in block 505 are not to obtain user-specified information, or in block 510 that the information or instructions received in block 505 are not to generate a floor plan for an indicated building, the routine continues instead to block 590 to perform one or more other indicated operations as appropriate. Such other operations may include, for example, receiving and storing panorama images for a building for later use, receiving or generating room layout information for rooms of a building (e.g., based on images and/or depth information acquired in the rooms) and storing it for later use, automatically combining information for two or more panorama images taken in a room to determine room layout information (e.g., based on one or more common points identified in each of the images), receiving and responding to requests for previously generated floor plans and/or other generated information (e.g., requests for such information for display on one or more client devices and/or to provide to one or more other devices for use in automated navigation), obtaining and storing information about buildings for use in later floor plan generation operations (e.g., information about exterior images, dimensions, numbers or types of rooms, total square footage, etc.), etc.
After blocks 588 or 590, the routine continues to block 595 to determine whether to continue, such as until an explicit indication to terminate is received, or instead only if an explicit indication to continue is received. If it is determined to continue, the routine returns to block 505 to wait for and receive additional instructions or information, and otherwise continues to block 599 and ends.
The illustrated embodiment of the routine begins at block 605, where instructions or information are received. After block 605, the routine continues to block 660 to determine whether the instructions or other information received in block 605 are to select one or more target buildings using specified criteria, and if not continues to block 670, where it obtains an indication of a target building to use from the user (e.g., based on a current user selection, such as from a displayed list or other user selection mechanism; based on information received in block 605; etc.). Otherwise, if it is determined in block 660 to select one or more target buildings from specified criteria, the routine continues instead to block 662, where it obtains indications of one or more search criteria to use, such as from current user selections or as indicated in the information or instructions received in block 605, and then searches stored information about buildings to determine one or more of the buildings that satisfy the search criteria. In the illustrated embodiment, the routine then further selects a best match target building from the one or more returned buildings (e.g., the returned other building with the highest similarity or other matching rating for the specified criteria, or using another selection technique indicated in the instructions or other information received in block 605).
After blocks 662 or 670, the routine continues to block 610 to determine whether the instructions or other information received in block 605 are to display information or otherwise present information about a target building (e.g., via a floor plan that includes information about the interior of the target building), such as the target building from blocks 662 or 670, and if not continues to block 690. Otherwise, the routine proceeds to block 612 to retrieve a floor plan map for the building, optionally with associated linked information for the floor plan and/or a surrounding location, and selects an initial view of the retrieved information (e.g., a view of the floor plan). In block 615, the routine then displays or otherwise presents the current view of the retrieved information, and waits in block 617 for a user selection. After a user selection in block 617, if it is determined in block 620 that the user selection corresponds to the current building or other current location (e.g., to change the current view), the routine continues to block 622 to update the current view in accordance with the user selection, and then returns to block 615 to update the displayed or otherwise presented information accordingly. The user selection and corresponding updating of the current view may include, for example, changing how the current view is displayed (e.g., zooming in or out, rotating information if appropriate, selecting a new portion of the current view to be displayed or otherwise presented that was not previously visible, etc.), displaying or otherwise presenting a piece of associated linked information that the user selects (e.g., a particular other image), etc.
If it is instead determined in block 610 that the instructions or other information recited in block 605 are not to present information representing a building interior, the routine continues instead to block 690 to perform any other indicated operations as appropriate, such as any housekeeping tasks, to configure parameters to be used in various operations of the system (e.g., based at least in part on information specified by a user of the system, such as a user of a mobile device who captures one or more building interiors, an operator user of the MIGM system, etc.), to obtain and store other information about users of the system, to respond to requests for generated and stored information, etc.
Following block 690, or if it is determined in block 620 that the user selection does not correspond to the current location, the routine proceeds to block 695 to determine whether to continue, such as until an explicit indication to terminate is received, or instead only if an explicit indication to continue is received. If it is determined to continue (e.g., if the user made a selection in block 617 related to a new building or other new location to present), the routine returns to block 605 to await additional instructions or information (or to continue on to block 612 if the user made a selection in block 617 related to a particular new building or other new location to present), and if not proceeds to step 699 and ends.
Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be appreciated that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions. It will be further appreciated that in some implementations the functionality provided by the routines discussed above may be provided in alternative ways, such as being split among more routines or consolidated into fewer routines. Similarly, in some implementations illustrated routines may provide more or less functionality than is described, such as when other illustrated routines instead lack or include such functionality respectively, or when the amount of functionality that is provided is altered. In addition, while various operations may be illustrated as being performed in a particular manner (e.g., in serial or in parallel, or synchronous or asynchronous) and/or in a particular order, in other implementations the operations may be performed in other orders and in other manners. Any data structures discussed above may also be structured in different manners, such as by having a single data structure split into multiple data structures and/or by having multiple data structures consolidated into a single data structure. Similarly, in some implementations illustrated data structures may store more or less information than is described, such as when other illustrated data structures instead lack or include such information respectively, or when the amount or types of information that is stored is altered.
From the foregoing it will be appreciated that, although specific embodiments have been described herein for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the invention. Accordingly, the invention is not limited except as by corresponding claims and the elements recited by those claims. In addition, while certain aspects of the invention may be presented in certain claim forms at certain times, the inventors contemplate the various aspects of the invention in any available claim form. For example, while only some aspects of the invention may be recited as being embodied in a computer-readable medium at particular times, other aspects may likewise be so embodied.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17069800 | Oct 2020 | US |
| Child | 18099608 | US |
