Patent Application
20220092227
The following disclosure relates generally to techniques for automatically determining attributes of buildings and their floor plans and for automatically identifying building floor plans that have attributes satisfying target criteria and for subsequently using the identified floor plans in one or more automated manners, such as to automatically determine one or more buildings with floor plans having similarities to those of one or more other indicated buildings based at least in part on attributes of rooms of the buildings, and/or to automatically determine one or more buildings with floor plans having attributes based at least in part on adjacency criteria for inter-connected rooms or based at least in part on other specified attributes related to room interiors and views.
In various fields and circumstances, such as architectural analysis, property inspection, real estate acquisition and development, general contracting, improvement cost estimation, etc., it may be desirable to know the interior of a house, office, or other building without having to physically travel to and enter the building. However, it can be difficult to effectively capture, represent and use such building interior information, including to identify buildings that satisfy criteria of interest, and including to display visual information captured within building interiors to users at remote locations (e.g., to enable a user to fully understand the layout and other details of the interior, including to control the display in a user-selected manner). In addition, while a floor plan of a building may provide some information about layout and other details of a building interior, such use of floor plans has some drawbacks, including that floor plans can be difficult to construct and maintain, to accurately scale and populate with information about room interiors, to visualize and otherwise use, etc.
The present disclosure describes techniques for using computing devices to perform automated operations related to identifying building floor plans that have attributes satisfying target criteria and to subsequently using the identified floor plans in one or more further automated manners, and/or related to determining attributes of buildings and their floor plans from acquired information about those buildings. In at least some embodiments, such identification of building floor plans is based at least in part on generating and using adjacency graphs generated for and associated with the floor plans, to represent inter-connections between rooms of the buildings and other attributes of the buildings, and in some cases is further based on generating and using embedding vectors that concisely represent the information of the adjacency graphs—such a floor plan may, in at least some embodiments, be for an as-built multi-room building (e.g., a house, office building, etc.) that is generated from or otherwise associated with panorama images or other images (e.g., rectilinear perspective images) acquired at one or more acquisition locations in an interior of the building (e.g., without having or using information from any depth sensors or other distance-measuring devices about distances from an image's acquisition location to walls or other objects in the surrounding building). In addition, in at least some embodiments, such determining of attributes of buildings and their floor plans may include generating at least a partial initial adjacency graph for a particular building's floor plan, and using that initial adjacency graph (and/or a corresponding embedding vector) to predict additional information regarding rooms of the building, such as to provide the initial adjacency graph (or a corresponding embedding vector) as input to one or more trained machine learning models (e.g., one or more trained neural networks) to predict types of the rooms from various information acquired about the building and/or to predict types of inter-room connections in the building from various information acquired about the building, and with such predicted information used to update the initial adjacency graph and/or the corresponding embedding vector and/or an underlying floor plan. Information in a building floor plan may be used in various manners (e.g., to identify the floor map as matching one or more specified criteria), and information about a building floor plan that is identified as having attributes satisfying target criteria may be further used in various manners in various embodiments, such as for controlling navigation of mobile devices (e.g., autonomous vehicles), for display or other presentation on one or more client devices in corresponding GUIs (graphical user interfaces), etc. Additional details are included below regarding the automated identification and use of building floor plans that have attributes satisfying target criteria, and some or all of the techniques described herein may, in at least some embodiments, be performed via automated operations of a Floor Plan Similarity Determination Manager (“FPSDM”) system, as discussed further below.
As noted above, automated operations of an FPSDM system may include generating and later using an adjacency graph for a floor plan in at least some embodiments, while otherwise generating adjacency information in other formats for the floor plan in other embodiments. Such a floor plan of a building may include a 2D (two-dimensional) representation of various information about the building (e.g., the rooms, doorways between rooms and other inter-room connections, exterior doorways, windows, etc.), and may be further associated with various types of supplemental or otherwise additional information (e.g., data for a plurality of attributes) about the building (including in some situations a 3D, or three-dimensional, model of the building and/or a 2.5D, or two-and-a-half dimensional, model of the building; images captured in rooms of the building, including panoramic images, etc.), as discussed in greater detail below. Such an adjacency graph may store or otherwise include some or all such attribute data for the building, such as with at least some such attribute data stored in or otherwise associated with nodes of the adjacency graph that represent some or all rooms of the floor plan (e.g., with each node containing information about attributes of the room represented by the node), and/or with at least some such attribute data stored in or otherwise associated with edges between nodes that represent connections between adjacent rooms via doors or other inter-room openings, or in some situations further represent adjacent rooms that share at least a portion of at least one wall and optionally a full wall without any direct inter-room opening between those two rooms (e.g., with each edge containing information about connectivity status between the rooms represented by the nodes that the edge inter-connects, such as whether an inter-room opening exists between the two rooms, and/or a type of inter-room opening or other type of adjacency between the two rooms such as without any direct inter-room wall opening). In other embodiments and situations, groups of two or more adjacent rooms may be represented, including in some embodiments and situations to have groups of three or more adjacent rooms rather than pairs of two adjacent rooms. In some embodiments and situations, the adjacency graph may further represent at least some information external to the building, such as exterior areas adjacent to doors or other openings between the building and the exterior and/or other structures on the same property as the building (e.g., a garage, shed, pool house, separate guest quarters, mother-in-law unit, pool, patio, deck, sidewalk, garden, yard, etc.)—such exterior areas and/or other structures may be represented in various manners in the adjacency graph, such as via separate nodes for each such exterior area or other structure, or instead as attribute information associated with corresponding nodes or edges or instead with the adjacency graph as a whole (for the building as a whole). The adjacency graph may further have associated attribute information for the corresponding rooms and inter-room connections in at least some embodiments, such as to represent within the adjacency graph some or all of the information available on a floor plan and otherwise associated with the floor plan (or in some embodiments and situations, information in and associated with a 3D model of the building)—for example, if there are images associated with the floor plan, corresponding visual attributes may be included within the adjacency graph. In embodiments with adjacency information in a form other than an adjacency graph, some or all of the above-indicated types of information may be stored in or otherwise associated with the adjacency information, including information about rooms, about adjacencies between rooms, about connectivity status between adjacent rooms, about attributes of the building, etc. Additional details are included below regarding the generation and use of adjacency graphs, including with respect to the examples of
As is also noted above, automated operations of an FPSDM system may further include generating and later using an embedding vector to concisely represent an adjacency graph for a floor plan of a building in at least some embodiments, such as to summarize the semantic meaning and spatial relationships of the floor plan in a manner that enables reconstruction of the floor plan from the embedding vector. Such an embedding vector may be generated in various manners in various embodiments, such as via the use of representation learning, and in at least some such embodiments may be encoded in a format that is not easily discernible to a human reader. Non-exclusive examples of techniques for generating such embedding vectors are included in the following documents, which are incorporated herein by reference in their entirety: “Symmetric Graph Convolution Autoencoder For Unsupervised Graph Representation Learning” by Jiwoong Park et al., 2019 International Conference On Computer Vision, Aug. 7, 2019; “Inductive Representation Learning On Large Graphs” by William L Hamilton et al., 31st Conference On Neural Information Processing Systems 2017, Jun. 7, 2017; and “Variational Graph Auto-Encoders” by Thomas N. Kipf et al., 30th Conference On Neural Information Processing Systems 2017 (Bayesian Deep Learning Workshop), Nov. 21, 2016.
Additional details are included below regarding the generation and use of embedding vectors, including with respect to the examples of
In addition, as noted above, a floor plan may have various information that is associated with individual rooms and/or with inter-room connections and/or with a corresponding building as a whole, and the corresponding adjacency graph and/or embedding vector for such a floor plan may include some or all such associated information (e.g., represented as attributes of nodes for rooms in an adjacency graph and/or attributes of edges for inter-room connections in an adjacency graph and/or represented as attributes of the adjacency graph as a whole, and with corresponding information encoded in the associated embedding vector). Such associated information may include a variety of types of information, including information about one or more of the following non-exclusive examples: room types, room dimensions, locations of windows and doors and other inter-room openings in a room, room shape, a view type for each exterior window, information about and/or copies of images taken in a room, information about and/or copies of audio or other data captured in a room, information of various types about features of one or more rooms (e.g., as automatically identified from analysis of images, as supplied by operator users of the FPSDM system and/or by end-users viewing information about the floor plan and/or by operator users of ICA and/or MIGM systems as part of capturing information about a building and generating a floor plan for the building, etc.), types of inter-room connections, dimensions of inter-room connections, etc. Furthermore, in at least some embodiments, one or more additional subjective attributes may be determined for and associated with the floor plan, such as via analysis of the floor plan information (e.g., an adjacency graph for the floor plan) by one or more trained classification neural networks (e.g., to identify floor plan characteristics such as an open floor plan; a typical/normal versus atypical/odd/unusual floor plan; a standard versus nonstandard floor plan; a floor plan that is accessibility friendly, such as by being accessible with respect to one or more characteristics such as disability and/or advanced age; etc.)—in at least some such embodiments, the one or more classification neural networks are part of the FPSDM system and are trained via supervised learning using labeled data that identifies floor plans having each of the possible characteristics, while in other embodiments such classification neural networks may instead use unsupervised clustering. Additional details are included below regarding the determination and use of attribute information for floor plans, including with respect to the examples of
After an adjacency graph and optionally an embedding vector is generated for a floor plan of a building, that generated information may be used by the FPSDM system as specified criteria to automatically determine one or more other similar floor plans of other buildings in various manners in various embodiments.
For example, in some embodiments, an initial floor plan is identified, and the corresponding embedding vector for the initial floor plan is generated and compared to generated embedding vectors for other candidate floor plans in order to determine a difference between the initial floor plan's embedding vector and the embedding vectors of some or all of the candidate floor plans, with smaller differences corresponding to higher degrees of similarity. Differences between two such embedding vectors may be determined in various manners in various embodiments, including, as non-exclusive examples, by using one or more of the following distance metrics: Euclidean distance, cosine distance, graph edit distance, a custom distance measure specified by a user, etc.; and/or otherwise determining similarity without use of such a distance metrics. In at least some embodiments, multiple such initial floor plans may be identified and used in the described manner to determine a combined distance between a group of embedding vectors for the multiple initial floor plans and the embedding vectors for each of multiple other candidate floor plans, such as by determining individual distances for each of the initial floor plans to a given other candidate floor plan and by combining the multiple individual determined distances in one or more manners (e.g., a mean or other average, a cumulative total, etc.) to generate the combined distance for the group of embedding vectors of the multiple initial floor plans to that given other candidate floor plan. Additional details are included below regarding comparing embedding vectors for floor plans to determine similarities of the floor plans, including with respect to the examples of
In addition, in some embodiments, an initial floor plan is identified, and corresponding adjacency information (optionally in the form of an adjacency graph, but alternatively able to be stored in a database or other data structure or other format) for the initial floor plan is generated and compared to generated and/or provided adjacency information (e.g., in the form of an adjacency graph, or otherwise in the same format as the adjacency information for the initial floor plan) for multiple other candidate floor plans in order to determine differences (e.g., using distance measures or other measures of similarities) between the initial floor plan's adjacency information and the adjacency information of some or all of the candidate floor plans. Such a determination of the similarity between two such groups of adjacency information may be performed, as non-exclusive examples, in at least some such embodiments by supplying two adjacency graphs to a trained similarity neural network and receiving an indication of a degree of similarity as an output (e.g., a binary yes or no, a value from a range or enumerated list of similarity degrees, etc.), by directly comparing two adjacency graphs (e.g., using graph edit distance and/or isomorphism, belief propagation, the eigenvalue method and/or other feature extraction, iterative methods, subgraph matching using indexing, approximate constrained subgraph matching, mining coherent dense subgraphs, tensor analysis, graph scope, subgraph matching via convex relaxation, etc.), by directly comparing two groups of adjacency information in a format other than adjacency graphs, etc.—in at least some such embodiments, the one or more such similarity neural networks are part of the FPSDM system and are trained using unlabeled or labeled data to identify similar floor plans, such as via supervised learning using labeled data or instead via unsupervised clustering using unlabeled data. In at least some embodiments, multiple such initial floor plans may be identified and used in the described manner to determine a combined degree of similarity between two or more groups of adjacency information for the multiple initial floor plans and additional groups of adjacency information for each of multiple other candidate floor plans, such as by determining individual similarity degrees for each of the initial floor plans to a given other candidate floor plan and by combining the multiple individual determined similarity degrees in one or more manners (e.g., a mean or other average, a cumulative total, etc.) to generate the combined similarity degree for the group of adjacency graphs of the multiple initial floor plans to that given other candidate floor plan. Additional details are included below regarding comparing adjacency graphs or other adjacency information generated for floor plans to determine similarities of the floor plans and/or of the associated buildings more generally, including with respect to the examples of
Furthermore, in some embodiments, one or more explicitly specified criteria other than one or more initial floor plans are received (whether in addition to or instead of receiving one or more initial floor plans), and the corresponding adjacency graph and/or embedding vector for each of multiple candidate floor plans are compared to the specified criteria in order to determine which of the candidate floor plans satisfy the specified criteria (e.g., are a match above a defined similarity threshold). The specified criteria may be of various types in various embodiments and situations, such as one or more of the following non-exclusive examples: search terms corresponding to specific attributes of rooms and/or inter-room connections and/or buildings as a whole (whether objective attributes that can be independently verified and/or replicated, and/or subjective attributes that are determined via use of corresponding classification neural networks); information identifying adjacency information between two or more rooms or other areas; information about views available from windows or other exterior openings of the building; information about directions of windows or other structural elements of the building (e.g., such as to determine natural lighting information available via those windows or other structural elements, optionally at specified days and/or seasons and/or times); etc. Non-exclusive illustrative examples of such specified criteria include the following: a bathroom adjacent to bedroom (i.e., without an intervening hall or other room); a deck adjacent to a family room (optionally with a specified type of connection between them, such as French doors); 2 bedrooms facing south; a kitchen with a tile-covered island and a northward-facing view; a master bedroom with a view of the ocean or more generally of water; any combination of such specified criteria; etc. Additional details are included below regarding the specification and use of criteria to identify matching or otherwise similar floor plans, including via use of adjacency graphs and/or embedding vectors for the floor plans, including with respect to the examples of
As noted above, automated operations of an FPSDM system may, in at least some embodiments, further include using one or more target floor plans that are identified as being sufficiently similar (e.g., above a defined similarity threshold) to specified criteria (e.g., one or more indicated initial floor plans for comparison, specified attributes or other adjacency information, etc.) in one or more further automated manners.
For example, in some embodiments, multiple target floor plans are identified that are similar to specified criteria, and are used for further automated analysis to determine characteristics of that group of target floor plans. Non-exclusive illustrative examples of such determined characteristics for the group of target floor plans may include one or more of the following: common or shared attributes of the target floor plans that are different from the specified criteria (e.g., that are shared by all or a specified minimum amount of the target floor plans); aggregated characteristics of the target floor plans (e.g., from all or a specified minimum amount of the target floor plans), such as an average or cumulative total assessed value of the buildings represented by the multiple target floor plans; etc. Information from the further automated analysis may then be used in further automated operations (e.g., by the FPSDM system, by another system, etc.) in various manners in various embodiments. Additional details are included below regarding the use of multiple identified target floor plans for further automated analysis, including with respect to the examples of
In addition, in some embodiments, one or more target floor plans are identified that are similar to specified criteria associated with a particular end-user (e.g., based on one or more initial target floor plans that are selected by the end-user and/or are identified as previously being of interest to the end-user, based on one or more search criteria specified by the end-user, etc.), and are used in further automated activities to personalize interactions with the end-user. Such further automated personalized interactions may be of various types in various embodiments. Additional details are included below regarding the use of one or more identified target floor plans for further automated end-user personalization, including with respect to the examples of
Furthermore, in some embodiments, one or more target floor plans are identified that are similar to specific criteria associated with a particular end-user (e.g., based on one or more initial target floor plans that are selected by the end-user and/or are identified as previously being of interest to the end-user, based on one or more search criteria specified by the end-user, etc.), and are used in further automated activities to display or otherwise present information to the end-user about the target floor plan(s) and/or additional information associated with those floor plans. Such further automated presentation activities may be of various types in various embodiments. Additional details are included below regarding the use of one or more identified target floor plans for further end-user presentation, including with respect to the examples of
The described techniques related to automated identification of building floor plans that have attributes satisfying target criteria may further include additional operations in some embodiments. For example, in at least some embodiments, machine learning techniques may be used to learn the attributes and/or other characteristics of adjacency graphs to encode in corresponding embedding vectors that are generated, such as the attributes and/or other characteristics that best enable subsequent automated identification of building floor plans having attributes satisfying target criteria (e.g., number of bedrooms; number of bathrooms; connectivity between rooms; size and/or dimensions of each room; number of windows/doors in each room; types of views available from exterior windows, such as water, mountain, a back yard or other exterior area of the property, etc.; location of windows/doors in each room; etc.). Additional details are included below regarding various automated operations that may be performed by the FPSDM system in at least some embodiments.
In at least some embodiments and situations, some or all of the images acquired for a building and associated with the building's floor plan may be panorama images that are each acquired at one of multiple acquisition locations in or around the building, such as to generate a panorama image at each such acquisition location from one or more of a video at that acquisition location (e.g., a 360° video taken from a smartphone or other mobile device held by a user turning at that acquisition location), or multiple images acquired in multiple directions from the acquisition location (e.g., from a smartphone or other mobile device held by a user turning at that acquisition location), or a simultaneous capture of all the image information (e.g., using one or more fisheye lenses), etc. It will be appreciated that such a panorama image may in some situations be represented in a spherical coordinate system and provide up to 360° coverage around horizontal and/or vertical axes, such that a user viewing a starting panorama image may move the viewing direction within the starting panorama image to different orientations to cause different images (or “views”) to be rendered within the starting panorama image (including, if the panorama image is represented in a spherical coordinate system, to convert the image being rendered into a planar coordinate system). Furthermore, acquisition metadata regarding the capture of such panorama images may be obtained and used in various manners, such as data acquired from IMU (inertial measurement unit) sensors or other sensors of a mobile device as it is carried by a user or otherwise moved between acquisition locations. Additional details are included below related to the acquisition and usage of panorama images or other images for a building.
As noted above, automated operations of an FPSDM system may, in at least some embodiments, further include analyzing acquired information about a building to determine further attributes of the building and its rooms. For example, as noted above, images acquired in and around the building may be analyzed to determine various types of features of one or more rooms, including with respect to doors and other non-door openings between rooms. In addition, in at least some embodiments, such determining of attributes of buildings and their floor plans may include generating at least a partial floor plan for a particular building and/or at least a partial initial adjacency graph for the building, and using that initial adjacency graph and/or a corresponding embedding vector to predict additional information regarding rooms of the building. For example, such determining of attributes of buildings and their floor plans may include providing an initial (optionally partial) adjacency graph and/or a corresponding embedding vector as input to one or more trained machine learning models (e.g., one or more trained neural networks) that predict types of the rooms from various information acquired about the building, such as based on information about room shape, size, other adjacent rooms or otherwise a position within the building, features identified (e.g., from image analysis) within the room, etc. In addition, such determining of attributes of buildings and their floor plans may include providing an initial (optionally partial) adjacency graph and/or a corresponding embedding vector as input to one or more trained machine learning models (e.g., one or more trained neural networks), whether the same or different machine learning models used for room type prediction, that predict types of inter-room connections or other adjacencies in the building from various information acquired about the building, such as based on types of rooms, other attributes of one or more rooms, position within the building, features identified (e.g., from image analysis) about the inter-room connection or other adjacency, etc. Such predicted room type information and/or inter-room connection/adjacency type information may be used in various manners in various embodiments, including to update the initial adjacency graph data structure and/or the corresponding embedding vector data structure and/or an underlying floor plan data structure, and with the updated data structure(s) and their information able to be used in various manners (e.g., to identify a building and/or its floor map as matching one or more specified criteria, for display or other presentation to one or more end users, etc. Additional details are included below regarding analyzing acquired information about a building to determine further attributes of the building and its rooms, including predicting room types and/or inter-room connection/adjacency types, such as with respect to the examples of
The described techniques provide various benefits in various embodiments, including to allow floor plans of multi-room buildings and other structures to be identified and used more efficiently and rapidly and in manners not previously available, including to identify building floor plans that match specified criteria to be automatically identified (e.g., based on one or more of similarity to one or more other floor plans; of adjacency information about which rooms are inter-connected and related inter-room relationship information; of information about room interiors and views that are determined at least in part on supplemental information, such as from analysis of one or more images captured at the building; of subjective attributes regarding a floor plan's characteristics, etc.). Such automated techniques allow such identification of matching floor plans to be determined more quickly than previously existing techniques, and in at least some embodiments with greater accuracy, including by using information acquired from the actual building environment (rather than from plans on how the building should theoretically be constructed), as well as enabling the capture of changes to structural elements and/or visual appearance elements that occur after a building is initially constructed. Such described techniques further provide benefits in allowing improved automated navigation of a building by mobile devices (e.g., semi-autonomous or fully-autonomous vehicles), based at least in part on the identification of building floor plans that match specified criteria, including to significantly reduce computing power and time used to attempt to otherwise learn a building's layout. In addition, in some embodiments the described techniques may be used to provide an improved GUI in which a user may more accurately and quickly identify one or more building floor plans matching specified criteria, and obtain information about one or more such buildings (e.g., for use in navigating an interior of the one or more buildings), including in response to search requests, as part of providing personalized information to the user, as part of providing value estimates and/or other information about a building to a user (e.g., after analysis of information about one or more target building floor plans that are similar to one or more initial floor plans or that otherwise match specified criteria), etc. Various other benefits are also provided by the described techniques, some of which are further described elsewhere herein.
As noted above, automated operations of an FPSDM system may include identifying building floor plans that have attributes satisfying target criteria and subsequently using the identified floor plans in one or more further automated manners. In at least some embodiments, such an FPSDM system may operate in conjunction with one or more separate ICA (Image Capture and Analysis) systems and/or with one or more separate MIGM (Mapping Information and Generation Manager) systems, such as to obtain and use floor plan and other associated information for buildings from the ICA and/or MIGM systems, while in other embodiments such an FPSDM system may incorporate some or all functionality of such ICA and/or MIGM systems as part of the FPSDM system. In yet other embodiments, the FPSDM system may operate without using some or all functionality of the ICA and/or MIGM systems, such as if the FPSDM system obtains information about building floor plans and associated information from other sources (e.g., from manual creation or provision of such building floor plans and/or associated information by one or more users).
With respect to functionality of such an ICA system, it may perform automated operations in at least some embodiments to acquire images (e.g., panorama images) at various acquisition locations associated with a building (e.g., in the interior of multiple rooms of the building), and optionally further acquire metadata related to the image acquisition process and/or to movement of a capture device between acquisition locations—in at least some embodiments, such acquisition and subsequent use of acquired information may occur without having or using information from depth sensors or other distance-measuring devices about distances from images' acquisition locations to walls or other objects in a surrounding building or other structure. For example, in at least some such embodiments, such techniques may include using one or more mobile devices (e.g., a camera having one or more fisheye lenses and mounted on a rotatable tripod or otherwise having an automated rotation mechanism; a camera having one or more fisheye lenses sufficient to capture 360 degrees horizontally without rotation; a smart phone held and moved by a user, such as to rotate the user's body and held smart phone in a 360° circle around a vertical axis; a camera held by or mounted on a user or the user's clothing; a camera mounted on an aerial and/or ground-based drone or other robotic device; etc.) to capture visual data from a sequence of multiple acquisition locations within multiple rooms of a house (or other building). Additional details are included elsewhere herein regarding operations of device(s) implementing an ICA system, such as to perform such automated operations, and in some cases to further interact with one or more ICA system operator user(s) in one or more manners to provide further functionality.
With respect to functionality of such an MIGM system, it may perform automated operations in at least some embodiments to analyze multiple 360° panorama images (and optionally other images) that have been acquired for a building interior (and optionally an exterior of the building), and determine room shapes and locations of passages connecting rooms for some or all of those panorama images, as well as to determine wall elements and other elements of some or all rooms of the building in at least some embodiments and situations. The types of connecting passages between two or more rooms may include one or more of doorway openings and other inter-room non-doorway wall openings, windows, stairways, non-room hallways, etc., and the automated analysis of the images may identify such elements based at least in part on identifying the outlines of the passages, identifying different content within the passages than outside them (e.g., different colors or shading), etc. The automated operations may further include using the determined information to generate a floor plan for the building and to optionally generate other mapping information for the building, such as by using the inter-room passage information and other information to determine relative positions of the associated room shapes to each other, and to optionally add distance scaling information and/or various other types of information to the generated floor plan. In addition, the MIGM system may in at least some embodiments perform further automated operations to determine and associate additional information with a building floor plan and/or specific rooms or locations within the floor plan, such as to analyze images and/or other environmental information (e.g., audio) captured within the building interior to determine particular attributes (e.g., a color and/or material type and/or other characteristics of particular elements, such as a floor, wall, ceiling, countertop, furniture, fixtures, appliances, etc.; the presence and/or absence of particular elements, such as an island in the kitchen; etc.), or to otherwise determine relevant attributes (e.g., directions that building elements face, such as windows;
views from particular windows or other locations; etc.). Additional details are included below regarding operations of computing device(s) implementing an MIGM system, such as to perform such automated operations and in some cases to further interact with one or more MIGM system operator user(s) in one or more manners to provide further functionality.
For illustrative purposes, some embodiments are described below in which specific types of information are acquired, used and/or presented in specific ways for specific types of structures and by using specific types of devices—however, it will be understood that the described techniques may be used in other manners in other embodiments, and that the invention is thus not limited to the exemplary details provided. As one non-exclusive example, while specific types of data structures (e.g., floor plans, adjacency graphs, embedding vectors, etc.) are generated and used in specific manners in some embodiments, it will be appreciated that other types of information to describe floor plans and other associated information may be similarly generated and used in other embodiments, including for buildings (or other structures or layouts) separate from houses, and that floor plans identified as matching specified criteria may be used in other manners in other embodiments. In addition, the term “building” refers herein to any partially or fully enclosed structure, typically but not necessarily encompassing one or more rooms that visually or otherwise divide the interior space of the structure—non-limiting examples of such buildings include houses, apartment buildings or individual apartments therein, condominiums, office buildings, commercial buildings or other wholesale and retail structures (e.g., shopping malls, department stores, warehouses, etc.), supplemental structures on a property with another main building (e.g., a detached garage or shed on a property with a house), etc. The term “acquire” or “capture” as used herein with reference to a building interior, acquisition location, or other location (unless context clearly indicates otherwise) may refer to any recording, storage, or logging of media, sensor data, and/or other information related to spatial characteristics and/or visual characteristics and/or otherwise perceivable characteristics of the building interior or subsets thereof, such as by a recording device or by another device that receives information from the recording device. As used herein, the term “panorama image” may refer to a visual representation that is based on, includes or is separable into multiple discrete component images originating from a substantially similar physical location in different directions and that depicts a larger field of view than any of the discrete component images depict individually, including images with a sufficiently wide-angle view from a physical location to include angles beyond that perceivable from a person's gaze in a single direction. The term “sequence” of acquisition locations, as used herein, refers generally to two or more acquisition locations that are each visited at least once in a corresponding order, whether or not other non-acquisition locations are visited between them, and whether or not the visits to the acquisition locations occur during a single continuous period of time or at multiple different times, or by a single user and/or device or by multiple different users and/or devices. In addition, various details are provided in the drawings and text for exemplary purposes, but are not intended to limit the scope of the invention. For example, sizes and relative positions of elements in the drawings are not necessarily drawn to scale, with some details omitted and/or provided with greater prominence (e.g., via size and positioning) to enhance legibility and/or clarity. Furthermore, identical reference numbers may be used in the drawings to identify the same or similar elements or acts.
One or more users (not shown) of one or more client computing devices 175 may further interact over one or more computer networks 170 with an FPSDM (Floor Plan Similarity Determination Manager) system 140 (and optionally the ICA system 160 and/or MIGM system 160), such as to assist in identifying building floor plans having attributes satisfying target criteria and in subsequently using the identified floor plans in one or more further automated manners—such interactions by the user(s) may include, for example, specifying target criteria to use in searching for corresponding floor plans or otherwise providing information about target criteria of interest to the users, or obtaining and optionally interacting with one or more particular identified floor plans and/or with additional associated information (e.g., to change between a floor plan view and a view of a particular image at an acquisition location within or near the floor plan; to change the horizontal and/or vertical viewing direction from which a corresponding view of a panorama image is displayed, such as to determine a portion of a panorama image to which a current user viewing direction is directed, etc.). In addition, a floor plan (or portion of it) may be linked to or otherwise associated with one or more other types of information, including for a floor plan of a multi-story or otherwise multi-level building to have multiple associated sub-floor plans for different stories or levels that are interlinked (e.g., via connecting stairway passages), for a two-dimensional (“2D”) floor plan of a building to be linked to or otherwise associated with a three-dimensional (“3D”) rendering of the building, etc. Also, while not illustrated in
In some embodiments, the ICA system 160 and/or MIGM system 160 and/or FPSDM system 140 may execute on the same server computing system(s), such as if multiple or all of those systems are operated by a single entity or are otherwise executed in coordination with each other (e.g., with some or all functionality of those systems integrated together into a larger system), while in other embodiments the FPSDM system may instead obtain floor plan information and/or additional associated information from one or more external sources and optionally store them locally with information 142 for further analysis and use by the FPSDM system. In addition, in the depicted computing environment of
In the example of
In operation, a user associated with the mobile device arrives at a first acquisition location 210A within a first room of the building interior (in this example, an entryway from an external door 190-1 to the living room), and captures a view of a portion of the building interior that is visible from that acquisition location 210A (e.g., some or all of the first room, and optionally small portions of one or more other adjacent or nearby rooms, such as through doors, halls, stairs or other connecting passages from the first room) as the mobile device is rotated around a vertical axis at the first acquisition location (e.g., with the user turning his or her body in a circle while holding the mobile device stationary relative to the user's body). The actions of the user and/or the mobile device may be controlled or facilitated via use of one or more programs executing on the mobile device, such as ICA application system 154, optional browser 162, control system 147, etc., and the view capture may be performed by recording a video and/or taking a succession of one or more images, including to capture visual information depicting a number of objects or other elements (e.g., structural details) that may be visible in images (e.g., video frames) captured from the acquisition location. In the example of
After the first acquisition location 210A has been adequately captured (e.g., by a full rotation of the mobile device), the user may proceed to a next acquisition location (such as acquisition location 210B), optionally recording movement data during movement between the acquisition locations, such as video and/or other data from the hardware components (e.g., from one or more IMUs, from the camera, etc.). At the next acquisition location, the user may similarly use the mobile device to capture one or more images from that acquisition location. This process may repeat from some or all rooms of the building and optionally external to the building, as illustrated for acquisition locations 210C-210J. The acquired video and/or other images for each acquisition location are further analyzed to generate a panorama image for each of acquisition locations 210A-210J, including in some embodiments to match objects and other elements in different images. In addition to generating such panorama images, further analysis may be performed in order to ‘link’ at least some of the panoramas together (with some corresponding lines 215 between them being shown for the sake of illustration), such as to determine relative positional information between pairs of acquisition locations that are visible to each other, to store corresponding inter-panorama links (e.g., links 215-AB, 215-BC and 215-AC between acquisition locations A and B, B and C, and A and C, respectively), and in some embodiments and situations to further link at least some acquisition locations that are not visible to each other (e.g., a link 215-BE, not shown, between acquisition locations 210B and 210E).
Additional details related to embodiments of generating and using linking information between panorama images, including using travel path information and/or elements or other features visible in multiple images, are included in co-pending U.S. Non-Provisional patent application Ser. No. 16/693,286, filed Nov. 23, 2019 and entitled “Connecting And Using Building Data Acquired From Mobile Devices” (which includes disclosure of an example BICA system that is generally directed to obtaining and using linking information to inter-connect multiple panorama images captured within one or more buildings or other structures), which is incorporated herein by reference in its entirety.
Various details are provided with respect to
In particular,
Additional details related to embodiments of a system providing at least some such functionality of an MIGM system or related system for generating floor plans and associated information and/or presenting floor plans and associated information are included in co-pending U.S. Non-Provisional patent application Ser. No. 16/190,162, filed Nov. 14, 2018 and entitled “Automated Mapping Information Generation From Inter-Connected Images” (which includes disclosure of an example Floor Map Generation Manager, or FMGM, system that is generally directed to automated operations for generating and displaying a floor map or other floor plan of a building using images acquired in and around the building); in U.S. Non-Provisional patent application Ser. No. 16/681,787, filed Nov. 12, 2019 and entitled “Presenting Integrated Building Information Using Three-Dimensional Building Models” (which includes disclosure of an example FMGM system that is generally directed to automated operations for displaying a floor map or other floor plan of a building and associated information); in U.S. Non-Provisional patent application Ser. No. 16/841,581, filed Apr. 6, 2020 and entitled “Providing Simulated Lighting Information For Three-Dimensional Building Models” (which includes disclosure of an example FMGM system that is generally directed to automated operations for displaying a floor map or other floor plan of a building and associated information); in U.S. Provisional Patent Application No. 62/927,032, filed Oct. 28, 2019 and entitled “Generating Floor Maps For Buildings From Automated Analysis Of Video Of The Buildings' Interiors” (which includes disclosure of an example Video-To-Floor Map, or FPSDM, system that is generally directed to automated operations for generating a floor map or other floor plan of a building using video data acquired in and around the building); in U.S. Non-Provisional patent application Ser. No. 16/807,135, filed Mar. 2, 2020 and entitled “Automated Tools For Generating Mapping Information For Buildings” (which includes disclosure of an example MIGM system that is generally directed to automated operations for generating a floor map or other floor plan of a building using images acquired in and around the building); and in U.S. Non-Provisional patent application Ser. No. 17/013,323, filed Sep. 4, 2020 and entitled “Automated Analysis Of Image Contents To Determine The Acquisition Location Of The Image” (which includes disclosure of an example MIGM system that is generally directed to automated operations for generating a floor map or other floor plan of a building using images acquired in and around the building, and an example ILMM system for determining the acquisition location of an image on a floor plan based at least in part on an analysis of the image's contents); each of which is incorporated herein by reference in its entirety.
The at least one machine learning model 286 then provides output 289 indicating predicted room types of some or all rooms in the building, with that information 289 stored and optionally used to generate an updated version 281b of the adjacency graph, and/or to otherwise update the information about the building (e.g., to update the building's floor plan, embedding vector(s), etc.). In addition, the at least one machine learning model 284 provides output 284o indicating predicted inter-room door connections (and optionally non-door opening connections) for at least some inter-room connections/adjacencies of the building, and the at least one machine learning model 285 provides output 285o indicating predicted inter-room non-connected adjacencies (e.g., one or more walls between two adjacent rooms) for at least some inter-room connections/adjacencies of the building. The information 284o and 285o is then provided in this example embodiment to an FPSDM component 287 that aggregates the different inter-room connection/adjacency predictions from the trained machine learning models 284 and 285, which provides output 288 indicating predicted inter-room connection/adjacency types for some or all rooms in the building, with that information 288 stored and optionally used to generate an updated version 281b of the adjacency graph, and/or to otherwise update the information about the building (e.g., to update the building's floor plan, embedding vector(s), etc.). Additional details are included below regarding additional examples for performing at least some of the automated operations discussed with respect to
In at least some embodiments, the prediction of room type information and inter-room connection/adjacency information includes various automated operations. For example, an adjacency graph may first be constructed for a building's floor plan, in which each node represents a room, and edges represent the connectivity or other adjacency of two rooms. Examples of node features that may be captured and used for the predictions including some or all of the following non-exclusive list: the number of doors, windows, and openings of the room; the room type; the perimeter of the room; the maximum length and width of room; the area of the room; the ratio between the room area and the room's bounding box area; chain code shape descriptors; shape descriptors to represent the centroid distance (e.g., the distance from the shape center to bound point); an order in which one or more images (e.g., panorama images) were captured in the room, relative to that of other rooms; features of the room extracted from analysis of one or more images (e.g., panorama images) captured in the room; center coordinates of the room; etc.
In at least some embodiments, such as the example of
For example, consider an example embodiment in which G=(A,X) is an undirected graph with its adjacency matrix A∈RN×N and node attributes X={x1, x2, . . . xN}, xi ∈RF, where N is the number of nodes in the graph and F is the number of features of each node. The goal of the model is to learn an encoding function H=g(A,X) and three decoding functions Pi=di(H) (i∈{1, 2, 3}), where H is the embedding from the encoder, and Pi is the output of edge prediction. The encoder g transforms the initial features X to new feature representations by aggregating information of the node neighborhoods, and then passes information to the next layers of one or more trained machine learning models (e.g., one or more graph neural networks, or GNNs). Then, the decoders d use the new representations to predict the probability of whether or not an edge exists between two nodes. The encoder may be, for example, a base Graph attention network (GAT), and the decoders may be, for example, fully connected layers to predict the connectivity score between pairs of nodes (i,j). The two different types of edges are predicted in this example by combining the loss function of each task, with the binary cross entropy loss for each task being minimized, and the loss function of the proposed system expressed as follows:
I=I
connectivity+α1Iadjacency+α2Iconnectivity+adjacency (1)
where I denotes the total loss, Iconnectivity is the loss for inter-room connectivity (which can be thought of as a special type of room adjacency in which the adjacent rooms are inter-connected, such as may be visually identified through a doorway or other opening between the rooms), Iadjacency is the loss for adjacency of two rooms that are not inter-connected (such as two rooms that lack any visual information from either of the rooms into the other room), and Iconnectivity+adjacency is the loss for both inter-room connectivity between two rooms and adjacency of two rooms that are not inter-connected. The terms α1 and α2 are the importance weights associated with the Iadjacency and Iconnectivity+adjacency terms, respectively.
In the example embodiment discussed above, the encoder is based on a graph attention network (GAT), which attempts to generate new feature representations that can help predict links. In this example, the input graph is first passed into several graph attention layers to capture structural information by aggregating the attributes of its neighbors. The graph attention layers will produce embeddings for each node Û=g(A,X). Furthermore, the initial feature X may itself contain some rooms properties (e.g., room shape), which also helps to improve the link prediction performance. Hence, a long skip connection is added by concatenating the initial features X and the output of graph attention layers Û. The final output of the encoder would be H=[X,g(A,X)]. In addition, the decoders of the example embodiment output the probability that two nodes are connected by an edge. In the example of
In these example embodiments, one or more machine learning models are trained to predict different types of connections (i.e., connectivity edges, non-connection adjacency edges, and a combination of either connectivity edges or non-connection adjacency edges) given building information (e.g., determined room attributes). As one example, let A be the adjacency matrix of the initial adjacency graph generated from initial edges of a floor plan that are a combination of either connectivity edges or non-connection adjacency edges—a subset of edges are then randomly removed to generate an incomplete adjacency matrix A. The removed edges are denoted Epos as positive edges, and the same amount of negative edges Eneg are then randomly sampled, which means the two rooms are not connected. Each positive edge is composed of a node pair (i,j) such that A(i,j)=1, and each negative edge is composed of a node pair (i,j) such that A(i,j)=0. Positive and negative edges are sampled for all of connectivity edges, non-connection adjacency edges, and a combination of either connectivity edges or non-connection adjacency edges. Given the node attributes X and the incomplete adjacency matrix Ā, the one or more machine learning models are trained on all types of positive and negative edges, and optimized by minimizing the binary cross entropy loss between the prediction and the corresponding ground truth.
Various details have been provided with respect to
As noted above, in some embodiments, the described techniques include using machine learning to learn the attributes and/or other characteristics of adjacency graphs to encode in corresponding embedding vectors that are generated, such as the attributes and/or other characteristics that best enable subsequent automated identification of building floor plans having attributes satisfying target criteria, and with the embedding vectors that are used in at least some embodiments to identify target building floor plans being encoded based on such learned attributes or other characteristics. In particular, in at least some such embodiments, graph representation learning is used to search for a mapping function that can map the nodes in a graph to d-dimensional vectors, such that in the learned space similar nodes in the graph have similar embeddings. Unlike traditional methods such as graph kernel methods (see, for example, “Graph Kernels” by S. V. N. Vishwanathan et al., Journal of Machine Learning Research, 11:1201-1242, 2010; and “A Survey On Graph Kernels”, Nils M. Kriege et al., arXiv:1903.11835, 2019), graph neural networks remove the process of hand-engineered features and directly learn the high-level embeddings from the raw features of nodes or the (sub)graph.
Various techniques exist for extending and re-defining convolutions in the graph domain, which can be categorized into the spectral approach and the spatial approach. The spectral approach employs the spectral representation of a graph, and are specific to a particular graph structure, such that the models trained on one graph are not applicable to a graph with a different structure (see, for example, “Spectral Networks And Locally Connected Networks On Graphs”, Joan Bruna et al., International Conference on Learning Representations 2014, 2014; “Convolutional Neural Networks On Graphs With Fast Localized Spectral Filtering”, Michael Defferrard et al., Proceedings of Neural Information Processing Systems 2016, 2016, pp. 3844-3852; and “Semi-Supervised Classification With Graph Convolutional Networks”, Thomas N. Kipf et al., International Conference on Learning Representations 2017, 2017). The convolution operation for the spectral approach is defined in the Fourier domain by computing the eigendecomposition of the graph Laplacian, and the filter may be approximated to reduce the expensive eigen-decomposition by Chebyshev expansion of graph Lapacian, generating local filters, with the filters optionally limited to work on neighbors one step away from the current node.
With respect to the spatial approach, it includes learning embeddings for a node by recursively aggregating information from its local neighbors. Various amounts of neighboring nodes and corresponding aggregation functions can be handled in various ways. For example, a fixed number of neighbors for each node may be sample, and different aggregation functions such as mean, max and long short term memory networks (LSTM) may be used (see, for example, “Inductive Representation Learning On Large Graphs”, Will Hamilton et al., Proceedings of Neural Information Processing Systems 2017, 2017, pp. 1024-1034). Alternatively, each neighboring node may be considered to contribute differently to a central node, with the contribution factors being learnable via self-attention models (see, for example, “Graph Attention Networks”, P. Velickovic et al., International Conference on Learning Representations 2018, 2018). Furthermore, each attention head captures feature correlation in a different representation subspace, and may be treated differently, such as by using a convolutional sub-network to weight the importance of each attention head (see, for example, “GaAN: Gated Attention Networks For Learning On Large And Spatiotemporal Graphs”, Jiani Zhang et al., Proceedings of Uncertainty in Artificial Intelligence 2018, 2018).
In addition, in some embodiments, the creation of an adjacency graph and/or associated embedding vector for a building may be further based in part on partial information that is provided for the building (e.g., by an operator user of the FPSDM system, by one or more end users, etc.). Such partial information may include, for example, one or more of the following: some or all room names for rooms of the building being provided, with the connections between the rooms to be automatically determined or otherwise established; some or all inter-room connections between rooms of the building being provided, with likely room names for the rooms to be automatically determined or otherwise established; some room names and inter-room connections being provided, with the other inter-room connections and/or likely room names to be automatically determined or otherwise established. In such embodiments, the automated techniques may include using the partial information as part of completing or otherwise generating a floor plan for the building, with the floor plan subsequently used for creating a corresponding adjacency graph and/or embedding vector.
The server computing system(s) 300 and executing FPSDM system 340, server computing system(s) 380 and executing ICA and MIGM systems 388-389, and executing Building Map Viewer system 345, may communicate with each other and with other computing systems and devices in this illustrated embodiment, such as via one or more networks 399 (e.g., the Internet, one or more cellular telephone networks, etc.), including to interact with user client computing devices 390 (e.g., used to view floor maps, and optionally associated images and/or other related information, such as by executing a copy of the Building Map Viewer system, not shown), and/or mobile video acquisition devices 360 (e.g., used to acquire video and optionally additional images and/or other information for buildings or other environments to be modeled), and/or optionally other navigable devices 395 that receive and use floor maps and optionally other generated information for navigation purposes (e.g., for use by semi-autonomous or fully autonomous vehicles or other devices). In other embodiments, some of the described functionality may be combined in less computing systems, such as to combine the FPSDM system 340 and the Building Map Viewer System 345 in a single system or device, to combine the FPSDM system 340 and the video acquisition functionality of device(s) 360 in a single system or device, to combine the ICA and MIGM systems 388-389 and the video acquisition functionality of device(s) 360 in a single system or device, to combine the FPSDM system 340 and the ICA and MIGM systems 388-389 in a single system or device, to combine the FPSDM system 340 and the ICA and MIGM systems 388-389 and the video acquisition functionality of device(s) 360 in a single system or device, etc.
In the illustrated embodiment, an embodiment of the FPSDM system 340 executes in memory 330 of the server computing system(s) 300 in order to perform at least some of the described techniques, such as by using the processor(s) 305 to execute software instructions of the system 340 in a manner that configures the processor(s) 305 and computing system 300 to perform automated operations that implement those described techniques. The illustrated embodiment of the FPSDM system may include one or more components, not shown, to each perform portions of the functionality of the FPSDM system, and the memory may further optionally execute one or more other programs 335—as one specific example, a copy of the ICA and/or MIGM systems may execute as one of the other programs 335 in at least some embodiments, such as instead of or in addition to the ICA and/or MIGM systems 388-389 on the server computing system(s) 380. The FPSDM system 340 may further, during its operation, store and/or retrieve various types of data on storage 320 (e.g., in one or more databases or other data structures), such as various types of user information 322, floor plans and other associated information 324 (e.g., generated and saved 2.5D and/or 3D models, building and room dimensions for use with associated floor plans, additional images and/or annotation information, etc.), generated floor plan adjacency graphs and/or associated embedding vectors 326, and/or various types of optional additional information 329 (e.g., various analytical information related to presentation or other use of one or more building interiors or other environments).
In addition, an embodiment of the ICA and MIGM systems 388-389 execute in memory 387 of the server computing system(s) 380 in the illustrated embodiment in order to perform techniques related to generating linked panorama images and floor plans for buildings, such as by using the processor(s) 381 to execute software instructions of the systems 388 and/or 389 in a manner that configures the processor(s) 381 and computing system(s) 380 to perform automated operations that implement those techniques. The illustrated embodiment of the ICA and MIGM systems may include one or more components, not shown, to each perform portions of the functionality of the ICA and MIGM systems, respectively, and the memory may further optionally execute one or more other programs 383. The ICA and/or MIGM systems 388-389 may further, during operation, store and/or retrieve various types of data on storage 384 (e.g., in one or more databases or other data structures), such as video and/or image information 386 acquired for one or more buildings (e.g., 360° video or images for analysis to generate floor maps, to provide to users of client computing devices 390 for display, etc.), floor plans and/or other generated mapping information 387, other information 385 (e.g., additional images and/or annotation information for use with associated floor plans, building and room dimensions for use with associated floor plans, various analytical information related to presentation or other use of one or more building interiors or other environments, etc.)—while not illustrated in
Some or all of the user client computing devices 390 (e.g., mobile devices), mobile image acquisition devices 360, optional other navigable devices 395 and other computing systems (not shown) may similarly include some or all of the same types of components illustrated for server computing system 300. As one non-limiting example, the mobile image acquisition devices 360 are each shown to include one or more hardware CPU(s) 361, I/O components 362, storage 364, and memory 367, one or more imaging systems 365 and IMU hardware sensors 369 (e.g., for use in acquisition of video and/or images, associated device movement data, etc.). In the illustrated example, one or both of a browser and one or more client applications 368 (e.g., an application specific to the FPSDM system and/or to ICA system and/or to the MIGM system) are executing within memory 367, such as to participate in communication with the FPSDM system 340, ICA system 388, MIGM system 389 and/or other computing systems. While particular components are not illustrated for the other navigable devices 395 or other computing devices/systems 390, it will be appreciated that they may include similar and/or additional components.
It will also be appreciated that computing systems 300 and 380 and the other systems and devices included within
It will also be appreciated that, while various items are illustrated as being stored in memory or on storage while being used, these items or portions of them may be transferred between memory and other storage devices for purposes of memory management and data integrity. Alternatively, in other embodiments some or all of the software components and/or systems may execute in memory on another device and communicate with the illustrated computing systems via inter-computer communication. Thus, in some embodiments, some or all of the described techniques may be performed by hardware means that include one or more processors and/or memory and/or storage when configured by one or more software programs (e.g., by the FPSDM system 340 executing on server computing systems 300, by the Building Map Viewer System 345 executing on server computing systems 300 or other computing systems/devices, etc.) and/or data structures, such as by execution of software instructions of the one or more software programs and/or by storage of such software instructions and/or data structures, and such as to perform algorithms as described in the flow charts and other disclosure herein. Furthermore, in some embodiments, some or all of the systems and/or components may be implemented or provided in other manners, such as by consisting of one or more means that are implemented partially or fully in firmware and/or hardware (e.g., rather than as a means implemented in whole or in part by software instructions that configure a particular CPU or other processor), including, but not limited to, one or more application-specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and/or embedded controllers), field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the components, systems and data structures may also be stored (e.g., as software instructions or structured data) on a non-transitory computer-readable storage mediums, such as a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM or flash RAM), a network storage device, or a portable media article (e.g., a DVD disk, a CD disk, an optical disk, a flash memory device, etc.) to be read by an appropriate drive or via an appropriate connection. The systems, components and data structures may also in some embodiments be transmitted via generated data signals (e.g., as part of a carrier wave or other analog or digital propagated signal) on a variety of computer-readable transmission mediums, including wireless-based and wired/cable-based mediums, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). Such computer program products may also take other forms in other embodiments. Accordingly, embodiments of the present disclosure may be practiced with other computer system configurations.
The illustrated embodiment of the routine begins at block 405, where instructions or information are received. At block 410, the routine determines whether the received instructions or information indicate to acquire data representing a building interior, and if not continues to block 490. Otherwise, the routine proceeds to block 412 to receive an indication (e.g., from a user of a mobile image acquisition device) to begin the image acquisition process at a first acquisition location. After block 412, the routine proceeds to block 415 in order to perform acquisition location image acquisition activities in order to acquire a 360° panorama image for the acquisition location in the interior of the target building of interest, such as to provide horizontal coverage of at least 360° around a vertical axis. The routine may also optionally obtain annotation and/or other information from a user regarding the acquisition location and/or the surrounding environment, such as for later use in presentation of information regarding that acquisition location and/or surrounding environment.
After block 415 is completed, the routine continues to block 420 to determine if there are more acquisition locations at which to acquire images, such as based on corresponding information provided by the user of the mobile device. If so, the routine continues to block 422 to optionally initiate the capture of linking information (such as acceleration data) during movement of the mobile device along a travel path away from the current acquisition location and towards a next acquisition location within the building interior. As described elsewhere herein, the captured linking information may include additional sensor data (e.g., from one or more IMU, or inertial measurement units, on the mobile device or otherwise carried by the user) and/or additional video information, recorded during such movement. Initiating the capture of such linking information may be performed in response to an explicit indication from a user of the mobile device or based on one or more automated analyses of information recorded from the mobile device. In addition, the routine may further optionally monitor the motion of the mobile device in some embodiments during movement to the next acquisition location, and provide one or more guidance cues to the user regarding the motion of the mobile device, quality of the sensor data and/or video information being captured, associated lighting/environmental conditions, advisability of capturing a next acquisition location, and any other suitable aspects of capturing the linking information. Similarly, the routine may optionally obtain annotation and/or other information from the user regarding the travel path, such as for later use in presentation of information regarding that travel path or a resulting inter-panorama connection link. In block 424, the routine determines that the mobile device has arrived at the next acquisition location (e.g., based on an indication from the user, based on the forward movement of the user stopping for at least a predefined amount of time, etc.), for use as the new current acquisition location, and returns to block 415 in order to perform the acquisition location image acquisition activities for the new current acquisition location.
If it is instead determined in block 420 that there are not any more acquisition locations at which to acquire image information for the current building or other structure, the routine proceeds to block 425 to optionally analyze the acquisition location information for the building or other structure, such as to identify possible additional coverage (and/or other information) to acquire within the building interior. For example, the ICA system may provide one or more notifications to the user regarding the information acquired during capture of the multiple acquisition locations and optionally corresponding linking information, such as if it determines that one or more part of the recorded information are of insufficient or undesirable quality, or do not appear to provide complete coverage of the building (e.g., for one or more further acquisition locations that at which image acquisition has not occurred), and if so the routine may return to block 422 to initiate additional corresponding data acquisition activities. After block 425, if the routine does not return to block 422, the routine continues instead to block 435 to optionally preprocess the acquired 360° panorama images before their subsequent use for generating related mapping information. In block 477, the images and any associated generated or obtained information is stored for later use.
If it is instead determined in block 410 that the instructions or other information recited in block 405 are not to acquire images and other data representing a building interior, the routine continues instead to block 490 to perform any other indicated operations as appropriate, such as any housekeeping tasks, to configure parameters to be used in various operations of the system (e.g., based at least in part on information specified by a user of the system, such as a user of a mobile device who captures one or more building interiors, an operator user of the ICA system, etc.), to obtain and store other information about users of the system, to respond to requests for generated and stored information, etc.
Following blocks 477 or 490, the routine proceeds to block 495 to determine whether to continue, such as until an explicit indication to terminate is received, or instead only if an explicit indication to continue is received. If it is determined to continue, the routine returns to block 405 to await additional instructions or information, and if not proceeds to step 499 and ends.
The illustrated embodiment of the routine begins at block 505, where information or instructions are received. The routine continues to block 510 to determine whether the instructions received in block 505 indicate to generate mapping information for an indicated building, and if so the routine continues to perform blocks 515-588 to do so, and otherwise continues to block 590.
In block 515, the routine determines whether image information is already available for the building, or if such information instead needs to be acquired. If it is determined in block 515 that the information needs to be acquired, the routine continues to block 520 to acquire such information, optionally waiting for one or more users or devices to move throughout the building and acquire panoramas or other images at multiple acquisition locations in multiple rooms of the building, and to optionally further analyze the images and/or metadata information about their acquisition to interconnect the images, as discussed in greater detail elsewhere herein—
After blocks 520 or 530, the routine continues to block 535 to optionally obtain additional information about the building (whether based on activities performed during initial image acquisition and/or afterwards), such as based on acquired annotation information and/or information from one or more external sources (e.g., online databases, information provided by one or more end-users, etc.) and/or information from analysis of acquired images (e.g., initial panorama images and/or additional images, such as for additional images at locations different from acquisition locations of the initial panorama images)—such additional obtained information may include, for example, exterior dimensions and/or shape of the building, information about built-in features (e.g., a kitchen island), information about installed fixtures and/or appliances (e.g., kitchen appliances, bathroom items, etc.); information about visual appearance information for building interior locations (e.g., color and/or material type and/or texture for installed items such as floor coverings or wall coverings or surface coverings), information about views from particular windows or other building locations, other information about areas external to the building (e.g., other associated buildings or structures, such as sheds, garages, pools, decks, patios, walkways, gardens, etc.; a type of an external space; items present in an external space; etc.).
After block 535, the routine continues to block 550 to determine, for each room inside the building with one or more acquisition locations and associated acquired images, a room shape of the room from data in the image(s) taken inside the room, and optionally a position within the room of its acquisition location(s), such as in an automated manner. In block 555, the routine further uses visual data in the images and/or the acquisition metadata for them to determine, for each room in the building, any connecting passages in or out of the room (e.g., in an automated manner). In block 560, the routine further uses visual data in the images and/or the acquisition metadata for them to determine, for each room in the building, any wall elements in the room and their positions (e.g., in an automated manner), such as for windows, inter-wall borders, etc. It will be appreciated that, while blocks 550-560 are illustrated as separate operations in this example, in some embodiments a single analysis of the images may be performed to acquire or determine multiple types of information, such as those discussed with respect to blocks 550-560.
In block 565, the routine then determines estimated positions of the room shapes to create an initial 2D floor plan, such as by connecting inter-room passages in their respective rooms, by optionally positioning room shapes around determined acquisition location positions (e.g., if the acquisition location positions are inter-connected), and by optionally applying one or more constraints or optimizations. Such a floor plan may include, for example, relative position and shape information for the various rooms without providing any actual dimension information for the individual rooms or building as a whole, and may further include multiple linked or associated sub-maps (e.g., to reflect different stories, levels, sections, etc.) of the building. The routine further associates positions of the doors, wall openings and other identified wall elements on the floor plan.
After block 565, the routine optionally performs one or more steps 575-580 to determine and associate additional information with the floor plan. In block 575, the routine optionally estimates the dimensions of some or all of the rooms, such as from analysis of images and/or their acquisition metadata or from overall dimension information obtained for the exterior of the building, and associates the estimated dimensions with the floor plan—it will be appreciated that if sufficiently detailed dimension information were available, architectural drawings, blue prints, etc. may be generated from the floor plan. After block 575, the routine continues to block 580 to optionally associate further information with the floor plan (e.g., with particular rooms or other locations within the building), such as additional images and/or annotation information. In block 585, the routine further estimates heights of walls in some or all rooms, such as from analysis of images and optionally sizes of known objects in the images, as well as height information about a camera when the images were acquired, and further uses such information to generate a 3D computer model of the building, with the 3D model and the floor plan being associated with each other.
After block 585, the routine continues to block 588 to store the generated mapping information and optionally other generated information, and to optionally further use the generated mapping information, such as to provide the generated 2D floor plan and/or 3D computer model for display on one or more client devices, to provide that generated information to one or more other devices for use in automating navigation of those devices and/or associated vehicles or other entities, etc.
If it is instead determined in block 510 that the information or instructions received in block 505 are not to generate mapping information for an indicated building, the routine continues instead to block 590 to perform one or more other indicated operations as appropriate. Such other operations may include, for example, receiving and responding to requests for previously generated computer models and/or floor plans and/or other generated information (e.g., requests for such information for use by an FPSDM system, requests for such information for display on one or more client devices, requests for such information to provide it to one or more other devices for use in automated navigation, etc.), obtaining and storing information about buildings for use in later operations (e.g., information about dimensions, numbers or types of rooms, total square footage, adjacent or nearby other buildings, adjacent or nearby vegetation, exterior images, etc.), etc.
After blocks 588 or 590, the routine continues to block 595 to determine whether to continue, such as until an explicit indication to terminate is received, or instead only if an explicit indication to continue is received. If it is determined to continue, the routine returns to block 505 to wait for and receive additional instructions or information, and otherwise continues to block 599 and ends.
The illustrated embodiment of the routine begins at block 605, where information or instructions are received. The routine continues to block 610 to determine whether the instructions or other information received in block 605 indicate to use one or more indicated initial buildings to determine one or more other similar target buildings, and if so the routine continues to perform blocks 615-660 to do so, and otherwise continues to block 670.
In block 615, for each of the one or more indicated initial buildings (e.g., as identified in the instructions or information received in block 605), the routine performs several activities as part of determining the similarity of those indicated building(s) to various other building (e.g., from a database or other group of stored floor plans and associated information for various buildings, which optionally include other initial buildings previously supplied to the routine 600). In the illustrated example, the activities performed by the routine in block 615 for each indicated building include generating or otherwise obtaining information (e.g., retrieving stored information) to represent the building, such as to represent some or all information from a floor plan for the building in an adjacency graph that includes information about room adjacency and/or inter-connectedness and about various other room and building attributes, and/or in an embedding vector that represents some or all of the building information in a compact form. If the representative information for the building is not already generated and available for retrieval, the techniques may include obtaining a floor plan for the building that includes information about locations of doors and other inter-room openings, windows, etc., room labels and/or types, and any other available building information that is associated with the floor plan (e.g., images and/or other information acquired in one or more rooms of the building). The activities performed by the routine in block 615 for each indicated building may further include associating the other available building information (if any) with corresponding room(s) and inter-room openings of the floor plan for the building, if not already done, and optionally analyzing the acquired other building information to determine additional attributes for a corresponding room or inter-room opening (or for the building as a whole), with those additional determined attributes then also associated with the building floor plan (e.g., with corresponding rooms or inter-room openings, or with the building as a whole). The activities performed by the routine in block 615 for each indicated building may further include generating an adjacency graph for the building floor plan that includes nodes for the rooms (and optionally for other areas, such as adjacent spaces external to the building, including areas outside of doors or other openings from the building to the exterior), and with edges representing inter-room connectivity (e.g., based on doors or other inter-room openings between rooms) and/or other inter-room adjacency, and with attributes from the floor plan and its associated additional information being stored with or otherwise associated with the corresponding nodes and edges for the rooms and inter-room connections/adjacencies in the adjacency graph. The activities performed by the routine in block 615 for each indicated building may further include optionally providing the adjacency graph for the building to one or more trained classification neural network(s) that each classify the building floor plan according to one or more subjective factors (e.g., accessibility friendly, an open floor plan, an atypical floor plan, etc.), and similarly storing any resulting classification attributes with the floor plan and its adjacency graph.
After block 615, the routine continues to block 617 to determine whether to use information from block 615 to generate predictions of one or more types of additional representative information for the indicated buildings, such as to add new information to the indicated buildings floor plans and/or adjacency graphs, and/or to correct or otherwise supplement existing information in the indicated buildings' floor plans and/or adjacency graphs, and if not continues to block 620. Otherwise, the routine continues to block 619 to, for each indicated building, provide the indicated building's adjacency graph and/or a subset of information from it to one or more trained machine learning models (e.g., one or more trained neural networks) to obtain predictions about one or more types of information for the indicated building. In the illustrated embodiment, the predicted information includes a type of room for some or all rooms in the building and types of inter-room connections between some or all rooms (e.g., connected by a door or other opening, adjacent with an intervening wall but not otherwise connected, not adjacent, etc.), although in other embodiments only one of the two types of information may be predicted, and/or other types of building attribute information may be predicted. The information predicted in block 619 is then used to update the floor plan and adjacency graph for the indicated building.
After block 619, or if it is determined in block 617 not to predict further building information, the routine continues to block 620 to determine whether to compare the indicated initial building(s) to the other buildings using embedding vectors, and if not continues to block 655 to compare pairs of buildings using their adjacency graphs. Otherwise, the routine continues to block 622 to, for each indicated initial building, use representation learning to generate a floor plan embedding vector for the indicated building that concisely represents the information included in the adjacency graph for the building. After block 622, the routine continues to block 625 to, for each of multiple other buildings (e.g., all available/known other buildings, a subset of the available other buildings that satisfy one or more defined tests, a group of other buildings supplied or otherwise indicated in the information received in block 605, etc.), determine a value for a distance (or other measure of difference or similarity) between a stored embedding vector for the other building (or to optionally dynamically generate and use a new embedding vector for that other building, if not previously generated and stored) and the embedding vector for each indicated initial building that was generated in block 622—if there are multiple indicated initial buildings, the routine further generates a combined distance value (e.g., an average, a cumulative total, etc.) for each other building by combining the determined distance values between that other building and all of the multiple indicated initial buildings. As discussed elsewhere, one or more of various distance metrics may be used to determine the distance values. After block 625, the routine continues to block 635, where it rank orders the multiple other buildings using the distance values determined in block 625 (using the combined distance values if there are multiple indicated initial buildings), and selects one or more best matches to use as the identified target buildings (e.g., all matches above a defined threshold, the single best match, etc., and optionally based on instructions or other information received in block 605), with those selected one or more best matches having the smallest determined distance values (i.e., the highest similarity to the one or more indicated buildings).
If it is instead determined in block 620 to not use embedding vectors for the comparison, the routine continues instead to block 655 where, for each combination of one of multiple other buildings (whether the same other buildings discussed with respect to block 625 or a different group of other buildings, such as a subset of all available other buildings that satisfy one or more criteria that may be the same or different than those used in block 625) and one of the one or more indicated initial buildings, the routine provides information for that other building and that indicated building (e.g., the adjacency graphs for the two buildings, the floor plans for the two buildings, etc.) to one or more trained similarity machine learning models (e.g., one or more trained neural networks) to determine a degree of similarity between the two buildings (e.g., a bi-model yes or no value, from a range or enumeration of multiple possible similarity values, a probability or other likelihood that the two buildings have a similarity above a defined threshold, etc.)—if there are multiple indicated initial buildings, the routine further generates a combined similarity degree value (e.g., an average, a cumulative total, etc.) for each of the other buildings by combining the determined similarity degree values for the other building from all of the multiple indicated initial buildings. After block 655, the routine continues to block 660, where it rank orders the multiple other buildings using the similarity degree values determined in block 655 (using the combined similarity degree values if there are multiple indicated buildings), and selects one or more best matches to use as the identified target buildings (e.g., all matches above a defined threshold, the single best match, etc., and optionally based on instructions or other information received in block 605), with the selected one or more best matches having the largest determined similarity degree values.
If it is instead determined in block 610 not to determine other target buildings that are similar to one or more indicated buildings, the routine continues instead to block 670, where it determines whether the instructions or other information received in block 605 indicate to determine other target buildings that match or are otherwise similar to one or more specified criteria, and if not continues to block 690. In some embodiments and situations (e.g., based on the instructions received in block 605), only other buildings that completely match all specified criteria may be considered, while in other embodiments and situations, other buildings may be considered that are only a partial match to the specified criteria. If it is determined in block 670 to determine other target buildings that match or are otherwise similar to one or more specified criteria, the routine continues instead to block 675 where, for each of multiple other buildings (whether the same other buildings discussed with respect to block 625 or a different group of other buildings, such as a subset of all available other buildings that satisfy one or more tests that may optionally be different than those used in block 625), stored adjacency graph information for the other building (or dynamically generated adjacency graph information for the other building, if not previously generated and stored) is analyzed to determine a degree of match to the specified criteria. It will be appreciated that a variety of matching criteria may be used, as discussed elsewhere herein. After block 675, the routine continues to block 680, where it rank orders the multiple other buildings using the degrees of match determined in block 675, and selects one or more best matches to use as the identified target buildings (e.g., all matches above the defined threshold, a single best match, etc., and optionally based on instructions or other information received in block 605), with the selected one or more best matches having the largest determined degrees of match (i.e., the highest similarity to the one or more specified criteria).
After block 635, 660 or 680, the routine continues to block 688, where it stores some or all of the information determined and generated in blocks 615-680, and returns information about the one or more selected best match target buildings.
If it is instead determined in block 670 that the information or instructions received in block 605 are not to determine one or more other target buildings that match one or more specified criteria, the routine continues instead to block 690 to perform one or more other indicated operations as appropriate. Such other operations may include, for example, receiving and responding to requests for previously identified building floor plan information (e.g., requests for such information for display on one or more client devices, requests for such information to provide it to one or more other devices for use in automated navigation, etc.), training one or more neural networks or other machine learning models to recognize and predict types of floor plan information (e.g., room types, types of inter-room connections, etc.), training one or more neural networks or other machine learning models to recognize similarities between buildings based on similarities in the buildings' information (e.g., floor plans, adjacency graphs, encoding vectors, etc.), training one or more classification neural networks or other machine learning models to classify building floor plans according to one or more subjective factors (e.g., accessibility friendly, an open floor plan, an atypical floor plan, a non-standard floor plan, etc.), using machine learning techniques to learn the attributes and/or other characteristics of adjacency graphs to encode in corresponding embedding vectors that are generated (e.g., the best attributes and/or other characteristics to allow subsequent automated identification of building floor plans that have attributes satisfying target criteria), generating and storing representative information for buildings (e.g., floor plans, adjacency graphs, embedding vectors, etc.) for later use, obtaining and storing information about users of the routine (e.g., search and/or selection preferences of a current user), etc.
After blocks 688 or 690, the routine continues to block 695 to determine whether to continue, such as until an explicit indication to terminate is received, or instead only if an explicit indication to continue is received. If it is determined to continue, the routine returns to block 605 to wait for and receive additional instructions or information, and otherwise continues to block 699 and ends.
The illustrated embodiment of the routine begins at block 705, where instructions or information are received. After block 705, the routine continues to block 750, where it determines whether the instructions or other information received in block 705 indicate to select one or more target buildings for presentation based on one or more indicated other buildings, and if not continues to block 760. Otherwise, the routine continues to block 752, where it obtains indications of one or more initial buildings to use, such as from current user selections (e.g., as currently selected by the user and/or indicated in the information or instructions received in block 705) and/or from one or more buildings previously identified as being of interest to the user (e.g., based on previous user selections or other previous user activities). The routine then invokes the FPSDM routine and supplies information about the one or more indicated initial buildings with corresponding instructions to determine one or more other buildings that are most similar to the initial buildings, and selects a best match target building to further use from the one or more returned other buildings (e.g., the returned other building with the highest similarity rating, or using another selection technique indicated in the instructions or other information received in block 705).
After block 752, or if it was instead determined in block 750 that the instructions or other information received in block 705 are not to select one or more target buildings based on other buildings, the routine continues to block 760 to determine whether the instructions or other information received in block 705 are to select one or more target buildings using specified criteria, and if not continues to block 770, where it obtains an indication of a target building to use from the user (e.g., based on a current user selection, such as from a displayed list or other user selection mechanism; based on information received in block 705; etc.). Otherwise, if it is determined in block 760 to select one or more target buildings from specified criteria, the routine continues instead to block 762, where it obtains indications of one or more search criteria to use, such as from current user selections or as indicated in the information or instructions received in block 705, and then invokes the FPSDM routine and supplies the one or more specified criteria with corresponding instructions to determine one or more buildings that satisfy the search criteria, and then selects a best match target building from the one or more returned buildings (e.g., the returned other building with the highest similarity rating, or using another selection technique indicated in the instructions or other information received in block 705). In some embodiments and situations, the user of the routine 700 may indicate to select target buildings based on both one or more other buildings and one or more specified criteria, and if so both blocks 752 and 762 may be performed, such as with the invocation in block 762 of the FPSDM routine supplying not only the search criteria but also one or more determined other buildings from block 752 for use by the FPSDM routine as the multiple other buildings to consider as potentially satisfying the search criteria. In other embodiments, the user may specify either search criteria or other buildings (but not both), or instead the functionality for only one of the two types of searches may be provided.
After blocks 762 or 770, the routine continues to block 710 to determine whether the instructions or other information received in block 705 are to display information or otherwise present information about a target building (e.g., via a floor plan that includes information about the interior of the target building), such as the target building from blocks 752, 762 or 770, and if not continues to block 790. Otherwise, the routine proceeds to block 712 to obtain the floor plan and/or other generated mapping information (e.g., a 3D computer model) for the target building and optionally indications of associated or linked information for the building interior and/or a surrounding location (e.g., additional images taken within or around the building), and selects an initial view of the retrieved information (e.g., a view of the floor plan, of at least some of the 3D computer model, etc.). In block 715, the routine then displays or otherwise presents the current view of the retrieved information, and waits in block 717 for a user selection. After a user selection in block 717, if it is determined in block 720 that the user selection corresponds to the current target building location (e.g., to change the current view of the displayed mapping information for that target building), the routine continues to block 722 to update the current view in accordance with the user selection, and then returns to block 715 to update the displayed or otherwise presented information accordingly. The user selection and corresponding updating of the current view may include, for example, displaying or otherwise presenting a piece of associated linked information that the user selects (e.g., a particular image associated with a displayed visual indication of a determined acquisition location), changing how the current view is displayed (e.g., zooming in or out; rotating information if appropriate; selecting a new portion of the floor plan and/or 3D computer model to be displayed or otherwise presented, such as with some or all of the new portion not being previously visible, or instead with the new portion being a subset of the previously visible information; etc.).
If it is instead determined in block 710 that the instructions or other information received in block 705 are not to present information representing a building interior, the routine continues instead to block 790 to perform any other indicated operations as appropriate, such as any housekeeping tasks, to configure parameters to be used in various operations of the system (e.g., based at least in part on information specified by a user of the system, such as a user of a mobile device who captures one or more building interiors, an operator user of the FPSDM system, etc.), to obtain and store other information about users of the routine (e.g., presentation and/or search preferences of a current user), to respond to requests for generated and stored information, etc.
Following block 790, or if it is determined in block 720 that the user selection does not correspond to the current target building location, the routine proceeds to block 795 to determine whether to continue, such as until an explicit indication to terminate is received, or instead only if an explicit indication to continue is received. If it is determined to continue (e.g., if the user made a selection in block 717 related to a new target building location to present), the routine returns to block 705 to await additional instructions or information (or to continue on past block 705 if the user made a selection in block 717 related to a new building location to present), and if not proceeds to step 799 and ends. In the illustrated embodiment, the routine in blocks 752 and 762 selects a best match target building to use, optionally from multiple returned other building candidates—in at least some embodiments, a queue of other such returned other buildings that are not first selected as best matches may further be saved and subsequently used (e.g., for the user to consecutively display or otherwise present information for multiple such other buildings), such as with the user selection in block 717 optionally indicating to select and use a next returned other building from such a queue, and if so for the routine to proceed to block 770 in the next iteration of the routine after returning to block 705.
Non-exclusive example embodiments described herein are further described in the following clauses.
A01. A computer-implemented method comprising:
obtaining, by a computing device, and for each of a plurality of houses, information about the house that includes a floor plan for the house having at least shapes and relative positions of rooms of the house;
determining, by the computing device and via analysis of the floor plans for the plurality of houses, characteristics of floor plans associated with one or more indicated subjective attributes;
determining, by the computing device and for each of multiple indicated houses, whether a floor plan for that indicated house has characteristics matching at least some of the determined characteristics to be associated with at least one of the one or more indicated subjective attributes, wherein the multiple indicated houses include one or more houses that are not part of the plurality of houses;
receiving, by the computing device, an indication of one house of the multiple indicated houses and one or more search criteria;
generating, by the computing device, and for the one indicated house by using at least the floor plan of the one indicated house, an adjacency graph that represents the one indicated house and that stores attributes associated with the one indicated house including at least one subjective attribute determined for the one indicated house, wherein the adjacency graph has multiple nodes that are each associated with one of multiple rooms of the one indicated house and stores information about one or more of the attributes that correspond to the associated room, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represent an adjacency in the one indicated house of the associated rooms for those two nodes;
generating, by the computing device, and using representation learning, an embedding vector to represent information from the adjacency graph that corresponds to a subset of a plurality of attributes of the indicated house including the at least one subjective attribute determined for the one indicated house;
determining, by the computing device, and from multiple other houses of the multiple indicated houses separate from the one indicated house, at least one other house that is similar to the one indicated house and that satisfies the one or more search criteria, including:
selecting, by the computing device, one or more of the multiple other houses that each has an associated additional embedding vector with a determined degree of similarity to the generated embedding vector for the one indicated house that is above a determined threshold and that is determined to have information in the additional adjacency graph for that other house matching the one or more search criteria, and using the selected one or more other houses as the determined at least one other house; and
presenting, by the computing device, information about attributes of the determined at least one other house, to enable a determination of one or more relations to the plurality of attributes associated with the indicated house.
A02. The computer-implemented method of clause A01 further comprising:
generating, by the computing device, and for each of the plurality of houses based at least in part on the floor plan for the house, an adjacency graph that represents the house and stores attributes associated with the house, wherein the adjacency graph has multiple nodes that are each associated with one of multiple rooms of the house and stores information about one or more of the attributes associated with the house that correspond to the associated room, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represents an adjacency in the house of the associated rooms for those two nodes;
learning, by the computing device, the subset of attributes for use in representing houses in embedding vectors, wherein the learning is based at least in part on using graph representation learning to search for a mapping function to map nodes in the adjacency graphs for the plurality of houses to a learned space with d-dimensional vectors in such a manner that similar graph nodes have similar embeddings in the learned space,
and wherein the generating of the embedding vector for the indicated house is performed after the learning and includes using the learned subset of attributes for the generated embedding vector.
A03. A computer-implemented method comprising:
obtaining, by a computing device, information about an indicated building having multiple rooms, including a floor plan determined for the indicated building that includes information about the multiple rooms including at least two-dimensional shapes and relative positions;
generating, by the computing device, and using at least the floor plan, an adjacency graph that represents the indicated building and that stores attributes associated with the indicated building, wherein the adjacency graph has multiple nodes that are each associated with one of the multiple rooms and stores information about one or more of the attributes that correspond to the associated room, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represent an adjacency in the indicated building of the associated rooms for those two nodes;
generating, by the computing device, and using representation learning, an embedding vector to represent information from the adjacency graph that corresponds to a subset of a plurality of attributes of the indicated building;
determining, by the computing device, and from a plurality of other buildings, at least one other building similar to the indicated building, including:
presenting, by the computing device, information about attributes of the determined at least one other building, to enable a determination of one or more relations to the plurality of attributes associated with the indicated building.
A04. A computer-implemented method comprising:
obtaining, by a computing device, and for each of a plurality of buildings, information about the building that includes a floor plan for the building having at least shapes and relative positions of multiple rooms of the building, and that includes one or more labels for each floor plan of whether it satisfies each of one or more indicated subjective attributes;
learning, by the computing device and via analysis of the floor plans for the plurality of buildings and of the labels included with the floor plans, characteristics of floor plans associated with the one or more indicated subjective attributes;
determining, by the computing device and for each of multiple indicated buildings separate from the plurality of buildings, whether a floor plan for that indicated building has characteristics matching at least some of the determined characteristics so as to be associated with at least one of the one or more indicated subjective attributes;
receiving, by the computing device, one or more search criteria that includes at least one specified subjective attribute of the one or more indicated subjective attributes;
determining, by the computing device, and from the multiple indicated buildings, an indicated building that matches the one or more search criteria, including determining that the indicated building has the at least one specified subjective attribute; and
presenting, by the computing device, information about the indicated building, to enable a determination of one or more relations to the one or more search criteria.
A05. A computer-implemented method comprising:
obtaining, by a computing device, information about an indicated building having multiple rooms, including a floor plan determined for the indicated building that includes at least shapes and relative positions of the multiple rooms;
generating, by the computing device and using at least the floor plan, an adjacency graph that represents the indicated building and stores a plurality of attributes associated with the indicated building, wherein the adjacency graph has multiple nodes that are each associated with one of the multiple rooms and stores information about one or more of the plurality of attributes that correspond to the associated room, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represents an adjacency in the indicated building of the associated rooms for those two nodes;
determining, by the computing device, and from a plurality of other buildings, one of the other buildings similar to the indicated building, including:
presenting, by the computing device, information about attributes of the determined one other building, to enable a determination of one or more relations to the plurality of attributes associated with the indicated building.
A06. A computer-implemented method comprising:
obtaining, by a computing device, and for each of a plurality of buildings, information about the building that includes a floor plan for the building having at least shapes and relative positions of multiple rooms of the building;
generating, by the computing device, and for each of the plurality of buildings based at least in part on the floor plan for the building, an adjacency graph that represents the building and stores attributes associated with the building, wherein the adjacency graph has multiple nodes that are each associated with one of the multiple rooms of the building and stores information about one or more of the attributes associated with the building that correspond to the associated room, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represents an adjacency in the building of the associated rooms for those two nodes;
receiving, by the computing device, one or more search criteria that are based at least in part on an indicated adjacency of at least two types of rooms;
determining, by the computing device and from the plurality of buildings, an indicated building that matches the one or more search criteria, including searching the adjacency graph for the indicated building to identify one or more edges in the adjacency graph representing one or more adjacencies in the indicated building that satisfy the indicated adjacency; and
presenting, by the computing device, information about attributes of the indicated building, to enable a determination of one or more relations to the one or more search criteria.
A07. A computer-implemented method comprising:
obtaining, by a computing device and for each of a plurality of buildings, information about the building that includes a floor plan for the building having at least shapes and relative positions of multiple rooms of the building, and one or more images taken in an interior of the building;
analyzing, by the computing device and for each of the plurality of buildings, the one or more images taken in the interior of the building to identify one or more visual attributes of that interior;
generating, by the computing device and for each of the plurality of buildings using at least the floor plan for that building, an adjacency graph representing the building and storing attributes associated with the building that include the one or more visual attributes of the interior of the building, wherein the adjacency graph has multiple nodes that are each associated with one of the multiple rooms of the building and stores information about one or more attributes that correspond to the associated room, wherein the stored information for at least one of the multiple nodes includes at least one of the visual attributes of the building that relates to the associated room for the at least one node, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represents an adjacency in the building of the associated rooms for those two nodes;
receiving, by the computing device, one or more search criteria based at least in part on one or more indicated visual attributes of one or more rooms;
determining, by the computing device, that an indicated building of the plurality of buildings matches the one or more search criteria, including searching the adjacency graph of the indicated building to identify at least one of the multiple rooms of the indicated building whose associated node has stored information that includes at least one visual attribute satisfying the one or more indicated visual attributes; and presenting, by the computing device, information about attributes of the indicated building, to enable a determination of one or more relations to the one or more search criteria.
A08. A computer-implemented method comprising:
obtaining, by a computing device and for each of a plurality of buildings, information about the building that includes a floor plan for the building having at least shapes and relative positions of multiple rooms of the building;
generating, by the computing device and for each of the plurality of buildings using at least the floor plan for that building, an adjacency graph representing the building and storing attributes associated with the building including objective attributes about the building that are able to be independently verified, wherein the adjacency graph has multiple nodes that are each associated with one of the multiple rooms of the building and stores information about one or more attributes that correspond to the associated room, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represents an adjacency in the building of the associated rooms for those two nodes;
predicting, by the computing device and for each of the plurality of buildings, one or more additional subjective attributes for the building by supplying information about the building to one or more trained machine learning models and receiving output indicating the one or more additional subjective attributes, and updating the adjacency graph for the building to further store information about the one or more additional subjective attributes for the building;
determining, by the computing device and after the updating, that an indicated building of the plurality of buildings matches one or more specified criteria corresponding to at least one indicated subjective attribute and at least one indicated objective attribute, by searching the updated adjacency graph for the indicated building to determine that stored information in that updated adjacency graph satisfies the at least one indicated subjective attribute and the at least one indicated objective attribute; and
presenting, by the computing device, information about attributes of the indicated building, to enable a determination of one or more relations to the one or more specified criteria.
A09. A computer-implemented method comprising:
obtaining, by a computing device and for each of a plurality of buildings, information about the building that includes a floor plan for the building having at least shapes and relative positions of multiple rooms of the building;
generating, by the computing device and for each of the plurality of buildings using at least the floor plan for that building, an adjacency graph representing the building and storing attributes associated with the building including objective attributes about the building that are able to be independently verified, wherein the adjacency graph has multiple nodes that are each associated with one of the multiple rooms of the building and stores information about one or more attributes that correspond to the associated room, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represents an adjacency in the building of the associated rooms for those two nodes;
predicting, by the computing device and for each of the plurality of buildings, room types for the multiple rooms of the building by supplying information about the building to one or more trained machine learning models and receiving output indicating the room types of the multiple rooms, and updating the adjacency graph for the building to further store information about the room types;
determining, by the computing device and after the updating, that an indicated building of the plurality of buildings matches one or more specified criteria corresponding to at least one indicated room type, by searching the updated adjacency graph for the indicated building to determine that stored information in that updated adjacency graph satisfies the at least one indicated room type; and
presenting, by the computing device, information about attributes of the indicated building, to enable a determination of one or more relations to the one or more specified criteria.
A10. A computer-implemented method comprising:
obtaining, by a computing device and for each of a plurality of buildings, information about the building that includes a floor plan for the building having at least shapes and relative positions of multiple rooms of the building;
generating, by the computing device and for each of the plurality of buildings using at least the floor plan for that building, an adjacency graph representing the building and storing attributes associated with the building including objective attributes about the building that are able to be independently verified, wherein the adjacency graph has multiple nodes that are each associated with one of the multiple rooms of the building and stores information about one or more attributes that correspond to the associated room, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represents an adjacency in the building of the associated rooms for those two nodes;
predicting, by the computing device and for each of the plurality of buildings and for each of the edges representing an adjacency in the adjacency graph for that building between two rooms of that building, a connectivity status of whether the two rooms are connected by an inter-room wall opening by supplying information about that building to one or more trained machine learning models and receiving output indicating that connectivity status, and updating the adjacency graph for that building to further store information about the connectivity status for each of the edges in the adjacency graph for that building;
determining, by the computing device and after the updating, that an indicated building of the plurality of buildings matches one or more specified criteria corresponding to at least one indicated connectivity status between at least two types of rooms, by searching the updated adjacency graph for the indicated building to determine that stored information in that updated adjacency graph satisfies the at least one indicated connectivity status; and presenting, by the computing device, information about attributes of the indicated building, to enable a determination of one or more relations to the one or more specified criteria.
A11. A computer-implemented method comprising:
obtaining, by a computing device, and for each of a plurality of buildings, information about the building that includes a floor plan for the building having at least shapes and relative positions of multiple rooms of the building;
generating, by the computing device, and for each of the plurality of buildings based at least in part on the floor plan for the building, an adjacency graph that represents the building and stores attributes associated with the building, wherein the adjacency graph has multiple nodes that are each associated with one of the multiple rooms of the building and stores information about one or more of the attributes associated with the building that correspond to the associated room, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represents an adjacency in the building of the associated rooms for those two nodes;
learning, by the computing device, a subset of attributes to represent buildings based at least in part on using graph representation learning to search for a mapping function to map nodes in the adjacency graphs for the plurality of buildings to a learned space with d-dimensional vectors in such a manner that similar graph nodes have similar embeddings in the learned space;
obtaining, by the computing device, information about an indicated building that is separate from the plurality of buildings and has multiple rooms, including a floor plan determined for the indicated building that includes at least shapes and relative positions of the multiple rooms and that indicates a plurality of attributes associated with the indicated building;
generating, by the computing device, and using representation learning, an embedding vector to represent information about the indicated building that corresponds to the subset of attributes;
determining, by the computing device, that the indicated building matches one or more specified criteria corresponding to one or more of the subset of attributes, by measuring a distance from the generated embedding vector to an additional embedding vector corresponding to the one or more specified criteria; and presenting, by the computing device, information about attributes of the indicated building, to enable a determination of one or more relations to the one or more specified criteria.
B01. A non-transitory computer-readable medium having stored executable software instructions and/or other stored contents that cause one or more computing systems to perform automated operations to implement the method of any one of clauses A01-A11.
B02. A non-transitory computer-readable medium having stored executable software instructions and/or other stored contents that cause one or more computing systems to perform automated operations to implement described techniques substantially as disclosed herein.
B03. A non-transitory computer-readable medium having stored contents that cause one or more computing systems to perform automated operations, the automated operations including at least:
determining, by the one or more computing systems, information about an indicated building having multiple rooms, including obtaining an embedding vector for the indicated building that is generated to represent at least a subset of a plurality of attributes associated with the indicated building and that is based at least in part on adjacency information for the indicated building including at least one attribute for each of the multiple rooms and further including indications of pairs of the multiple rooms adjacent to each other in the indicated building;
determining, by the one or more computing systems and from a plurality of other buildings, an other building corresponding to the indicated building, including:
providing, by the one or more computing systems, information about attributes of the determined other building, to enable a determination of one or more relations to the plurality of attributes associated with the indicated building.
B04. The non-transitory computer-readable medium of clause B03 wherein the determining of the information about the indicated building includes:
obtaining, by the one or more computing systems, information about the indicated building that includes a floor plan determined for the indicated building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the building, wherein the floor plan has information about the multiple rooms including at least shapes of the multiple rooms and relative positions of the multiple rooms;
generating, by the one or more computing systems and using at least the floor plan, the adjacency information for the indicated building, including an adjacency graph that stores the plurality of attributes and that has multiple nodes each associated with one of the multiple rooms and storing information about one or more of the attributes corresponding to the associated room and that further has multiple edges between the multiple nodes that are each between two nodes and represent an adjacency in the indicated building of the associated rooms for those two nodes; and
generating, by the one or more computing systems and using at least the adjacency graph, the embedding vector to represent information from the adjacency graph corresponding to the subset of the plurality of attributes of the indicated building, including representing information about adjacencies between the multiple rooms of the building.
B05. The non-transitory computer-readable medium of clause B04 wherein the stored contents include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform further automated operations including obtaining the plurality of images, wherein the plurality of images further include one or more images acquired at one or more acquisition locations external to the building, wherein the selecting of the one other building includes using a similarity distance as the measure of difference to measure a degree of similarity for that other building between the associated additional embedding vector for that other building and the embedding vector for the indicated building, and further includes selecting the other building based each of the one or more other buildings being above a defined threshold, and wherein the providing of the information about the attributes of the determined other building includes transmitting the information about the attributes of the determined other building over one or more computer networks to at least one client computing device for display.
B06. The non-transitory computer-readable medium of any one of clauses B03-B05 wherein the automated operations further include receiving, by the one or more computing systems, one or more search criteria and identifying the indicated building based at least in part on the one or more search criteria, and wherein the providing of the information about the attributes of the determined other building includes providing search results for presentation that include the determined other building.
B07. The non-transitory computer-readable medium of clause B06 wherein the one or more search criteria include one or more criteria that are based on adjacency of at least two types of rooms, wherein the embedding vector includes information about adjacencies of the multiple rooms in the indicated building, and wherein the additional embedding vector for the determined other building represents information about adjacencies of rooms in that other building, and the determined measure of difference for that additional embedding vector for the determined other building to the embedding vector for the indicated building is based at least in part on the adjacencies of the multiple rooms in the indicated building and the adjacencies of rooms in that other building.
B08. The non-transitory computer-readable medium of clause B06 wherein the one or more search criteria include one or more criteria that are based on visual attributes of a building interior, wherein the embedding vector includes information about visual attributes of an interior of the indicated building, and wherein the additional embedding vector for the determined other building represents information about additional visual attributes of an interior of that other building, and the determined measure of difference for that additional embedding vector for the determined other building to the embedding vector for the indicated building is based at least in part on the visual attributes of the interior of the indicated building and the additional visual attributes of the interior of that other building.
B09. The non-transitory computer-readable medium of clause B06 wherein the one or more search criteria include one or more criteria that are based on one or more types of exterior views from a building, wherein the embedding vector includes information about views from the indicated building to its surroundings, and wherein the additional embedding vector for the determined other building represents information about additional views from that other building to its surroundings, and the determined measure of difference for that additional embedding vector for the determined other building to the embedding vector for the indicated building is based at least in part on the views from the indicated building to its surroundings and the additional views from that other building to its surroundings.
B10. The non-transitory computer-readable medium of any one of clauses B03-B09 wherein the automated operations further include receiving, by the one or more computing systems, information about the indicated building being associated with a user, wherein the determining of the at least one other building is performed in response to the receiving of the information and includes determining information about the attributes of the determined other building that is personalized to the user, and wherein the providing of the information about the attributes of the determined other building includes presenting to the user the information about the attributes of the determined other building.
B11. The non-transitory computer-readable medium of any one of clauses B03-610 wherein the determined other building includes multiple other buildings, and wherein the providing of the information about the attributes of the determined other building includes determining, by the one or more computing systems, an expected assessment of at least one of condition or quality or value of the indicated building based at least in part on assessments of the multiple other buildings, and providing information about the determined expected assessment.
B12. The non-transitory computer-readable medium of any one of clauses B03-B11 wherein the adjacency information for the indicated building includes an adjacency graph that stores the plurality of attributes and that has multiple nodes each associated with one of the multiple rooms and storing information about one or more of the attributes corresponding to the associated room and that further has multiple edges between the multiple nodes that are each between two nodes and represent an adjacency in the indicated building of the associated rooms for those two nodes, and wherein the automated operations further include automatically learning, by the one or more computing systems, a subset of some attributes from the plurality of attributes of the indicated building to include in the embedding vector based at least in part on using graph representation learning to search for a mapping function to map nodes in the adjacency graph to a learned space with d-dimensional vectors in such a manner that similar graph nodes have similar embeddings in the learned space, and generating the embedding vector to encode information about the some attributes of the indicated building.
B13. The non-transitory computer-readable medium of any one of clauses B03-B12 wherein the automated operations further include generating, by the one or more computing systems, the embedding vector for the indicated building, including incorporating information in the embedding vector about, for each of the multiple rooms, at least one attribute that corresponds to the room and about information about adjacencies of rooms in the indicated building.
B14. The non-transitory computer-readable medium of clause B13 wherein the generating of the embedding vector further includes incorporating, by the one or more computing systems, information in the embedding vector about visual attributes of an interior of the indicated building that are determined based at least in part on an analysis of one or more images acquired in the interior of the indicated building.
B15. The non-transitory computer-readable medium of clause B13 wherein the generating of the embedding vector further includes incorporating, by the one or more computing systems, information in the embedding vector about views from the indicated building to its surroundings that are determined based on at least one of an analysis of one or more images acquired for the indicated building or information from one or more public records about the surroundings of the indicated building.
B16. The non-transitory computer-readable medium of clause B13 wherein the generating of the embedding vector further includes incorporating, by the one or more computing systems, information in the embedding vector about an exterior of the indicated building that is determined based at least in part on an analysis of one or more images acquired from the exterior of the indicated building.
B17. The non-transitory computer-readable medium of clause B13 wherein the plurality of attributes associated with the indicated building are objective attributes that are independently verifiable, wherein the automated operations further include predicting, by the one or more computing systems, one or more additional subjective attributes by supplying information about the indicated building to one or more trained machine learning models and receiving output indicating the one or more additional subjective attributes, and wherein the generating of the embedding vector further includes incorporating, by the one or more computing systems, information in the embedding vector about the one or more additional subjective attributes and about at least some of the objective attributes.
B18. The non-transitory computer-readable medium of clause B17 wherein the one or more additional subjective attributes include at least one of an atypical floor plan that differs from typical floor plans, or an open floor plan, or an accessible floor plan, or a non-standard floor plan.
B19. The non-transitory computer-readable medium of clause B13 wherein the automated operations further include predicting, by the one or more computing systems, room types of the multiple rooms by supplying information about the indicated building to one or more trained machine learning models and receiving output indicating the room types of the multiple room, and wherein the generating of the embedding vector further includes incorporating, by the one or more computing systems, information in the embedding vector about the room types of the multiple rooms.
B20. The non-transitory computer-readable medium of clause B19 wherein the predicting of the room types of the multiple rooms includes using, by the one or more computing systems and for each of the multiple rooms, information about any adjacencies of that room to any other rooms of the indicated building that are indicated by the adjacency information.
B21. The non-transitory computer-readable medium of clause B13 wherein the automated operations further include predicting, by the one or more computing systems, and for each adjacency in the indicated building between two rooms of the indicated building, a connectivity status of whether the two rooms are connected via an inter-room wall opening by supplying information about the indicated building to one or more trained machine learning models and receiving output indicating the connectivity status for each of the edges, and wherein the generating of the embedding vector further includes incorporating, by the one or more computing systems, information in the embedding vector about the connectivity status for each of the edges.
B22. The non-transitory computer-readable medium of clause B21 wherein the predicting, for each adjacency in the indicated building between two rooms of the indicated building, of the connectivity status includes at least one of predicting a wall between the two rooms without an inter-room wall opening or predicting a doorway between the two rooms or predicting a non-doorway wall opening between the two rooms, and wherein the incorporated information in the embedding vector includes information about the at least one of the predicted wall or the predicted doorway or other predicted non-doorway wall opening.
B23. The non-transitory computer-readable medium of clause B13 wherein the adjacency information for the indicated building includes an adjacency graph that stores the plurality of attributes and that has multiple nodes each associated with one of the multiple rooms and storing information about one or more of the attributes corresponding to the associated room and that further has multiple edges between the multiple nodes that are each between two nodes and represent an adjacency in the indicated building of the associated rooms for those two nodes, wherein the edges of the adjacency graph include one or more connectivity edges that each represents that two rooms whose adjacency is represented by the connectivity edge are connected in the indicated building via a doorway or a non-doorway wall opening, wherein the one or more connectivity edges each further stores information about characteristics of the doorway or the non-doorway wall opening for that connectivity edge, and wherein the generating of the embedding vector further includes incorporating, by the computing device, information in the embedding vector about characteristics of the doorway or the non-doorway wall opening for each of the one or more connectivity edges.
B24. The non-transitory computer-readable medium of clause B13 wherein the automated operations further include generating, by the one or more computing systems, the adjacency information, including generating an adjacency graph that stores the plurality of attributes and that has multiple nodes each associated with one of the multiple rooms and storing information about one or more of the attributes corresponding to the associated room and that further has multiple edges between the multiple nodes that are each between two nodes and represent an adjacency in the indicated building of the associated rooms for those two nodes and that further has one or more additional nodes that each corresponds to at least one of an exterior area outside of the indicated building or an external view from an interior of the building to an exterior of the indicated building and that further has at least one additional edge for each of the one or more additional nodes that connects that additional node to another node of the adjacency graph, and wherein the generating of the embedding vector for the indicated building includes incorporating information in the embedding vector about the at least one of the exterior area or the external view for each of the one or more additional nodes.
B25. The non-transitory computer-readable medium of any one of clauses B03-B24 wherein the automated operations further include receiving information about multiple buildings that include the indicated building and one or more additional indicated buildings, and obtaining a further embedding vector for each of the one or more additional indicated buildings, wherein the determining of the measure of difference is further performed for each of the one or more additional indicated buildings between the further embedding vector for that additional indicated building and the additional embedding vectors for each of the plurality of other buildings, and wherein the selecting of the one or more other buildings is further based on the determined measures of difference between the associated additional embedding vector for each of the one or more other buildings and the further embedding vectors for each of the one or more additional indicated buildings, such that selection of the one or more other buildings is based on aggregate differences for the embedding vector of the indicated building and the further embedding vectors for the additional indicated buildings to the associated additional embedding vector for each of the one or more other buildings.
B26. A non-transitory computer-readable medium having stored contents that cause one or more computing systems to perform automated operations, the automated operations including at least:
determining, by the one or more computing systems and via analysis of floor plans for a plurality of buildings, floor plan characteristics associated with one or more indicated subjective attributes;
determining, by the one or more computing systems and for each of multiple indicated buildings that include one or more buildings separate from the plurality of buildings, whether a floor plan for that indicated building has characteristics matching at least some of the determined characteristics so as to be associated with at least one of the one or more indicated subjective attributes;
determining, by the one or more computing systems, a building from the multiple indicated buildings that has at least one specified subjective attribute of the one or more indicated subjective attributes; and
providing, by the one or more computing systems, information about the determined building, to enable a determination of information related to the at least one specified subjective attribute.
B27. The non-transitory computer-readable medium of clause B26 wherein the stored contents include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to receive information for the floor plans of the plurality of buildings that includes one or more supplied indications for each floor plan of whether it satisfies each of the one or more indicated subjective attributes, and to perform the determining of the floor plan characteristics by learning the floor plan characteristics based at least in part on the supplied indications for the floor plans of the plurality of buildings, and to perform the providing of the information about the determined building by transmitting the information about the determined building over one or more computer networks to a client computing device of a user for display, wherein the automated operations further include receiving one or more search criteria separate from the at least one specified subjective attribute, and wherein the determining of the building further includes determining that the building further matches the one or more search criteria.
B28. The non-transitory computer-readable medium of clause B26 wherein the at least one specified subjective attribute includes at least one of an atypical floor plan that differs from typical floor plans, or an open floor plan, or an accessible floor plan, or a non-standard floor plan.
B29. A non-transitory computer-readable medium having stored contents that cause one or more computing systems to perform automated operations, the automated operations including at least:
determining, by the one or more computing systems, information about an indicated building having multiple rooms, including obtaining adjacency information for the indicated building that includes a plurality of attributes associated with the indicated building and further includes indications of adjacencies in the indicated building of the multiple rooms, wherein each of the multiple rooms is associated with at least one attribute;
determining, by the one or more computing systems, an other building corresponding to the indicated building, including:
B30. The non-transitory computer-readable medium of clause B29 wherein the stored contents include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the determining of the information about the indicated building by:
obtaining information about the indicated building that includes a floor plan determined for the indicated building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the building, wherein the floor plan has information about the multiple rooms including at least shapes and relative positions of the multiple rooms; and
generating, using at least the floor plan, an adjacency graph that includes the adjacency information for the indicated building, wherein the adjacency graph has multiple nodes that are each associated with one of the multiple rooms and stores information about one or more of the plurality of attributes that correspond to the associated room, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represents an adjacency in the indicated building of the associated rooms for those two nodes,
and wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the providing of the information about the determined other building by transmitting the information about the determined other building over one or more computer networks to a client computing device for display to a user.
B31. A non-transitory computer-readable medium having stored contents that cause one or more computing systems to perform automated operations, the automated operations including at least:
obtaining, by the one or more computing systems, adjacency information for an indicated building that includes a plurality of attributes associated with the indicated building and further includes indications of adjacencies between multiple rooms of the indicated building, wherein each of the multiple rooms is associated with at least one attribute about the indicated building;
receiving, by the one or more computing systems, one or more criteria that are based at least in part on an indicated adjacency of at least two types of rooms;
determining, by the one or more computing systems, that the indicated building matches the one or more specified criteria, including searching the adjacency information to identify one or more adjacencies in the indicated building that satisfy the indicated adjacency; and
providing, by the one or more computing systems, information about the indicated building, to enable a determination of one or more relations of the indicated building to the one or more specified criteria.
B32. The non-transitory computer-readable medium of clause B31 wherein the stored contents include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the obtaining of the adjacency information by:
obtaining information about the indicated building that includes a floor plan determined for the indicated building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the indicated building, wherein the floor plan has information about the multiple rooms of the indicated building including at least shapes and relative positions of the multiple rooms; and
generating, using at least the floor plan, an adjacency graph that includes the adjacency information for the indicated building, wherein the adjacency graph has multiple nodes that are each associated with one of the multiple rooms and stores information about one or more of the plurality of attributes associated with the indicated building that correspond to the associated room, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represents an adjacency in the indicated building of the associated rooms for those two nodes,
and wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the providing of the information about the indicated building by transmitting the information about the indicated building over one or more computer networks to a client computing device for display to a user.
B33. A non-transitory computer-readable medium having stored contents that cause one or more computing systems to perform automated operations, the automated operations including at least:
obtaining, by the one or more computing systems, information about an indicated building having multiple rooms, including one or more visual attributes of an interior of the indicated building that are determined from analysis of visual data of one or more images taken in that interior, and further including adjacency information for the indicated building that includes a plurality of attributes associated with the indicated building and further includes indications of adjacencies between multiple rooms of the indicated building, wherein each of the multiple rooms is associated with at least one attribute about the indicated building, and wherein at least one of the multiple rooms is associated with at least one of the visual attributes of the indicated building that relates to that room;
receiving, by the one or more computing systems, one or more criteria based at least in part on one or more indicated visual attributes of one or more rooms;
determining, by the one or more computing systems, that the indicated building matches the one or more criteria, including searching the adjacency information to identify at least one of the multiple rooms having associated information that includes at least one visual attribute satisfying the one or more indicated visual attributes; and
providing, by the one or more computing systems, information about the indicated building, to enable a determination of one or more relations to the plurality of attributes associated with the indicated building.
B34. The non-transitory computer-readable medium of clause B33 wherein the stored contents include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the obtaining of the information about the indicated building by:
obtaining information about the indicated building that includes a floor plan determined for the indicated building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the indicated building, wherein the floor plan has information about the multiple rooms of the indicated building including at least shapes and relative positions of the multiple rooms; and generating, using at least the floor plan, an adjacency graph that includes the adjacency information for the indicated building, wherein the adjacency graph has multiple nodes that are each associated with one of the multiple rooms and stores information about one or more of the plurality of attributes associated with the indicated building that correspond to the associated room, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represents an adjacency in the indicated building of the associated rooms for those two nodes, and wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the providing of the information about the indicated building by transmitting the information about the indicated building over one or more computer networks to a client computing device for display to a user.
B35. A non-transitory computer-readable medium having stored contents that cause one or more computing systems to perform automated operations, the automated operations including at least:
obtaining, by the one or more computing systems, information about an indicated building having multiple rooms, including adjacency information for the indicated building that includes a plurality of attributes associated with the indicated building and further includes indications of adjacencies between multiple rooms of the indicated building, wherein the plurality of attributes include objective attributes about the indicated building that are able to be independently verified, wherein each of the multiple rooms is associated with at least one attribute about the indicated building, and wherein at least one of the multiple rooms is associated with at least one of the visual attributes of the indicated building that relates to that room;
predicting, by the one or more computing systems, one or more additional subjective attributes for the indicated building, and updating the adjacency information for the indicated building to further store information about the one or more additional subjective attributes for the indicated building;
determining, by the one or more computing systems and after the updating, that the indicated building matches one or more specified criteria corresponding to at least one indicated subjective attribute, by searching the updated adjacency information for the indicated building to determine that stored information in that updated adjacency information satisfies the at least one indicated subjective attribute; and
providing, by the one or more computing systems, information about the indicated building, to enable a determination of one or more relations of the indicated building to the one or more specified criteria.
B36. The non-transitory computer-readable medium of clause B35 wherein the stored contents include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the obtaining of the information about the indicated building by:
obtaining information about the indicated building that includes a floor plan determined for the indicated building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the indicated building, wherein the floor plan has information about the multiple rooms of the indicated building including at least shapes and relative positions of the multiple rooms; and
generating, using at least the floor plan, an adjacency graph that includes the adjacency information for the indicated building, wherein the adjacency graph has multiple nodes that are each associated with one of the multiple rooms and stores information about one or more of the plurality of attributes associated with the indicated building that correspond to the associated room, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represents an adjacency in the indicated building of the associated rooms for those two nodes,
and wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the providing of the information about the indicated building by transmitting the information about the indicated building over one or more computer networks to a client computing device for display to a user.
B37. A non-transitory computer-readable medium having stored contents that cause one or more computing systems to perform automated operations, the automated operations including at least:
obtaining, by the one or more computing systems, information about an indicated building having multiple rooms, including adjacency information for the indicated building that includes a plurality of attributes associated with the indicated building and further includes indications of adjacencies between multiple rooms of the indicated building, wherein each of the multiple rooms is associated with at least one attribute about the indicated building, and wherein at least one of the multiple rooms is associated with at least one of the visual attributes of the indicated building that relates to that room;
predicting, by the one or more computing systems, one or more room types for one or more rooms in the indicated building, and updating the adjacency information for the indicated building to further store information about the one or more room types for the one or more rooms;
determining, by the one or more computing systems and after the updating, that the indicated building matches one or more specified criteria corresponding to at least one indicated room type, by searching the updated adjacency information for the indicated building to determine that stored information in that updated adjacency information satisfies the at least one indicated room type; and
providing, by the one or more computing systems, information about the indicated building, to enable a determination of one or more relations of the indicated building to the one or more specified criteria.
B38. The non-transitory computer-readable medium of clause B37 wherein the stored contents include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the obtaining of the information about the indicated building by:
obtaining information about the indicated building that includes a floor plan determined for the indicated building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the indicated building, wherein the floor plan has information about the multiple rooms of the indicated building including at least shapes and relative positions of the multiple rooms; and
generating, using at least the floor plan, an adjacency graph that includes the adjacency information for the indicated building, wherein the adjacency graph has multiple nodes that are each associated with one of the multiple rooms and stores information about one or more of the plurality of attributes associated with the indicated building that correspond to the associated room, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represents an adjacency in the indicated building of the associated rooms for those two nodes,
and wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the providing of the information about the indicated building by transmitting the information about the indicated building over one or more computer networks to a client computing device for display to a user.
B39. A non-transitory computer-readable medium having stored contents that cause one or more computing systems to perform automated operations, the automated operations including at least:
obtaining, by the one or more computing systems, information about an indicated building having multiple rooms, including adjacency information for the indicated building that includes a plurality of attributes associated with the indicated building and further includes indications of adjacencies between multiple rooms of the indicated building, wherein each of the multiple rooms is associated with at least one attribute about the indicated building, and wherein at least one of the multiple rooms is associated with at least one of the visual attributes of the indicated building that relates to that room;
predicting, by the one or more computing systems, one or more connectivity statuses between two or more rooms in indicated building, and updating the adjacency information for the indicated building to further store information about the one or more connectivity statuses for the two or more rooms;
determining, by the one or more computing systems and after the updating, that the indicated building matches one or more specified criteria corresponding to at least one indicated connectivity status, by searching the updated adjacency information for the indicated building to determine that stored information in that updated adjacency information satisfies the at least one indicated connectivity status; and
providing, by the one or more computing systems, information about the indicated building, to enable a determination of one or more relations of the indicated building to the one or more specified criteria.
B40. The non-transitory computer-readable medium of clause B39 wherein the stored contents include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the obtaining of the information about the indicated building by:
obtaining information about the indicated building that includes a floor plan determined for the indicated building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the indicated building, wherein the floor plan has information about the multiple rooms of the indicated building including at least shapes and relative positions of the multiple rooms; and
generating, using at least the floor plan, an adjacency graph that includes the adjacency information for the indicated building, wherein the adjacency graph has multiple nodes that are each associated with one of the multiple rooms and stores information about one or more of the plurality of attributes associated with the indicated building that correspond to the associated room, and wherein the adjacency graph further has multiple edges between the multiple nodes that are each between two nodes and represents an adjacency in the indicated building of the associated rooms for those two nodes,
and wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the providing of the information about the indicated building by transmitting the information about the indicated building over one or more computer networks to a client computing device for display to a user.
B41. A non-transitory computer-readable medium having stored contents that cause one or more computing systems to perform automated operations, the automated operations including at least:
obtaining, by the one or more computing systems and for each of a plurality of buildings, adjacency information for the building that includes a plurality of attributes associated with the building and further includes indications of adjacencies between multiple rooms of the building, wherein each of the multiple rooms is associated with at least one attribute about the building;
learning, by the one or more computing systems and based at least in part on the adjacency information for each of the plurality of buildings, a subset of attributes to represent buildings such that similar buildings have similar information about the subset of attributes for those similar buildings;
generating, by the one or more computing systems and for an indicated building separate from the plurality of buildings, an embedding vector to represent information about the indicated building that corresponds to the subset of attributes;
determining, by the one or more computing systems, that the generated embedding vector for the indicated building matches one or more specified criteria corresponding to one or more of the subset of attributes; and
providing, by the one or more computing systems, information about the indicated building, to enable a determination of one or more relations of the indicated building to the one or more specified criteria.
B42. The non-transitory computer-readable medium of clause B41 wherein the stored contents include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the generating of the embedding vector for the indicated building by:
obtaining information about the indicated building that includes a floor plan determined for the indicated building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the indicated building, wherein the floor plan has information about the multiple rooms of the indicated building including at least shapes and relative positions of the multiple rooms; and
generating, using at least the floor plan, adjacency information for the indicated building that includes a plurality of attributes associated with the indicated building and further includes indications of adjacencies between multiple rooms of the indicated building, wherein each of the multiple rooms is associated with at least one attribute about the indicated building; and
performing the generating of the embedding vector for the indicated building based at least in part on the generated adjacency information for the indicated building,
and wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the providing of the information about the indicated building by transmitting the information about the indicated building over one or more computer networks to a client computing device for display to a user.
C01. One or more computing systems comprising one or more hardware processors and one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations to implement the method of any of clauses A01-A11.
C02. One or more computing systems comprising one or more hardware processors and one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations to implement described techniques substantially as disclosed herein.
C03. A system comprising:
one or more hardware processors of one or more computing systems; and
one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations including at least:
C04. The system of clause C03 wherein the determining of the information about the indicated building includes:
obtaining information about the indicated building that includes a floor plan determined for the indicated building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the building, wherein the floor plan has information about the multiple rooms including at least shapes and relative positions of the multiple rooms;
generating, using at least the floor plan, the adjacency graph; and
generating, using at least the adjacency graph, the embedding vector to represent information from the adjacency graph corresponding to the subset of the plurality of attributes of the indicated building, including representing information about adjacencies between the multiple rooms of the building.
C05. The system of any one of clauses C03-C04 further comprising a client computing device of a user, wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform further automated operations including generating the adjacency graph based at least in part on analysis of visual data of a plurality of images acquired at a plurality of acquisition locations that are associated with the building and that include multiple acquisition locations with the multiple rooms of the building and that further include one or more acquisition locations external to the building, wherein the providing of the information about the attributes of the determined at least one other building includes transmitting the information about the attributes of the determined at least one other building over one or more computer networks to the client computing device, and wherein the automated operations further include receiving by the client computing device and displaying on the client computing device the provided information about the attributes of the determined at least one other building, and transmitting, by the client computing device and to the one or more computing systems, information from an interaction of the user with a user-selectable control on the client computing device to cause a modification of information displayed on the client computing device for the determined at least one other building.
C06. A system comprising:
one or more hardware processors of one or more computing systems; and
one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations including at least:
C07. The system of clause C06 further comprising a client computing device of a user, wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to receive information for the floor plans of the plurality of buildings that includes one or more supplied indications for each floor plan of whether it satisfies each of the one or more indicated subjective attributes, and to perform the determining of the floor plan characteristics by learning the floor plan characteristics based at least in part on the supplied indications for the floor plans of the plurality of buildings, and to perform the providing of the information about the at least one indicated building by transmitting the information about the at least one indicated building over one or more computer networks to the client computing device, and wherein the automated operations further include receiving by the client computing device and displaying on the client computing device the provided information about the at least one indicated building, and transmitting, by the client computing device and to the one or more computing systems, information from an interaction of the user with a user-selectable control on the client computing device to cause a modification of information displayed on the client computing device for the at least one indicated building.
C08. A system comprising:
one or more hardware processors of one or more computing systems; and
one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations including at least:
C09. The system of clause C08 further comprising a client computing device of a user, wherein the obtaining of the information about the indicated building further includes:
obtaining information about the indicated building that includes a floor plan determined for the indicated building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the building, wherein the floor plan has information about the multiple rooms including at least shapes and relative positions of the multiple rooms; and
generating, using at least the floor plan, the adjacency graph,
and wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the providing of the information about the determined at least one other building by transmitting the information about the determined at least one other building over one or more computer networks to the client computing device, and wherein the automated operations further include receiving by the client computing device and displaying on the client computing device the provided information about the determined at least one other building, and transmitting, by the client computing device and to the one or more computing systems, information from an interaction of the user with a user-selectable control on the client computing device to cause a modification of information displayed on the client computing device for the determined at least one other building.
C10. A system comprising:
one or more hardware processors of one or more computing systems; and
one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations including at least:
C11. The system of clause C10 further comprising a client computing device of a user, wherein the obtaining of the information about each of the plurality of buildings further includes:
obtaining information about the building that includes a floor plan determined for the building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the building, wherein the floor plan has information about the multiple rooms including at least shapes and relative positions of the multiple rooms; and
generating, using at least the floor plan, the adjacency graph for the building,
and wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the providing of the information about the at least one indicated building by transmitting the information about the at least one indicated building over one or more computer networks to the client computing device, and wherein the automated operations further include receiving by the client computing device and displaying on the client computing device the provided information about the at least one indicated building, and transmitting, by the client computing device and to the one or more computing systems, information from an interaction of the user with a user-selectable control on the client computing device to cause a modification of information displayed on the client computing device for the at least one indicated building.
C12. A system comprising:
one or more hardware processors of one or more computing systems; and
one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations including at least:
C13. The system of clause C12 further comprising a client computing device of a user, wherein the obtaining of the information about each of the plurality of buildings further includes:
obtaining information about the building that includes a floor plan determined for the building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the building, wherein the floor plan has information about the multiple rooms including at least shapes and relative positions of the multiple rooms; and
generating, using at least the floor plan, the adjacency graph for the building,
and wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the providing of the information about the at least one indicated building by transmitting the information about the at least one indicated building over one or more computer networks to the client computing device, and wherein the automated operations further include receiving by the client computing device and displaying on the client computing device the provided information about the at least one indicated building, and transmitting, by the client computing device and to the one or more computing systems, information from an interaction of the user with a user-selectable control on the client computing device to cause a modification of information displayed on the client computing device for the at least one indicated building.
C14. A system comprising:
one or more hardware processors of one or more computing systems; and
one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations including at least:
C15. The system of clause C14 further comprising a client computing device of a user, wherein the obtaining of the information about each of the plurality of buildings further includes:
obtaining information about the building that includes a floor plan determined for the building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the building, wherein the floor plan has information about the multiple rooms including at least shapes and relative positions of the multiple rooms; and
generating, using at least the floor plan, the adjacency graph for the building,
and wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the providing of the information about the at least one indicated building by transmitting the information about the at least one indicated building over one or more computer networks to the client computing device, and wherein the automated operations further include receiving by the client computing device and displaying on the client computing device the provided information about the at least one indicated building, and transmitting, by the client computing device and to the one or more computing systems, information from an interaction of the user with a user-selectable control on the client computing device to cause a modification of information displayed on the client computing device for the at least one indicated building.
C16. A system comprising:
one or more hardware processors of one or more computing systems; and
one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations including at least:
C17. The system of clause C16 further comprising a client computing device of a user, wherein the obtaining of the information about each of the plurality of buildings further includes:
obtaining information about the building that includes a floor plan determined for the building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the building, wherein the floor plan has information about the multiple rooms including at least shapes and relative positions of the multiple rooms; and
generating, using at least the floor plan, the adjacency graph for the building,
and wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the providing of the information about the at least one indicated building by transmitting the information about the at least one indicated building over one or more computer networks to the client computing device, and wherein the automated operations further include receiving by the client computing device and displaying on the client computing device the provided information about the at least one indicated building, and transmitting, by the client computing device and to the one or more computing systems, information from an interaction of the user with a user-selectable control on the client computing device to cause a modification of information displayed on the client computing device for the at least one indicated building.
C18. A system comprising:
one or more hardware processors of one or more computing systems; and
one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations including at least:
C19. The system of clause C18 further comprising a client computing device of a user, wherein the obtaining of the information about each of the plurality of buildings further includes:
obtaining information about the building that includes a floor plan determined for the building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the building, wherein the floor plan has information about the multiple rooms including at least shapes and relative positions of the multiple rooms; and
generating, using at least the floor plan, the adjacency graph for the building,
and wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the providing of the information about the at least one indicated building by transmitting the information about the at least one indicated building over one or more computer networks to the client computing device, and wherein the automated operations further include receiving by the client computing device and displaying on the client computing device the provided information about the at least one indicated building, and transmitting, by the client computing device and to the one or more computing systems, information from an interaction of the user with a user-selectable control on the client computing device to cause a modification of information displayed on the client computing device for the at least one indicated building.
C20. A system comprising:
one or more hardware processors of one or more computing systems; and
one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations including at least:
C21. The system of clause C20 further comprising a client computing device of a user, wherein the multiple indicated buildings are separate from the plurality of buildings, and wherein the obtaining of the information about each of the plurality of buildings further includes:
obtaining information about the building that includes a floor plan determined for the building based at least in part on analysis of visual data of a plurality of images acquired at multiple acquisition locations within the building, wherein the floor plan has information about the multiple rooms including at least shapes and relative positions of the multiple rooms; and
generating, using at least the floor plan, the adjacency graph for the building,
and wherein the stored instructions include software instructions that, when executed by at least one of the one or more computing systems, cause the at least one computing system to perform the providing of the information about the at least one indicated building by transmitting the information about the at least one indicated building over one or more computer networks to the client computing device, and wherein the automated operations further include receiving by the client computing device and displaying on the client computing device the provided information about the at least one indicated building, and transmitting, by the client computing device and to the one or more computing systems, information from an interaction of the user with a user-selectable control on the client computing device to cause a modification of information displayed on the client computing device for the at least one indicated building.
D01. A computer program adapted to perform the method of any of clauses A01-A11 when the computer program is run on a computer.
D02. A computer program adapted to perform the automated operations of any of clauses B01-B42 when the computer program is run on a computer.
D03. A computer program adapted to perform the automated operations of any of clauses C01-C21 when the computer program is run on a computer.
Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be appreciated that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions. It will be further appreciated that in some implementations the functionality provided by the routines discussed above may be provided in alternative ways, such as being split among more routines or consolidated into fewer routines. Similarly, in some implementations illustrated routines may provide more or less functionality than is described, such as when other illustrated routines instead lack or include such functionality respectively, or when the amount of functionality that is provided is altered. In addition, while various operations may be illustrated as being performed in a particular manner (e.g., in serial or in parallel, or synchronous or asynchronous) and/or in a particular order, in other implementations the operations may be performed in other orders and in other manners. Any data structures discussed above may also be structured in different manners, such as by having a single data structure split into multiple data structures and/or by having multiple data structures consolidated into a single data structure. Similarly, in some implementations illustrated data structures may store more or less information than is described, such as when other illustrated data structures instead lack or include such information respectively, or when the amount or types of information that is stored is altered.
From the foregoing it will be appreciated that, although specific embodiments have been described herein for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the invention. Accordingly, the invention is not limited except as by corresponding claims and the elements recited by those claims. In addition, while certain aspects of the invention may be presented in certain claim forms at certain times, the inventors contemplate the various aspects of the invention in any available claim form. For example, while only some aspects of the invention may be recited as being embodied in a computer-readable medium at particular times, other aspects may likewise be so embodied.
This application claims the benefit of U.S. Provisional Patent Application No. 63/081,744, filed Sep. 22, 2020 and entitled “Automated Identification And Use Of Building Floor Plan Information,” which is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63081744 | Sep 2020 | US |
