Patent Application
20230073366
In some cases, to find out the height of a building an individual must access records (e.g., building plans, city documents, etc.) but this is not practical on a larger scale, such as measuring all buildings in a city. Building heights cannot currently be measured with mono two-dimensional (2D) aerial imagery or satellite imagery, due to the imagery not containing true three-dimensional (3D) information. Traditionally stereo pairs of 2D imagery have been used to measure building heights, but these stereo pairs are seldom available “off the shelf” and must often be tasked specifically for a job, which can lead to increased costs.
The techniques introduced here may be better understood by referring to the following Detailed Description in conjunction with the accompanying drawings, in which like reference numerals indicate identical or functionally similar elements.
Aspects of the present disclosure are directed to methods and systems for calculating building heights from mono imagery. Light detection and ranging (LiDAR) scanning and stereo photogrammetry are used (e.g., in digital surface models (DSMs)) to measure building heights. LiDAR can be cost prohibitive and challenging to use in many regions. Stereo photogrammetry relies on having stereo pairs of images which are not always available and more expensive than mono imagery. Satellites, planes, and aerial vehicles can capture an image of a building, and a building height calculation system can determine the height of the building in the image. The disclosed method utilizes techniques to identify the footprint and rooftop of a building (e.g., rooftop vectors and footprint vectors) in an image, determine the horizontal offset separating the footprint and rooftop, and calculate the height of the building based on the horizontal offset and the angle the image was captured from. “Building” as used herein can refer to any structure or object in terrain. Note that while this description will refer to a single building, the singular implicates the plural and multiple buildings could be involved.
Several implementations are discussed below in more detail in reference to the figures.
At step 102, process 100 performs orthorectification of the image against the DTM to remove effects of terrain distortion from the image. Orthorectification to a DTM is a process in earth imaging, in which the map (geographic or projected) coordinates of each pixel in an image of the earth are identified with the aid of a source DTM. This allows the formation of a new image. As a result of applying orthorectification, one or more pixels in the new image can be placed in its correct position(s) with respect to map coordinates.
At step 104, process 100 can execute an edge detection algorithm on the image to identify the edges (e.g., contrast changes) of the building in the image.
At step 106, process 100 calculates a building lean direction based on image meta data of the angle the camera was facing when the image was captured. At step 108, process 100 executes a machine learning algorithm to determine the footprint and rooftop vectors of the building in the image. Process 100 can convert the image into an input for a machine learning model, apply the input to the machine learning model, and in response obtaining the rooftop vector and the footprint vector of the building based on output from the machine learning model. Additional details on machine leaning are described in FIG. 10. At step 110, process 100 receives the footprint vector and rooftop vector of the building from a third party (e.g., company, organization, documents, etc.). Example 500 of
At step 112, process 100 executes an algorithm with the building footprint vector and scans the edge raster in the direction along a search corridor (e.g., 2 or 3 pixels wide). Process 100 can identify a connection between the building footprint and the edge raster. For example, the connection is expected to occur at the rooftop of the building. Process 100 can search for the highest correlation between the rooftop and footprint vectors rather than an exact match, so inaccuracies (e.g., small inaccuracies, such as 95% accuracy) in the building footprint vectors do not affect the detection of the rooftop vector. Process 100 can perform a correlation calculation between the rooftop and footprint vectors. For example, process 100 uses the building footprint vector as a template and the edge raster as the comparison image. Process 100 calculates the mean of all the template pixels (MT), the mean of all the image pixels (MI), and loops through each template pixel and image pixel and calculates equation 1 as the correlation (where T and I are the template and image pixel values). Process 100 can perform the calculation at each pixel step along the search corridor and use the highest correlation for the roof position.
Sum((T−MT)*(I−MI))/Sqrt(Sum((T−TT)2)*Sum((I−TI)2)) (Equation 1)
Example 600 of
At step 202, process 200 creates a shadow mask for the building using the footprint vector as a reference and the image. The shadow mask is created by setting a threshold value on the pixel intensity of the panchromatic image. If the pixel intensity is below the threshold, that pixel is flagged as a shadow. The shadow length is measured by counting the number of flagged shadow pixels in the opposite direction of the sun azimuth, starting from the building footprint.
At step 204, process 200 measures a building height based on the camera angle, the distance from the camera to the building, and the angle of the sun (e.g., from latitude, longitude, and time of day). The relationship between shadow length and building height is as follows: BH=L×tan(θ) where BH is the building height, L is the shadow length to the building footprint, and θ is the sun elevation. Both the sun elevation and azimuth are provided in the image metadata.
At step 206, process 200 compares the shadow building height (from step 204) to the RPC calculated building height (from step 114 of
Processors 810 can be a single processing unit or multiple processing units in a device or distributed across multiple devices. Processors 810 can be coupled to other hardware devices, for example, with the use of a bus, such as a PCI bus or SCSI bus. The processors 810 can communicate with a hardware controller for devices, such as for a display 830. Display 830 can be used to display text and graphics. In some implementations, display 830 provides graphical and textual visual feedback to a user. In some implementations, display 830 includes the input device as part of the display, such as when the input device is a touchscreen or is equipped with an eye direction monitoring system. In some implementations, the display is separate from the input device. Examples of display devices are: an LCD display screen, an LED display screen, a projected, holographic, or augmented reality display (such as a heads-up display device or a head-mounted device), and so on. Other I/O devices 840 can also be coupled to the processor, such as a network card, video card, audio card, USB, firewire or other external device, camera, printer, speakers, CD-ROM drive, DVD drive, disk drive, or Blu-Ray device.
In some implementations, the device 800 also includes a communication device capable of communicating wirelessly or wire-based with a network node. The communication device can communicate with another device or a server through a network using, for example, TCP/IP protocols. Device 800 can utilize the communication device to distribute operations across multiple network devices.
The processors 810 can have access to a memory 850 in a device or distributed across multiple devices. A memory includes one or more of various hardware devices for volatile and non-volatile storage, and can include both read-only and writable memory. For example, a memory can comprise random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard drives, floppy disks, CDs, DVDs, magnetic storage devices, tape drives, and so forth. A memory is not a propagating signal divorced from underlying hardware; a memory is thus non-transitory. Memory 850 can include program memory 860 that stores programs and software, such as an operating system 862, building height calculation system 864, and other application programs 866. Memory 850 can also include data memory 870, LIDAR data, structure data, image data, footprint data, rooftop data, device data, satellite data, machine learning data, vector data, shadow mask data, edge detection data, terrain data, camera data, retrieval data, management data, notification data, configuration data, settings, user options or preferences, etc., which can be provided to the program memory 860 or any element of the device 800.
Some implementations can be operational with numerous other computing system environments or configurations. Examples of computing systems, environments, and/or configurations that may be suitable for use with the technology include, but are not limited to, personal computers, server computers, handheld or laptop devices, cellular telephones, wearable electronics, gaming consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, or the like.
In some implementations, server 910 can be an edge server which receives client requests and coordinates fulfillment of those requests through other servers, such as servers 920A-C. Server computing devices 910 and 920 can comprise computing systems, such as device 800. Though each server computing device 910 and 920 is displayed logically as a single server, server computing devices can each be a distributed computing environment encompassing multiple computing devices located at the same or at geographically disparate physical locations. In some implementations, each server 920 corresponds to a group of servers.
Client computing devices 905 and server computing devices 910 and 920 can each act as a server or client to other server/client devices. Server 910 can connect to a database 915. Servers 920A-C can each connect to a corresponding database 925A-C. As discussed above, each server 920 can correspond to a group of servers, and each of these servers can share a database or can have their own database. Databases 915 and 925 can warehouse (e.g. store) information such as implement data, LIDAR data, structure data, image data, footprint data, rooftop data, device data, satellite data, camera data, machine learning data, vector data, shadow mask data, edge detection data, and terrain data. Though databases 915 and 925 are displayed logically as single units, databases 915 and 925 can each be a distributed computing environment encompassing multiple computing devices, can be located within their corresponding server, or can be located at the same or at geographically disparate physical locations.
Network 930 can be a local area network (LAN) or a wide area network (WAN), but can also be other wired or wireless networks. Network 930 may be the Internet or some other public or private network. Client computing devices 905 can be connected to network 930 through a network interface, such as by wired or wireless communication. While the connections between server 910 and servers 920 are shown as separate connections, these connections can be any kind of local, wide area, wired, or wireless network, including network 930 or a separate public or private network.
General software 1020 can include various applications including an operating system 1022, local programs 1024, and a basic input output system (BIOS) 1026. Specialized components 1040 can be subcomponents of a general software application 1020, such as local programs 1024. Specialized components 1040 can include edge detection module 1044, shadow mask module 1046, height calculation module 1048, machine learning module 1050, and components which can be used for providing user interfaces, transferring data, and controlling the specialized components, such as interfaces 1042. In some implementations, components 1000 can be in a computing system that is distributed across multiple computing devices or can be an interface to a server-based application executing one or more of specialized components 1040. Although depicted as separate components, specialized components 1040 may be logical or other nonphysical differentiations of functions and/or may be submodules or code-blocks of one or more applications.
In some embodiments, the edge detection module 1044 is configured to execute an edge detection algorithm on the image to identify the edges (e.g., contrast changes) of the building in the image. In some cases, the edges can provide a rooftop vector (e.g., outline of the rooftop of the building) on the building. The edge detection module 1044 can export the edges to a raster file. In some embodiments, the shadow mask module 1046 is configured to create a shadow mask for buildings using the footprint as a reference and the satellite/aerial imagery. The shadow mask module 1046 measures the building heights based on the camera angle, distance from camera to building and the angle of the sun (e.g., using the latitude, longitude and time of day). In some embodiments, the height calculation module 1048 is configured to calculate the building height based on a camera angle, a distance from the camera to the one or more structures, and a pixel offset from the footprint vector to the rooftop vector.
In some embodiments, the machine learning module 1050 is configured to analyze the input data (e.g., image meta data) from the cameras and determine the rooftop vector and the footprint vector of the building. The machine learning module 105 may be configured to determine a footprint vector and/or a rooftop vector based on at least one machine-learning algorithm trained on at least one dataset of rooftop and/or footprint vectors. The at least one machine-learning algorithms (and models) may be stored locally at databases and/or externally at databases. Height calculation devices may be equipped to access these machine learning algorithms and intelligently determine rooftop or footprint vectors based on at least one machine-learning model that is trained on a dataset of building footprint vectors and rooftop vectors. As described herein, a machine-learning (ML) model may refer to a predictive or statistical utility or program that may be used to determine a probability distribution over one or more-character sequences, classes, objects, result sets or events, and/or to predict a response value from one or more predictors. A model may be based on, or incorporate, one or more rule sets, machine learning, a neural network, or the like. In examples, the ML models may be located on the client device, service device, a network appliance (e.g., a firewall, a router, etc.), or some combination thereof. The ML models may process building height databases and other data stores to determine a building footprint or rooftop vector.
Based on building height data and image data from building height databases and platforms and other user data stores, at least one ML model may be trained and subsequently deployed to automatically determine rooftop vectors and footprint vectors and calculate a building height. The trained ML model may be deployed to one or more devices. As a specific example, an instance of a trained ML model may be deployed to a server device and to a client device which communicate with a camera. The ML model deployed to a server device may be configured to be used by the client device when, for example, the client device is connected to the Internet. Conversely, the ML model deployed to a client device may be configured to be used by the client device when, for example, the client device is not connected to the Internet. In some instances, a client device may not be connected to the Internet but still configured to receive satellite signals with item information, such as specific image or building information. In such examples, the ML model may be locally cached by the client device.
Those skilled in the art will appreciate that the components illustrated in
Several implementations of the disclosed technology are described above in reference to the figures. The computing devices on which the described technology may be implemented can include one or more central processing units, memory, input devices (e.g., keyboard and pointing devices), output devices (e.g., display devices), storage devices (e.g., disk drives), and network devices (e.g., network interfaces). The memory and storage devices are computer-readable storage media that can store instructions that implement at least portions of the described technology. In addition, the data structures and message structures can be stored or transmitted via a data transmission medium, such as a signal on a communications link. Various communications links can be used, such as the Internet, a local area network, a wide area network, or a point-to-point dial-up connection. Thus, computer-readable media can comprise computer-readable storage media (e.g., “non-transitory” media) and computer-readable transmission media.
Reference in this specification to “implementations” (e.g. “some implementations,” “various implementations,” “one implementation,” “an implementation,” etc.) means that a particular feature, structure, or characteristic described in connection with the implementation is included in at least one implementation of the disclosure. The appearances of these phrases in various places in the specification are not necessarily all referring to the same implementation, nor are separate or alternative implementations mutually exclusive of other implementations. Moreover, various features are described which may be exhibited by some implementations and not by others. Similarly, various requirements are described which may be requirements for some implementations but not for other implementations.
As used herein, being above a threshold means that a value for an item under comparison is above a specified other value, that an item under comparison is among a certain specified number of items with the largest value, or that an item under comparison has a value within a specified top percentage value. As used herein, being below a threshold means that a value for an item under comparison is below a specified other value, that an item under comparison is among a certain specified number of items with the smallest value, or that an item under comparison has a value within a specified bottom percentage value. As used herein, being within a threshold means that a value for an item under comparison is between two specified other values, that an item under comparison is among a middle-specified number of items, or that an item under comparison has a value within a middle-specified percentage range. Relative terms, such as high or unimportant, when not otherwise defined, can be understood as assigning a value and determining how that value compares to an established threshold. For example, the phrase “selecting a fast connection” can be understood to mean selecting a connection that has a value assigned corresponding to its connection speed that is above a threshold.
Unless explicitly excluded, the use of the singular to describe a component, structure, or operation does not exclude the use of plural such components, structures, or operations. As used herein, the word “or” refers to any possible permutation of a set of items. For example, the phrase “A, B, or C” refers to at least one of A, B, C, or any combination thereof, such as any of: A; B; C; A and B; A and C; B and C; A, B, and C; or multiple of any item such as A and A; B, B, and C; A, A, B, C, and C; etc.
As used herein, the expression “at least one of A, B, and C” is intended to cover all permutations of A, B and C. For example, that expression covers the presentation of at least one A, the presentation of at least one B, the presentation of at least one C, the presentation of at least one A and at least one B, the presentation of at least one A and at least one C, the presentation of at least one B and at least one C, and the presentation of at least one A and at least one B and at least one C.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Specific embodiments and implementations have been described herein for purposes of illustration, but various modifications can be made without deviating from the scope of the embodiments and implementations. The specific features and acts described above are disclosed as example forms of implementing the claims that follow. Accordingly, the embodiments and implementations are not limited except as by the appended claims.
Any patents, patent applications, and other references noted above are incorporated herein by reference. Aspects can be modified, if necessary, to employ the systems, functions, and concepts of the various references described above to provide yet further implementations. If statements or subject matter in a document incorporated by reference conflicts with statements or subject matter of this application, then this application shall control.
