This application is related to co-pending and jointly-owned U.S. patent application Ser. No. 11/415,960, for “Coverage Mask Generation For Large Images,” filed May 2, 2006, and Ser. No. 11/437,553, for “Large-Scale Image Processing Using Mass Parallelization Techniques,” filed May 19, 2006. Each of these applications is incorporated by reference herein in its entirety.
The subject matter of this application is generally related to image processing.
Dramatic improvements in computer processing power and broadband streaming technology have lead to the development of interactive systems for navigating imagery (e.g., map imagery). Some interactive map navigation systems provide a user interface (UI) with navigation controls for dynamically navigating cities, neighborhoods and other terrain in three dimensions. The navigation controls enable users to tilt, pan, rotate, zoom and activate terrain and buildings for different perspectives at a point of interest. An example of an interactive 3D map system for navigating Earth imagery is Google Earth™ developed by Google Inc. (Mountain View, Calif.).
The production imagery used by interactive map navigation systems is typically derived by processing large pieces of geo-located imagery or “assets,” which can be taken from a single pass of a satellite or stitched together from multiple aerial photos. Once the imagery is processed it can be moved to datacenters where it can be distributed to client devices.
As a user manipulates the navigation controls of an interactive map system hosted on a client device, the map system will periodically request new imagery, terrain and/or vector data (e.g., gas stations, restaurants, points of interest, etc.) from the datacenters. Prior to making these requests, the map system needs to know if such data is available at the point of interest for the requested level of detail. For example, if a user zooms into a location on the Earth to see more detail, the map system needs to know if data is available, so that such data can be requested from the datacenter. Requesting data from a datacenter when no data is available, or when the data is already available to the mapping system, consumes bandwidth and can be wasteful.
The disclosed implementations are related to hierarchical spatial data structures with 3D data versioning for generating packet data. The packet data can be used by a client application to determine the availability of data at certain levels of a hierarchical spatial data structure. A 3D index table storing data version information can be used to support data updates and to ensure that a consistent view of data is provided to clients without suffering interruptions in service.
In some implementations, a method includes: determining the availability of a data type at one or more levels of a hierarchical spatial data structure; generating information indicating the availability of the data type at the one or more levels of the hierarchical spatial data structure; and generating a packet that includes the information.
In some implementations, a method includes: receiving imagery; dividing the imagery into tiles, where each tile is associated with tile data; generating a hierarchical spatial data structure for organizing the tile data; and generating information for signaling the availability of a data type at a level in the hierarchical spatial data structure.
In some implementations, a method includes: receiving a data packet including information indicating the availability of one or more data types at one or more levels of a hierarchical spatial data structure; receiving a navigation command generated by a user interacting with a navigation control of an interactive mapping system; and requesting data from a data source in response to the navigation command using the information received in the data packet.
In some implementations, a method includes: updating a plurality of data sources with new data; creating data packets associated with the new data, the data packets including data version identifiers; serving the data packets from at least some of the data sources with an old epoch number; and after all the data sources are serving data packets, updating the epoch number, and serving the data packets with the updated epoch number.
Other implementations of hierarchical spatial data structures and 3D data versioning are disclosed, including but not limited to implementations directed to systems, methods, apparatuses, user interfaces and computer-readable mediums.
In some implementations, the UI 100 is generated and presented by a user device. A client application running on the user device can communicate with one or more datacenters over a network (e.g., the Internet, intranet, wireless network, etc.) to retrieve imagery and associated metadata from one or more server systems in the datacenters. Such a client/server architecture reduces the amount of data that a user device stores locally to operate the interactive 3D map system. The imagery provided by datacenters can be generated from raw satellite imagery and other information (e.g., terrain data, vector data, etc.) which is processed before being served to user devices, as described with respect to
In some implementations, the blending process 204 orders and blends together processed images generated by the ingestion process 202. The blended image products can be made available to datacenters 210 through a file system 206 and a delivery channel 208. The preproduction phase can be implemented using mass parallelization techniques, as described with respect to
In the production phase, one or more datacenters 210 retrieve the image products from the file system 206 and deliver the image products to user devices 212 through a network 214 (e.g., Internet, intranet, Ethernet, wireless network, etc.). The image products can include imagery and associated meta-data for one or more locations on the Earth.
User devices 212 can be any electronic device capable of displaying a map, including but not limited to: personal computers (portable or desktop), mobile phones, smart phones, personal digital assistants (PDAs), game consoles, high definition televisions, set-top boxes, navigation systems (e.g., global positioning system (GPS)), avionics displays, etc. The system 200 is exemplary and other configurations and arrangements for image processing and delivery are possible. For example, the ingestion and blending processes could be performed in the datacenters. Also, the tile imagery and meta-data could be provided to the datacenters by different sources.
Large pieces of geo-located imagery are taken from a single pass of a satellite or are stitched together from multiple aerial photos. These raw images or “assets” can be received from one or more sources and can have a variety of orientations. The assets can be re-projected 302 into a suitable coordinate system for the map system (e.g., a geospatial coordinate system) and stored in one or more data structures 312 (e.g., database table). In some implementations, the re-projected assets are divided 304 into tiles which are processed independently in a parallel processing infrastructure. Ideally, tiles are stored so tiles that include imagery for geographic locations that are close to each other have a high probability of being stored on the same machine or in the same machine cluster to reduce the overhead associated with accessing information located on multiple machines. To achieve this ideal condition, the tiles can be sized to fall within the storage constraints of the machines or a cluster of machines. The assets can be divided into any desired shape. A tile shape, however, typically requires less computational and/or representational overhead during processing.
The quadtree data structure 400 is particularly well-suited for storing imagery and associated metadata. In the example shown, the root R of the quadtree data structure 400 can be mapped to the tile 402. The tile 402 can be generated as described in reference to
In the description that follows, the nodes of a quadtree will be referred to as quadnodes. Additionally, a quadnode plus J levels of its descendents will be referred to as quadsets. The number of quadnodes N in a quadset is defined by
The number of levels J is an implementation choice and can be any desired value. For example, if J=3 then a single quadset will have 85 nodes. As will be described later, quadsets are used to improve efficiency of communication of data to a client.
In some implementations, tile imagery and metadata are associated with quadnodes of a quadtree data structure. The locations of the files that store the data for each quadnode can be stored in the index table 500, as shown in
In the example shown, the index table 500 can include a row for each of N quadnodes in the quadtree data structure. Each row can include data for several epochs, as described in reference to
In some implementations, clients periodically request data from datacenters or other data sources while the user navigates imagery. For example, a user may zoom-in on a particular point of interest on the Earth, which can result in a request to a datacenter server for more detailed data for the targeted zoom level. The datacenter can respond by serving the requested data to the client. In some implementations, data is served in quadtree packets, which can include data for one or more quadsets. In some implementations, a quadtree packet can contain version numbers for imagery, terrain data and vector data, if such data is available. For nodes at the bottom level of a quadset, “child” presence information (e.g., a flag or other indicator) can be included to indicate whether any data is present at the quadset below a given node in the quadtree (hereinafter also referred to as a “child quadset”). The quadree packet can also include header information that indicates the size of the packet and any other desired data (e.g., cyclic redundancy check (CRC) information).
Unlike imagery and terrain data, there are potentially hundreds of “layers” of vector data that need to be transferred to client devices. Examples include road data, restaurants, parks, etc. In some implementations, each vector data can be one layer of data that is associated with a version number.
Quadsets can be used to improve efficiency when communicating data from datacenters to clients, since a quadset includes data for J levels of descendents of a quadnode. If the user continues to zoom-in on the point of interest, the new data for the new resolution level can be retrieved from local cache at the user device because it was previously delivered to the client in a quadtree packet. If quadtree packets were to include data for a single quadnode, then the same zooming action could require multiple quadtree packet transactions with one or more datacenters, resulting in diminished system performance.
In a large-scale mass parallelization infrastructure, data can be mapped and collected in a preproduction center using mapping and data reduction processes 502. Exemplary mapping and data reduction processes are described in Dean, Jeffrey, et al., “MapReduce: Simplified Data Processing on Large Clusters,” Symposium on Operating System Design (OSDI), Dec. 6-8, 2004, San Francisco, Calif., which article is incorporated by reference in its entirety.
In the example shown in
While the description above is in reference to quadtree data structures, other hierarchical spatial data structures can be used in the disclosed implementations, such as octrees, k-d-trees, b-trees, by-trees and BSP-trees.
For each quadnode in the index table, the data version associated with the quadnode is sent to a collector process corresponding to the quadnode (602). The collector process can be, for example, a “worker” process, as described in Dean, Jeffrey, et al., “MapReduce: Simplified Data Processing on Large Clusters,” Symposium on Operating System Design (OSDI), Dec. 6-8, 2004, San Francisco, Calif. Additionally, presence information can be sent to the collector process for every “parent” quadnode (604). This process can be described in pseudocode as follows:
The presence information indicates the availability of data at the descendent quadnodes below the parent quadnodes in the quadtree data structure. Presence information can be implemented using a variety of data types including bits, flags, strings, etc. Table I below illustrates one example of a data structure for transmitting presence information in a quadtree packet for a quadset having 2 levels (i.e., J=2). The data structure can be implemented, for example, as one or more bit fields in a header or payload portion of a quadtree packet.
Referring to Table I, a “1” indicates the presence of data and a “0” indicates the absence of data. At level 1, the presence information indicates that imagery, terrain and vector data are available. At level 2, the presence information indicates that imagery and terrain data are available. At level 3, the presence information indicates that only imagery data is available.
Other data structures for presence information are possible. For example, the presence information can be data version numbers rather than a bit. Representing presence information with data version numbers can be useful in systems where the client requesting the quadtree packet uses the version number in its request for data. In addition to version numbers, nodes at the bottom level of a quadset can be associated with “child” presence information that indicate whether there is any data below a given node in the quadtree. Clients can use “child” presence information to determine whether they should request data below a given node in the quadtree. For example, if the user zooms in closer on a point of interest while navigating, the client can use the “child” presence information to determine whether data is available at the level of detail requested by the user. If data is available, the client can request the data from a datacenter. The “child” presence information can be represented in a variety of formats, including one or more bits, variables, keys, logical operators, strings, etc.
Referring again to
Imagery and metadata can be read from the preproduction center and delivered to one or more datacenters, where it can be used to populate an index table (e.g., the row name and filename for each image). At each datacenter, data mapping and reduction processes can be used to create quadtree packets (including presence information) in a quadtree table that can be served to clients, as described in reference to
In some implementations, the presence information is generated by the “mapping” and “reduce” phases of the process 600. In the “mapping” phase, each quadnode N in the index table sends version numbers for its data (e.g., imagery, terrain and vector data) to a worker process assigned to the quadset containing N. Additionally, during the “mapping” phase, each node N in the index table sends a “child” indicator to a worker process assigned to each quadset above N in the quadtree. In the “reduce” phase, a worker process for a quadset S combines the version numbers and child flags for each node in S into a quadtree packet, and writes the packet into the quadtree table at row S. If necessary, an empty quadnode can be created in the quadset that includes information for indicating the presence of data further down the quadtree.
In some implementations, a cache system can be used to support data updates and present a consistent view of the data to a user without suffering interruptions. In such implementations, version numbers can be assigned to the data, which can be transmitted in quadtree packets separately from actual imagery, terrain and vector data. Also, at least two versions of data (e.g., an “old” and “new” version) can be served to different clients.
In such implementations, there is a possibility that old and new data will be served to a client from different datacenters that are at different stages of the update process, which could result in the display of artifacts at old/new data boundaries. If during an update a client navigates into an area on Earth for which its cache does not have data, then the client can request data for that area. When the data is received, some clients may tag the data with an old version number. For example, a client may tag the new data, with a version number associated with old data stored in cache. To avoid a client tagging old data with a new version number (essentially poisoning its cache), a special data loading process 800 can be performed, as described in reference to
The process 800 is similar to the preproduction process described in reference to
Various modifications may be made to the disclosed implementations and still be within the scope of the following claims.
Number | Name | Date | Kind |
---|---|---|---|
2818211 | Macklem | Dec 1957 | A |
5030117 | Delorme | Jul 1991 | A |
5581637 | Cass et al. | Dec 1996 | A |
5652717 | Miller et al. | Jul 1997 | A |
5778092 | MacLeod et al. | Jul 1998 | A |
5796868 | Dutta-Choudhury | Aug 1998 | A |
5987189 | Schmucker et al. | Nov 1999 | A |
6005978 | Garakani | Dec 1999 | A |
6075567 | Ohnishi | Jun 2000 | A |
6075905 | Herman et al. | Jun 2000 | A |
6188804 | Weldy et al. | Feb 2001 | B1 |
6313837 | Assa et al. | Nov 2001 | B1 |
6326965 | Castelli et al. | Dec 2001 | B1 |
6359617 | Xiong | Mar 2002 | B1 |
6363392 | Halstead et al. | Mar 2002 | B1 |
6434265 | Xiong et al. | Aug 2002 | B1 |
6453233 | Kato | Sep 2002 | B1 |
6466210 | Carlsen et al. | Oct 2002 | B1 |
6470265 | Tanaka | Oct 2002 | B1 |
6470344 | Kothuri et al. | Oct 2002 | B1 |
6493021 | Rouge et al. | Dec 2002 | B1 |
6526176 | Kovacevic et al. | Feb 2003 | B1 |
6591004 | VanEssen et al. | Jul 2003 | B1 |
6625611 | Teig et al. | Sep 2003 | B1 |
6636215 | Greene | Oct 2003 | B1 |
6646639 | Greene et al. | Nov 2003 | B1 |
6684219 | Shaw et al. | Jan 2004 | B1 |
6694064 | Benkelman | Feb 2004 | B1 |
6720997 | Horie et al. | Apr 2004 | B1 |
6732120 | Du | May 2004 | B1 |
6735348 | Dial, Jr. et al. | May 2004 | B2 |
6757445 | Knopp | Jun 2004 | B1 |
6766248 | Miyahara | Jul 2004 | B2 |
6768487 | Greene | Jul 2004 | B1 |
6842638 | Suri et al. | Jan 2005 | B1 |
6882853 | Meyers | Apr 2005 | B2 |
6917711 | Wang et al. | Jul 2005 | B1 |
6985903 | Biacs | Jan 2006 | B2 |
6985929 | Wilson et al. | Jan 2006 | B1 |
7006110 | Crisu et al. | Feb 2006 | B2 |
7023437 | Voorhies et al. | Apr 2006 | B1 |
7103237 | Labelle | Sep 2006 | B2 |
7138998 | Forest et al. | Nov 2006 | B2 |
7190839 | Feather et al. | Mar 2007 | B1 |
7248965 | Tanizaki et al. | Jul 2007 | B2 |
7298733 | Sakai et al. | Nov 2007 | B2 |
7298869 | Abernathy | Nov 2007 | B1 |
7439969 | Chithambaram et al. | Oct 2008 | B2 |
7483025 | Roy et al. | Jan 2009 | B2 |
7490084 | Kothuri et al. | Feb 2009 | B2 |
7519603 | Parker | Apr 2009 | B2 |
7552008 | Newstrom et al. | Jun 2009 | B2 |
7561156 | Levanon et al. | Jul 2009 | B2 |
7689621 | Huber et al. | Mar 2010 | B1 |
8018458 | Peterson | Sep 2011 | B2 |
8400451 | Peterson | Mar 2013 | B2 |
20010039487 | Hammersley et al. | Nov 2001 | A1 |
20020101438 | Ham et al. | Aug 2002 | A1 |
20020141640 | Kraft | Oct 2002 | A1 |
20020163582 | Gruber et al. | Nov 2002 | A1 |
20030007078 | Feldis | Jan 2003 | A1 |
20030040970 | Miller | Feb 2003 | A1 |
20030114173 | Carroll | Jun 2003 | A1 |
20030179301 | Feldis et al. | Sep 2003 | A1 |
20030220981 | Nakamura et al. | Nov 2003 | A1 |
20040021770 | Krill | Feb 2004 | A1 |
20040057633 | Mai et al. | Mar 2004 | A1 |
20040081355 | Takahashi | Apr 2004 | A1 |
20040095343 | Forest et al. | May 2004 | A1 |
20040204849 | Shipley et al. | Oct 2004 | A1 |
20040234162 | Jalobeanu et al. | Nov 2004 | A1 |
20040252880 | Takizawa et al. | Dec 2004 | A1 |
20050041842 | Frakes et al. | Feb 2005 | A1 |
20050091223 | Shaw et al. | Apr 2005 | A1 |
20050109363 | Matsuoka | May 2005 | A1 |
20050265631 | Mai et al. | Dec 2005 | A1 |
20050270311 | Rasmussen et al. | Dec 2005 | A1 |
20060143202 | Parker | Jun 2006 | A1 |
20060184519 | Smartt | Aug 2006 | A1 |
20060206264 | Rasmussen | Sep 2006 | A1 |
20060222079 | Park et al. | Oct 2006 | A1 |
20060265350 | Klein | Nov 2006 | A1 |
20070182734 | Levanon et al. | Aug 2007 | A1 |
20070276970 | Werner et al. | Nov 2007 | A1 |
20090074275 | O Ruanaidh | Mar 2009 | A1 |
20120078723 | Stewart | Mar 2012 | A1 |
Entry |
---|
Samet, Hanan et al., “Hierarchical Spatial Data Structures,” Proc. of 1st Symp. on Large SpatialDatabases, Lecture Notes in Computer Science 409, Springer-Verlag, 1989, pp. 193-212. |
Wei, Zu-Kuan et al., “Efficient Spatial Data Transmission in Web-Based GIS” 1999, WIDM '99 Proceedings of the 2nd International workshop on Web information and data management, pp. 38-42. |
Samet, Hanan, “Spatial Data Structures” 1995, Modern Database Systems: The Object Model, Interoperability, and Beyond, Addison Wesley/ACM Press, Reading, MA. pp. 361-358. |
Barroso, Luiz Andre et al., “Web Search for a Planet: The Google Cluster Architecture,” IEEE Micro, IEEE Computer Society, Mar.-Apr. 2003, pp. 22-28. |
Dean, Jeffrey, et al. “MapReduce: Simplified Data Processing on Large Clusters,” Symposium on Operating System Design (OSDI), Dec. 6-8, 2004, San Francisco, California, pp. 1-13. |
Foley, James D. et al., Ch. 15, “Visible-Surface Determination,” “Computer Graphics: Principals and Practice Second Edition in C,” Addison-Wesley (2006), pp. 649-720. |
Ghemawat, Sanjay et al., “The Google File System,” Association for Computing Machinery (ACM), 19th Symposium on Operating System Principles (SOSP), Oct. 19-22, 2003, Lake George, New York, 15 pages. |
Grimm, Cindy, “Technical Report WUCS-2002-9: Creating View-dependent Texture Maps,” Nov. 22, 2000, Washington University in St. Louis, Department of Computing Science, 9 pages. |
Langley, Richard, B. “The UTM Grid System,” GPS World, Feb. 1998, pp. 46-50. |
Lawder, J.K. et al., “Using Space-filling Curves for Multi-dimensional Indexing,” School of Computer Science and Information Systems, Birkbeck College, University of London, Malet Street, London WC1E 7HX, United Kingdom, 16 pages. |
Lindstrom, Peter et al., “An Integrated Global GIS and Visual Simulation System,” Mar. 1997. Georgia Institute of Technology: Graphics, Visualization & Usability Center. Technical Report GIT-GVU-97-07, pp. 1-9. |
Mesick, H., Ioup, E., Sample, J., “A Faster Technique for the Transformation of Universal Transverse Mercator Projected Raster Images into a Geodetic Projection,” Naval Research Laboratory: Memorandum Report, Mar. 11, 2005, 16 pages. |
Skodras, A., Christopoulos, C., Ebrahimi, T., “JPEG 2000 Still Image Compression Standard,” IEEE Signal Processing Magazine, Sep. 2001, pp. 36-58. |