Patent Application
20100235356
This Background is intended to provide the basic context of this patent application and it is not intended to describe a specific problem to be solved.
Queries to find objects in photo or illustrations can return a wide variety of results. Often, the results are somewhat related but are not exactly what the user seeks. A user often has to sort through photos and illustrations manually to locate the desire object in the desired size at the desired resolution. Related, the storage of photos of objects is just as jumbled as what may seem related by a title is not related by the content of the photo or illustration.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
A method of organizing sensor data in a database is disclosed. In some examples, the sensor may be an a photo, a radar reading or audio measurement. If an object is detected in the measurement then it may be represented, otherwise the whole measurement may be used as “the object”. A measurement of an object from which data is collected may be determined or captured along with the measurement. The “image” is a general definition to describe captured data. It may be an image, a radar, a LIDAR scan, a depth camera capture, a sonar image, etc. A scale of the object may be determined by the magnitude of the object in comparison to a magnitude of surrounding objects such as the total magnitude of the illustration. An appropriate container size for the object may be determined by searching for a container size with shape and scale similar to the shape and scale of the object. The object may be stored in a database along with the appropriate container size and the scale being attributes. Queries to the database may be entertained using the container shape and/or the scale as the attribute to be searched.
Although the following text sets forth a detailed description of numerous different embodiments, it should be understood that the legal scope of the description is defined by the words of the claims set forth at the end of this patent. The detailed description is to be construed as exemplary only and does not describe every possible embodiment since describing every possible embodiment would be impractical, if not impossible. Numerous alternative embodiments could be implemented, using either current technology or technology developed after the filing date of this patent, which would still fall within the scope of the claims.
It should also be understood that, unless a term is expressly defined in this patent using the sentence “As used herein, the term ‘______ ’ is hereby defined to mean . . . ” or a similar sentence, there is no intent to limit the meaning of that term, either expressly or by implication, beyond its plain or ordinary meaning, and such term should not be interpreted to be limited in scope based on any statement made in any section of this patent (other than the language of the claims). To the extent that any term recited in the claims at the end of this patent is referred to in this patent in a manner consistent with a single meaning, that is done for sake of clarity only so as to not confuse the reader, and it is not intended that such claim term by limited, by implication or otherwise, to that single meaning. Finally, unless a claim element is defined by reciting the word “means” and a function without the recital of any structure, it is not intended that the scope of any claim element be interpreted based on the application of 35 U.S.C. § 112, sixth paragraph.
With reference to
The computer 110 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180, via a local area network (LAN) 171 and/or a wide area network (WAN) 173 via a modem 172 or other network interface 170.
Computer 110 typically includes a variety of computer readable media that may be any available media that may be accessed by computer 110 and includes both volatile and nonvolatile media, removable and non-removable media. The system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 131 and random access memory (RAM) 132. The ROM may include a basic input/output system 133 (BIOS). RAM 132 typically contains data and/or program modules that include operating system 134, application programs 135, other program modules 136, and program data 137. The computer 110 may also include other removable/non-removable, volatile/nonvolatile computer storage media such as a hard disk drive 141 a magnetic disk drive 151 that reads from or writes to a magnetic disk 152, and an optical disk drive 155 that reads from or writes to an optical disk 156. The hard disk drive 141, 151, and 155 may interface with system bus 121 via interfaces 140, 150.
A user may enter commands and information into the computer 20 through input devices such as a keyboard 162 and pointing device 161, commonly referred to as a mouse, trackball or touch pad. Other input devices (not illustrated) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 120 through a user input interface 160 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor 191 or other type of display device may also be connected to the system bus 121 via an interface, such as a video interface 190. In addition to the monitor, computers may also include other peripheral output devices such as speakers 197 and printer 196, which may be connected through an output peripheral interface 190.
At block 200, sensor data is captured. The sensor data may be a focal length, a GPS position of a camera, a sound pressure reading, an altitude, etc, The sensor data may provide a general area related to the measurement taken. For example, for a photo, the sensor data may provide a general location of the photograph. For a sound reading, the sensor data may be an initial sound pressure reading. For sensors that contain multiple measurements (such as cameras that comprise of many separate pixels, audio that comprises of many different time samples, LIDAR data that comprises of many different laser directions) one actual measurement can be broken into several meaningful parts. Of course, other sensor data is possible and is contemplated.
At block 210, a spatial extent of the measurement is estimated. In a database, there may be many measurements of the environment illustrated in
The description of the shape may be induced by the reference query space. Appropriate shape primitive to describe parts of this space may be used in the system. For example, when organizing photographs from an outdoor trip, the space may comprise of a two dimensional map, parallel to the ground. Shapes in this space may be two dimensional rectangles or other types of polygons. When organizing photos of a rock climbing competition, the map may be the two-dimensional plane parallel to the wall. When organizing astronomical measurements, the domain may be a representation of outer space which may be three-dimensional. When the measurement is a temperature, the shape can be a one dimensional interval. Here, the shape is also referred as ‘footprint’.
In some embodiments, if parts are detected in the measurements, these are analyzed as well. When parts are identified inside the measurement (such as elements inside a photograph) these parts may be treated as independent measurements. In such embodiment, pats of a measurement may be treated as independent or derived entities in the system. In such embodiment, a measurement is taken, objects within the measurement are recognized and analyzed. Their spatial extent is then measured or estimated, and are stored in the system. In the following, we refer both to complete measurements and to sub parts of it as ‘object’.
At block 220, a measurement of an object 305 from which data is collected may be determined. As mentioned previously, the object 305 may be an item in a photo such as in
In one embodiment, the measurement is a footprint measurement or size of the object.
In another embodiment, the coordinates of latitude and longitudinal lengths of each object 305 are calculated and a bounding box is created where the bounding box has the minimum and maximum longitude and a minimum and maximum latitude. The latitude and longitude may be determined using a LIDAR device, a LIDAR camera or from known latitude and longitude coordinates. The resulting bounding boxes may then be associated with an object 305 and the bounding box and object 305 may be stored in the database. Of course, other manners and methods of creating a footprint 410 are possible and are contemplated.
In one embodiment, the measurement is a footprint measurement or size of the image. An image is retrieved if the search point of interest falls within it's foot print (that is the object is visible in the image). The relative position of the object in the footprint determines the distance of the object from the camera, and may be used to estimate the object size in the image (thus, it's relevance for this query). The foot print can be calculated using the image parameters (camera's position, orientation and internal parameters such as the focal length or view angle), and some representation of the scene geometry to estimate the visibility. The geometry may be given by LIDAR scanning, stereo reconstruction, existing 3D models (such as Virtual Earth 3D), a digital terrain model, or just by approximating the scene by some simple geometry, such as a ground plane.
At block 230, a scale of the object 305 may be determined. The scale may be determined in several ways. In one embodiment, a scale is created by determining a magnitude of the object 305 in comparison to a magnitude of surrounding objects 310320. For example in
In another embodiment, the scale of the object 305 is determined by comparing the magnitude of the object 305 with the magnitude of the photo 300. In this way, the percentage of the photo 300 that is devoted to the object 305 may be determined. For example, the flower 320 may be 1% of the photo 300 while the flower 310 may be 10% of the photo 300.
In yet another embodiment, the measurements from block 220 are used to determine the area of the object 305 in comparison to the area of the photo 300. For example, the base of the object 305 (the Space Needle) is known to be 100 feet and the base takes up ten percent of the horizontal distance across the photo 300, the entire photo 300 length may be estimated as being 1,000 feet (100 feet/10%).
In another embodiment, objects 305 are automatically recognizing and the measurement and/or location of the recognized objects 305 may be used estimate the scale of the objects 305. For example, a databases of photos with pre-identified objects 305, including the size and location of the objects 305, may be used to identify and estimate the location of the objects 305 in front of the camera. One such application is Virtual Earth™ from Microsoft®. Of course, other methods and approaches to determining the scale are possible and are contemplated.
At block 240, an appropriate container size may be determined for the object 305. The determination may comprise searching for a container size with a scale similar to the scale of the object 305. For example, some containers may contain photos where the object 305 is less than 5% of the photo. Some container may contain photos where the object 305 is more than 5% of the photo but less than 25% of the photo. Yet another set may contain objects 305 that are more than 25% but less than 50% of the photo. Finally, another container may contain photos where the object 305 is more than 50% of the photo. As can be imagined, this additional attribute of scale may be of great benefit when searching for appropriate photos.
At block 250, the object 305 may be stored in a database with the appropriate container size and the scale being attributes. Other attributes also may be added to the database. For example, an additional attribute may be a description of the object 305 in the photo. In this way a search for “Space Needle” and “scale>50%” would likely result in a small number of very targeted photos.
In another embodiment, the description is used to determine a classification for the object 305. For example, the Space Needle may be classified as a “Building with a view,” a “Restaurant,” “Open to the public” but would not be classified as “Golf Course.” In this way, if the name of the restaurant is forgotten, a search for “restaurant” and “scale<25%” would return more targeted results.
Another attribute that may be useful to add to the database is a view direction attribute. For example, a search may be created for the object 320 Mount Rainier. Viewing the object Mount Rainier 320 from Seattle is different than viewing the object 320 Mount Rainier 320 from Portland. By adding a view direction, such as “looking east”, “from the west”, etc., an even better match may be made in searching for a photo.
It also may be useful to add an attribute regarding whether the object 305 is visible in the photo. While using a two dimensional model, in dense cities, some objects 305 may be not be seen from a photo from certain angles. However, a two dimensional outline may indicate that the object 305 would be visible. By marking whether the object 305 is truly visible in the photo, better results may be created.
In some embodiments, the object 305 may have a scale, a footprint, a classification, a description and a direction. These attributes (scale, a footprint, a classification, a description and a direction) may be stored as metadata to the object 305 or as attributes in a database.
At block 260, queries to the database for an object 305 may be permitted using the container size or the scale as the attribute to be searched. Other attributes also may be used to refine the object 305 search such as description, classification, matching polygons, matching bounding boxes, etc.
In another embodiment, a query may be expressed as a rectangle.
In action, the results of the attribute of scale may result in better query results. Better query results saves processor time, user time, memory, electricity, reduces user frustration and increases user satisfaction. In conclusion, the detailed description is to be construed as exemplary only and does not describe every possible embodiment since describing every possible embodiment would be impractical, if not impossible. Numerous alternative embodiments could be implemented, using either current technology or technology developed after the filing date of this patent, which would still fall within the scope of the claims.
