Patent Application
20060190812
1. Field
The following inventions disclosure is generally concerned with pointing systems used to address objects and specifically concerned with such pointing systems having an imaging function which includes providing ‘hyperlink’ type devices in combination with images.
2. Prior Art
A relatively new device provides powerful connectivity to remote information sources. Known as a ‘hyperlink’, an object such as a textual word or phrase has an underlying (sometimes hidden) network address associated therewith. Triggering the link (sometimes arranged as a “point-and-click” action), results in redirection of the medium to present the information recalled from the remote source. Of course, all users of the Internet are quite familiar with this device and it is quite well known.
While textual hyperlinks are most common, it is not necessary that a hyperlink be associated with a block of text. Indeed, hyperlink have been arranged to cooperate in conjunction with a graphical body. A ‘push button’ type object may be part of a presentation on a graphical web page. A user triggers the push button by addressing it with a ‘mouse’ pointing periphery and ‘clicking’ on the push button. The computer responds by redirecting the browser display to a new web resource which is defined by the link address which may look like this: “http://www.geovector.com/appdemos/”.
Hyperlinks are not restricted to “push button” type graphical objects. Hyperlinks are used in conjunction with “drop down” menus, “thumbnail” objects, “toolbar” objects, among others. Of particular interest, very special hyperlinks are constructed in conjunction with an “image map” object. An image map can include a digital or ‘pixelated’ image with one or more image areas which correspond to a particular subject. The image map suggests that each pixel may be a member of particular group of pixels. These groups of pixels map to certain portions of the overall image. For example,
The image may be presented in a web page presentation played in a browser computer application. As such, the browser enables special functionality relating to interaction with various parts of the image by way of an image map. In example, a hyperlink can be arranged whereby when addressed and triggered (point-and-click), the browser can be redirected to web resource which relates particularly to the group of pixels; for example a detailed web site relating specifically to the Lincoln Memorial. Thus the portion of the image depicted as an area enclosed by outline 22 can be associated with the web address: http://www.nps.gov/linc/. When viewing the image map presented as
The image map is a computer ‘object’ and it is created by a web designer who views the image and selects and defines mathematically an area of which to associate with a particular web address. Creating these images maps is highly specialized work and takes great skill and effort. The procedure is manual, time consuming, and tedious. Accordingly, there is great need for a technique and system to automatically create such devices with little or no effort.
Advanced computer pointing systems for addressing objects have been presented in several forms. Of particular interest for this disclosure are the pointing systems for addressing objects having a well defined spatial definition—one that is stable in time or otherwise of a predictable nature. For example, a building occupies a certain space and tomorrow it is very likely to occupy the identical space.
Of considerable interest are the present inventors previous disclosures presented in U.S. Pat. Nos. 6,522,292; 6,414,696; 6,396,475; 6,173,239; and 6,037,936. Each of these is directed to pointing systems which address objects in the real world. In some cases, a computer response may be initiated whereby the particular response relates to the object being addressed.
Inventions presented in U.S. Pat. No. 6,522,292 include those which rely upon positioning systems to detect the location of the system and to permit a manual input for direction references. Together this information forms a basis upon which pointing functionality may be used to control a computer in an environment which is known to the computer.
Teachings presented in U.S. Pat. No. 6,414,696, relates to non-imaging pointing systems which are responsive to a user's surrounding by way of position and attitude determinations. Information relating to objects in the environment are recalled and presented at a display interface.
A mapping system which includes highly responsive “toolbar” type user interfaces is presented in U.S. Pat. No. 6,396,475. These toolbars respond to position and attitude measurement to implicitly determine what subject matter is of interest to a user. The toolbar features are dynamic and change with changing address conditions.
Inventor Thomas Ellenby presents in U.S. Pat. No. 6,173,239 a general pointing system for addressing objects to trigger computer response; these systems are based upon pointing and attitude determinations and specialized data searches which result in computer response being taken up when objects are addressed via user pointing actions.
U.S. Patent No. 6,037,936 by inventors Ellenby, J. et al, relates to an imaging system which captures images and displays those images alongside graphical objects such as menu items, labels, controls, et cetera. These objects may be considered graphical user interface GUI objects and they are provided with information known to relate to objects detected within the image being presented simultaneous with the GUIs.
U.S. application Ser. No. 09/769,012 sets forth in considerable detail best versions of pointing systems which recall information about objects being addressed by the system. Principles presented in this document are important to the concepts further taught herein.
Each of these pointing systems provides user means of interaction with a 3-space surrounding environment by way of position and direction information which permits the computer to distinguish objects from others nearby. The computer provides information relating to the objects as they are addressed. These disclosures and each of them is hereby incorporated into this disclosure by reference.
While systems and inventions of the art are designed to achieve particular goals and objectives, some of those being no less than remarkable, these inventions have limitations which prevent their use in new ways now possible. Inventions of the art are not used and cannot be used to realize the advantages and objectives of the inventions taught herefollowing.
Comes now, Thomas, Peter, and John Ellenbyto teach new inventions of pointing image systems which include dynamic information linking including devices for and methods of connecting information stored on the Internet with image objects having a well defined spatial definition associated therewith It is a primary function of these inventions to couple pointing image system functionality with network addresses and related information connected by network addresses.
Pointing imaging systems of these inventions are used to make advanced high function digital image files. Image files produced via these systems support storage of information related to the scene being imaged. Further, very special automated image mapping function is provided. Such image mapping functions permit these images to be used at playback with point-and-click actions to link the images to particular Internet addresses. Association between objects in scenes and web address is completely automated; as is the division of image space into appropriate image maps.
Imaging systems arranged to make images and simultaneously record physical parameters relating to the image scene and the imaging device are presented. These imaging systems, sometimes herein called ‘pointing image systems’, may be used to record data about the image scene and imager at the same time an image is formed. An imager of these systems first forms an image. At the time the image is formed, the physical state of the imager, particularly with regard to its position and pointing nature, among others, is determined. These data relating to position and pointing are used in a database search to retrieve information previously stored. The database search produces information relating to the scene or objects in the scene. This information is ‘attached’ to the pixel image data and associated with the image or particular parts of the image. Such associations may be made in a special image data file with a format to support such associations.
In one version, a mobile phone including camera, location measuring capacity and compass subsystems. While forming an image of the Golden Gate bridge, the phone-imager subsystems determine that the phone is pointing North and slightly West and further than the location of the phone- imager is on the San Francisco side of the channel slightly East of the bridge landing. With this position and direction information, the system searches a database to determine that Brown's Bay Campsite is in or part of the image. As such, a specia 1 image file is created whereby pixel image data is stored along with additional information such as: the time the image was made; the city from which it was made; a list of objects in the image; among many other image related information elements.
Thus, imaging systems of these inventions include imaging systems having position and attitude determining means, a database of pre-stored information, programming to effect storage of images along with associated information.
It is a primary object of these inventions to provide advanced imaging systems.
It is an object of these inventions to provide imaging systems which store images along with associated image information.
It is a further object to provide imaging systems which store images and associated image information which depends upon the address nature of the imaging system.
It is an object of these inventions to provide imaging systems to record images and associated image information recalled from a database of prerecorded information.
A better understanding can be had with reference to detailed description of preferred embodiments and with reference to appended drawings. Embodiments presented are particular ways to realize these inventions and are not inclusive of all ways possible. Therefore, there may exist embodiments that do not deviate from the spirit and scope of this disclosure as set forth by appended claims, but do not appear here as specific examples. It will be appreciated that a great plurality of alternative versions are possible.
These and other features, aspects, and advantages of the present invention will become better understood with regard to the following description, appended claims and drawings where:
Throughout this disclosure, reference is made to some terms which may or may not be exactly defined in popular dictionaries as they are defined here. To provide a more precise disclosure, the following terms are presented with a view to clarity so that the true breadth and scope may be more readily appreciated. Although every attempt is made to be precise and thorough, it is a necessary condition that not all meanings associated with each term can be completely set forth. Accordingly, each term is intended to also include its common meaning which may be derived from general usage within the pertinent arts or by dictionary meaning. Where the presented definition is in conflict with a dictionary or arts definition, one must use the context of use and liberal discretion to arrive at an intended meaning. One will be well advised to error on the side of attaching broader meanings to terms used in order to fully appreciate the depth of the teaching and to understand all the intended variations. For purposes of this disclosure:
Pointing Imaging System—A ‘pointing imaging system’ is an imager or camera equipped with mean for measuring its pointing state or pointing attitude. In addition, sometimes these systems include position measurement and zoom state measurement sub-systems.
Geometric Descriptor—is the definition of a geometric body or geometric construct, for example a plane. Geometric descriptor as generally arranged to correspond to the space occupied by an object for example the space in which a building occupies.
Address Indicator—address indicator is a description of the pointing nature of a device. Usually an address indicator is a vector having its origin and direction specified. In some cases, an address indicator is a solid angle construct which corresponds to the field-of-view of an imager.
Solid Angle Field-of-Address—The field-of-view of an imager subtends a space having a point origin, rectangular cross section which increases proportionally with respect to the distance from the origin, and infinite extent.
View State—An imager ‘view-state’ is specified by physical parameters which define the particular viewing nature of the imager. These parameters include at least: it's position and it's pointing direction. In some cases, it also includes the zoom/magnification state, field-of-view, time, among others.
Image Map—An image map is a digital image file comprising pixel data and spatial definitions of sub-fields described as part of the image file.
Image Region—An Image Region is an image area or sub-field which is a subset or fractional portion of an entire image.
Internet Address—is a network address which specifies a network node's handle; in example a URL, or uniform resource locator, is a network address.
In accordance with each of the preferred embodiments of these inventions, there is provided apparatus for and methods of forming image map hyperlinks integrated with image data. It will be appreciated that each of these embodiments described include both an apparatus and method and that the apparatus and method of one preferred embodiment may be different than the apparatus and method of another embodiment.
Pointing imaging systems produce special digital image files having advanced features. These imaging system not only capture image pixel data but additionally capture information relating to the scene which was previously stored in a database. Further, these systems capture information relating to the state of the imaging system at the time the image was made. Still further, the information is processed together to form special image files containing information which will support image map functionality with point-and-click hyperlinks when the images are played in suitable viewers/browsers.
Camera phones, or mobile telephones having imaging systems integrated therewith, are quite popular and now nearly ubiquitous. Full- service digital cameras are also quickly replacing those cameras known to many generations which form images on the chemical film medium. Both of these electronic devices provide a good platform upon which these inventions might be installed. These inventions require an imager of the digital electronic nature. Further, these inventions incorporate with such imagers additional subsystems such as position determining means, attitude determining means, view-state determining means, computer processors and database digital storage facility.
In short, image pixel data is captured. The computer determines which objects the scene is comprised. This is done by an implicit reasoning in view of prerecorded information. In an advanced database, the geometric properties of a great plurality of objects are stored. When it is determined that an object as defined by its geometric descriptor lies in the address field of the camera/imager, then it is said to be within the scene being addressed. Only objects known to the database are subject to recall. Objects which arrive in a scene after preparation of a database will be omitted. Similarly, objects taken from the scene (for example by fire) without a database update cause an error. However, when detailed and frequently updated databases are used, the objects which make up some image scenes will be well defined and known to these systems. Certainly, landmark buildings and the ir geometric definitions will be included in the most brief databases set up for these systems.
Systems taught herein account for images made from any viewpoint. When an image is made with a pointing imaging system, the imager determines viewpoint information by measure the position and pointing direction of the imager at the time an image is captured. In addition, information such as: lens magnification power; field-of-view; time-of-day; among others, may be determined and recorded. When in Washington D.C., a tourist having a pointing imaging system may form the image of
A database prepared with recorded information is queried at the time of image pixel data capture. Thus, previously recorded information may be recalled in response to an image capture event. When the pointing nature of these imaging systems implies certain objects are being addressed, i.e. are at least partly within the imager's field-of-view, during an image capture event, information relating to those objects is recalled.
When an image is captured, geometric descriptors are converted to area descriptions for each object for the in the proper perspective with respect to the point of view from which the image was made. Thereafter, associations are made between captured pixel data and area descriptions formed from the geometric descriptors.
An ‘image map’ is a relatively new computer object or device. Computer software experts have developed a particular human interface functionality well known as “point-and-click” actions. A pointer is aligned with a space on a computer monitor and a mouse click initiates a computer response. The response depends upon where the pointer is pointing. This is nicely illustrated by the ‘toolbar objects’ used in most computer applications. While most point-and-click actions involve icons; toolbars; or drop-down menus, for example, a special point-and-click action is devised for use with images. A normal image of simple pixel data may be converted to a special high performance image object with ‘hot spots’. Hot spots are particular regions within an image which can be made responsive to ‘mouse clicks’. Generally, an ‘image map object’ is embodied as a module of computer code, i.e. a set of computer instructions in combination with image pixel data. Hot spots are defined in the computer code modules. These are distinct from the image maps of these inventions.
When an image is made in accordance with these inventions, sometimes an image map which includes the pixel data and image region definitions is formed.
Image files of these inventions are not limited to the simple image map concepts of the art. Rather, these image files contain additional information elements. For example, in addition to the pixel data and image region definitions, compound image files first presented here may also contain Internet network address information (URLs). These URLs are not merely contained in a list of network addresses, but rather they are well connected with select spatial regions in the image. An image region defined in the image map may have associated therewith a URL. A URL which is appropriate for the any specific image map region is automatically assigned and associated with the region. For example, when an imager of these inventions is addressing a scene in Washington D.C., the scene including the Lincoln Memorial, the image may form an image file by first, capturing the image, second determining which objects are in the image via a database search based upon the position, attitude and zoom state of the imager, forming image region definitions, forming associations between the URLs with those particular image regions, and constructing a data file in accordance with a predetermined scheme, and storing the compound image file with image map and network address information.
A special digital image file is thereafter prepared for storage. The image file not only contains pixel data, but in addition, also contains information associated with the image space, the imager properties, the state of the image capture event. In a first illustrative example, the Washington D.C. image is again considered. During image capture, it is determined by the computer that the image field includes the Lincoln Memorial, the Washington Monument, and the Capitol Building. Further, the geometric descriptors associated with these objects are converted to two dimensional image regions. These regions are properly aligned and associated with the image space in agreement with the image pixel data to form an image mapping system. Finally, simple label information is generated and connected with the image map system. These labels have text information which is particular to the object with which it is associated.
A better understanding is had in view of the block diagram of
Similarly, image files created via devices and methods of these inventions contain pixel image data and imager state data. Further, they contain very special information relating to certain objects in the image scene. Namely, the objects which are determined to be in the scene as a result of considering the pointing state of the imager. An image map is formed automatically with image sub- field areas corresponding to the area occupied by objects as seen from the perspective of the imager. A careful observer will notice that for any viewpoint, the perspective and shape of image area for any object will be different for another viewpoint. Thus, the image map depends upon the viewpoint. A user does not have to determine the image area occupied by an object. Rather, a three dimensional geometric descriptor associated with the object and stored in the database is converted to a two dimensional area description which approximates the area occupied by an object from the viewpoint in which the image was made. This information element is certainly not found in any image file format.
An image file 61 is comprised of pixel data 62, image region descriptions 63, Internet addresses 64. In addition, these file formats may also include other data 65, such as viewpoint data, zoom state data, resolution data, time stamp data, temperature data, author/artist data, among others.
In review, we move to the United States west coast where one finds another famous landmark the Golden Gate Bridge 72. A certain viewpoint of the bridge necessarily implies a unique perspective thereof. A three dimensional model of the bridge stored in a computer memory can be adjusted to represent any perspective when viewed on a two dimensional medium. A photographer, located below and just East of the bridge on the San Francisco side of the bay would view the bridge as shown in the image 71 of
An imager equipped with position and attitude determining means, as well as zoom and view state determining means, captures pixel data. A database search which depends upon the imager position and attitude reveals that the Golden Gate Bridge is within (at least partly) the field-of-view. A geometric descriptor, a three dimensional model representing the space occupied by the bridge is recalled. A computation is made and applied to the model such that it is converted into a two dimensional area definition, an image region, which corresponds to a portion of the image space captured as pixel data.
Because information is known about objects in an image scene via the database, it is possible that images are sorted and classified at the moment they are created. Image files therefore may include a classification tag. In example, images of landmarks may be labeled as such. Images of sunsets may be marked and identified accordingly. Images from a specific city, town center, country region, et cetera may all be properly catalogued automatically by way of marking the image file with class information recalled from the database. In this way, one records without effort, much more information about a favored scene. Such systems permit one to enjoy playback and sorting of images in a much more sophisticated way.
In our examples presented above, one might associate a ‘government buildings’ classification to the objects in Washington D.C. while attaching a ‘bridges and structures’ tag to the Golden Gate Bridge of San Francisco. A playback system could then sort the images accordingly either by structure type, or by city/state or by any of a large plurality of other sorting schemes.
Apparatus of these Inventions
Apparatus of these inventions can be better understood in view of the following. One will appreciate that new digital technology permits that small hand-held devices now easily accommodate sub-systems such as GPS and electronic compass. Thus, a digital camera or mobile phone with integrated camera imager can also support in combination therewith these advanced measurement systems which detail the physical state of the imager at any time.
Methods of these Inventions
In review, at image capture time pointing imagers of these inventions capture pixel data, determine position and attitude of the imager, recall geometric descriptors type three dimensional models of objects, converts those models to two dimensional image region definitions in proper perspective, and associates URLs, text labels, among others with these image regions to form a correspondence between image space and Internet space.
In most general terms, methods of the inventions may precisely be described as including the steps of: capturing a digital pixel image; determining imager view-state parameters; searching a database based upon view-state parameters; defining image region areas corresponding to objects recalled in database search; associating said image region areas with corresponding image space in said pixel image; and forming a compound data file comprising pixel image information and associated information relating to the scene.
Searching a database further includes recalling information which is related to objects within the field-of-view of the imager. This is done by finding geometric intersection between a geometric descriptor of a stored record and the solid angle field-of-address of the imager at the time pixel data is captured. Where stored records also include network addresses, those may also be recalled from memory and associated with appropriate image regions. Similarly, text labels may also be recalled and associated with image regions.
Image scenes may be classified via classification identifiers which also are recalled from memory in database search operations. Information element relating to the imager state includes those of the group: present time, f-stop, shutter speed, and artist/author, may also be attached to a image map data file of these systems.
One will now fully appreciate how pointing images create advanced images having associated therewith important related information elements. Further, how image map systems including hyperlink functionality is automated. Although the present inventions have been described in considerable detail with clear and concise language and with reference to certain preferred versions thereof including best modes anticipated by the inventors, other versions are possible. Therefore, the spirit and scope of the invention should not be limited by the description of the preferred versions contained therein, but rather by the claims appended hereto.
