Claims
- 1. A method of presenting a particular two-dimensional video image of a real-world three dimensional scene to a viewer comprising:
- imaging in multiple video cameras each at a different spatial location multiple two-dimensional images of a real-world scene each at a different spatial perspective;
- combining in a computer the multiple two-dimensional images of the scene into a three-dimensional model of the scene;
- receiving in a the computer from a prospective viewer of the scene a viewer-specified criterion relative to which criterion the viewer wishes to view the scene;
- synthesizing in the computer from the three-dimensional model a particular two-dimensional image of the scene in accordance with the received viewer criterion; and
- displaying in a video display the particular synthesized two-dimensional image of the real-world scene to the viewer.
- 2. The method according to claim 1
- wherein the receiving is of the viewer-specified criterion of a particular spatial perspective, relative to which particular spatial perspective the viewer wishes to view the scene; and
- wherein the synthesizing in the computer from the three-dimensional model is of a particular two-dimensional image of the scene in accordance with the particular spatial perspective criterion received from the viewer; and
- wherein the displaying in the video display is of the particular synthesized two-dimensional image of the scene that is in accordance with the particular spatial perspective received from the viewer.
- 3. The method according to claim 2
- wherein the synthesizing is of a virtual image that is without correspondence to any of the images of the scene that are imaged by any of the multiple video cameras, this synthesized virtual image being in accordance with the particular spatial perspective criterion received from the viewer.
- 4. The method according to claim 1
- wherein the combining is so as generate a three-dimensional model of the scene in which model objects in the scene are identified;
- wherein the receiving is of the viewer-specified criterion of a selected object that the viewer wishes to particularly view within the scene; and
- wherein the synthesizing in the computer from the three-dimensional model is of a particular two-dimensional image of the selected object in the scene; and
- wherein the displaying in the video display is of the particular synthesized two-dimensional image of the scene showing the viewer-selected object.
- 5. The method according to claim 4 wherein the viewer-selected object in the scene is static, and unmoving, in the scene.
- 6. The method according to claim 4 wherein the viewer-selected object in the scene is dynamic, and moving, in the scene.
- 7. The method according to claim 4 wherein the viewer selects the object that he or she wishes to particularly view in the scene by act of positioning a cursor on the video display, which cursor unambiguously specifies an object in the scene by an association between the object position and the cursor position in three dimensions and is thus a three-dimensional cursor.
- 8. The method according to claim 1
- wherein the combining is so as generate a three-dimensional model of the scene in which model events in the scene are identified;
- wherein the receiving is of the viewer-specified criterion of a selected event that the viewer wishes to particularly view within the scene; and
- wherein the synthesizing in the computer from the three-dimensional model is of a particular two-dimensional image of the selected event in the scene; and
- wherein the displaying in the video display is of the particular synthesized two-dimensional image of the scene showing the viewer-selected event.
- 9. The method according to claim 8 wherein the viewer selects the event that he or she wishes to particularly view in the scene by act of positioning a cursor on the video display, which cursor unambiguously specifies an event in the scene by an association between the event position and the cursor position in three dimensions and is thus a three-dimensional cursor.
- 10. The method according to claim 1 performed in real time as virtual television presented to a viewer interactively in accordance with the viewer-specified criterion.
- 11. A method of presenting a particular two-dimensional video image of a real-world three dimensional scene to a viewer comprising:
- imaging in multiple video cameras each at a different spatial location multiple two-dimensional images of a real-world scene each at a different spatial perspective;
- combining in a computer the multiple two-dimensional images of the scene into a three-dimensional model of the scene: receiving in a the computer from a prospective viewer of the scene a viewer-specified criterion relative to which criterion the viewer wishes to view the scene;
- synthesizing in the computer from the three-dimensional model a particular two-dimensional image of the scene that is without exact correspondence to any of the images of the real-world scene that are imaged by any of the multiple video cameras in accordance with the received viewer criterion; and
- displaying in a video display the particular synthesized two-dimensional image of the real-world scene to the viewer.
- 12. A method of synthesizing a virtual video image from real video images obtained by a multiple real video cameras, the method comprising:
- storing in a video image database the real two-dimensional video images of a scene from each of a multiplicity of real video cameras;
- creating in a computer from the multiplicity of stored two-dimensional video images a three-dimensional video database containing a three-dimensional video image of the scene; and
- generating a two-dimensional virtual video image of the scene from the three-dimensional video database.
- 13. The method according to claim 12 that, between the creating and the generating, further comprises:
- selecting a spatial perspective, which spatial perspective is not that of any of the multiplicity of real video cameras, on the scene as is imaged within the three-dimensional video database;
- wherein the generating of the two-dimensional virtual video image is so as to show the scene from the selected spatial perspective.
- 14. The method according to claim 13 wherein the selected spatial perspective is static, and fixed, during the video of the scene.
- 15. The method according to claim 13 wherein the selected spatial perspective is dynamic, and variable, during the video of the scene.
- 16. The method according to claim 13 wherein the selected spatial perspective is so dynamic and variable dependent upon occurrences in the scene.
- 17. The method according to claim 12 that, between the creating and the generating, further comprises:
- locating a selected object in the scene as is imaged within the three-dimensional video database;
- wherein the generating of the two-dimensional virtual video image is so as to best show the selected object.
- 18. The method according to claim 12 that, between the creating and the generating, further comprises:
- dynamically tracking the scene as is imaged within the three-dimensional video database in order to recognize any occurrence of a predetermined event in the scene;
- wherein the generating of the two-dimensional virtual video image is so as to best show the predetermined event.
- 19. The method according to claim 12 wherein the generating is of a selected two-dimensional virtual video image, on demand.
- 20. The method according to claim 12 wherein the generating of the selected two-dimensional virtual video image is in real time on demand, thus interactive virtual television.
- 21. A method of synthesizing a virtual video image from real video images obtained by a multiple real video cameras, the method comprising:
- storing in a video image database the real two-dimensional video images of a scene from each of a multiplicity of real video cameras;
- creating in a computer from the multiplicity of stored two-dimensional video images a three-dimensional video database containing a three-dimensional video image of the scene; and
- generating a two-dimensional virtual video image of the scene from the three-dimensional video database by selecting from the three-dimensional video database a two-dimensional virtual video image of the scene that corresponds to a real two-dimensional video image of a scene.
- 22. A system for presenting video images of a real-world scene in accordance with a predetermined criterion, the system comprising:
- multiple video imagers each at a different spatial location for producing multiple two-dimensional video images of a real-world scene each at a different spatial perspective;
- a viewer interface at which a prospective viewer of the scene may specify a criterion relative to which criterion the viewer wishes to view the scene;
- a computer, receiving the multiple two-dimensional video images of the scene from the multiple video imagers and the viewer-specified criterion from the viewer interface,
- for producing from the multiple two-dimensional video images of the scene a three-dimensional model of the scene; and
- for synthesizing from the three-dimensional model a particular two-dimensional virtual video image of the scene in accordance with the viewer-specified criterion; and
- a video display, receiving the particular two-dimensional video image of the scene from the computer, for displaying the particular two-dimensional video image of the real-world scene to the viewer.
- 23. The system according to claim 22
- wherein the viewer interface has and presents a three-dimensional cursor manipulatable by a prospective viewer of the scene so as to unambiguously specify any object in the scene even when the specified object is partially obscured by other objects in the scene.
- 24. A system for presenting video images of a real-world scene in accordance with a predetermined criterion, the system comprising:
- multiple video cameras, each having an orientation and a lens parameter and a location that is separate from all other video cameras, for producing multiple raw two-dimensional video images of a real-world scene each at a different spatial perspective;
- a camera scene builder computer, receiving the multiple raw video images from the multiple video cameras, for producing in consideration of the orientation, the lens parameter, and the location of each of the multiple video cameras, multiple two-dimensional video images of a real-world scene that are of a known spatial relationship, as well as at a different spatial perspective, one to the next;
- wherein the spatial positions of all the all the multiple two-dimensional video images of a real-world scene are known;
- a viewer interface at which a prospective viewer of the scene may specify a criterion relative to which criterion the viewer wishes to view the scene;
- a computer, receiving the multiple two-dimensional video images of the scene from the multiple video imagers and the viewer-specified criterion from the viewer interface,
- for producing from the multiple two-dimensional video images of the scene a three-dimensional model of the scene; and
- for producing from the three-dimensional model a particular two-dimensional video image of the scene in accordance with the viewer-specified criterion; and
- a video display, receiving the particular two-dimensional video image of the scene from the computer, for displaying the particular two-dimensional video image of the real-world scene to the viewer.
- 25. A method of building a three-dimensional video model of a three-dimensional real-world scene, and of extracting video information regarding the real world scene from the model built, the method comprising:
- imaging in multiple video cameras multiple frames of two-dimensional video of the three-dimensional real world scene, the two-dimensional frames from each camera arising from a unique spatial perspective on the scene;
- first-analyzing the scene in two dimensions by extracting feature points from the two-dimensional video frames in order to annotate the two-dimensional video frames by certain image information contained therein, thus producing multiple annotated two-dimensional video frames;
- second-analyzing in a computer the scene in three dimensions by
- transforming the multiple annotated two-dimensional video frames into a three-dimensional video model in which model is contained three-dimensional video of the scene, while
- extracting and correlating information from the annotated two-dimensional video frames so as to annotate the three-dimensional video model of the scene with such information, thus producing a three-dimensional video model annotated with scene image information, thus producing an annotated three dimensional video model;
- generating in a computer from the annotated three-dimensional video model (i) a two-dimensional virtual video image (ii) synthesized in accordance with some criterion interpretable and interpreted by reference to the scene image information, thus producing a synthesized virtual two-dimensional video image; and
- displaying in a display the selected two-dimensional video image;
- wherein frames from multiple video cameras were first-analyzed in order to produce the annotated two-dimensional video frames;
- wherein the annotated two-dimensional video frames were themselves second-analyzed to produce the annotated three-dimensional video model;
- wherein the interpreting, in the generating step, of the criterion by reference to the three-dimensional scene image information is thus, ultimately, an interpretation by reference to scene image information that arose from multiple video cameras;
- wherein the image displayed is selected by reference to scene image information that is arose from more than just one video camera, and, indeed, is selected by reference to scene image information that arose from multiple video cameras.
- 26. A method of building a three-dimensional video model of a three-dimensional real-world scene, and of extracting video information regarding the real world scene from the model built, the method comprising:
- imaging a three-dimensional real world scene having coordinates (x,y,z) by multiple cameras each having reference frame coordinates (p,q,s) that are different than are the camera reference frame coordinates of any other camera so as to produce multiple frames of two-dimensional video each having coordinates (p,q);
- first-analyzing the scene in two dimensions by extracting feature points from the two-dimensional video frames in order to annotate the two-dimensional video frames by certain image information contained therein, thus producing multiple annotated two-dimensional video frames, the first-analyzing serving to extract feature points of coordinates (p.sub.0,q.sub.0) from the two-dimensional video frames;
- second-analyzing in a computer the scene in three dimensions by
- transforming the multiple annotated two-dimensional video frames into a three-dimensional video model in which model is contained three-dimensional video of the scene, particularly by transforming a point (x,y,z) in the world coordinate system to a point (p,q,s) in the camera coordinate system by ##EQU3## where R is a transformation matrix from the world coordinate system to the camera coordinate system, and (x.sub.0,y.sub.0,z.sub.0) is the position of the camera, and
- by projecting a point (p,q,s) in the camera coordinate system to a point (u,v) on the image plane according by ##EQU4## where f is camera parameter that determines the degree of zoom in or zoom out;
- wherein an image coordinate (u,v) that corresponds to world coordinate (x,y,z) is determined depending on the (i) camera position, (ii) camera angle and (ii) camera parameter, while
- extracting and correlating information from the annotated two-dimensional video frames so as to annotate the three-dimensional video model of the scene with such information, thus producing a three-dimensional video model annotated with scene image information, thus producing an annotated three dimensional video model;
- generating in a computer from the annotated three-dimensional video model (i) a two-dimensional video image (ii) selected in accordance with some criterion interpretable and interpreted by reference to the scene image information, thus producing a selected two-dimensional video image; and
- displaying in a display the selected two-dimensional video image;
- wherein frames from multiple video cameras were first-analyzed in order to produce the annotated two-dimensional video frames;
- wherein the annotated two-dimensional video frames were themselves second-analyzed to produce the annotated three-dimensional video model;
- wherein the interpreting, in the generating step, of the criterion by reference to the three-dimensional scene image information is thus, ultimately, an interpretation by reference to scene image information that arose from multiple video cameras;
- wherein the image displayed is selected by reference to scene image information that is arose from more than just one video camera, and, indeed, is selected by reference to scene image information that arose from multiple video cameras.
- 27. The method according to claim 26 that, a first step, further comprises:
- calibrating each of the multiple cameras by
- observing a known point,
- knowing thereby the observed point a pair of image coordinates and corresponding world coordinates,
- applying this known pair to the equations of claim 28 so as to obtain two equations regarding the seven parameters that determine camera status,
- repeating the observing, the knowing and the applying for at least four known points so as to, the minimum equations to solve the seven unknown parameters thus being provided, solve the equations and calibrate the camera coordinate system (p,q,s) to the world coordinate system (x,y,z).
- 28. The method according to claim 27
- wherein the transforming a point (x,y,z) in the world coordinate system to a point (p,q,s) in the camera coordinate system, and the projecting of the point (p,q,s) in the camera coordinate system to a point (u,v) on the image plane, assumes, a simplifying assumption, that all points (u,v) are constrained to lie in a plane.
Parent Case Info
This application is a divisional of Ser. No. 08/414,437 filed on Mar. 31, 1995, now U.S. Pat. No. 5,729,471.
Divisions (1)
|
Number |
Date |
Country |
Parent |
414437 |
Mar 1995 |
|