Fields of the invention include interactive data display, exploration and collaboration.
Standard computer displays greatly limit the ability of a user to explore, interact and collaborate with others. Relatively small amounts of data are presented on a standard computer display. Use of multiple displays is common, but multiple displays do little to solve the difficulties encountered when attempting to view and explore complex data.
Scientists, designers and engineers increasingly focus on complex phenomena, rely on instruments that produce greater volumes of data, and collaborate with geographically distributed teams. General purpose computing, gaming and other applications also can present complex and highly detailed environments and interactions. A central challenge for researchers using scientific systems and other users of gaming and general purpose systems is the ability to manage the increased scale and complexity of the information and environment presented by a display. Greater scale and complexity places a heavy strain on computational systems and infrastructure. Additionally, the usability of such systems is also limited by human factors, such as their cognition and/or attention-span.
Large interactive displays have been developed, primarily for the field of scientific research and collaboration. One example is the LambdaVision 100-Megapixel wall-sized LCD tiled display introduced by Electronic Visualization Laboratory (“EVL”), which quickly resulted in over a dozen research laboratories constructing compatible instruments, called OptIPortals. EVL also developed the Scalable Adaptive Graphics Environment (“SAGE”) operating system software to enable domain scientists to work and collaborate using these displays The massive resolution afforded by these displays enabled users to view large collections of high-resolution visualizations generated in real-time from compute clusters housed at supercomputing facilities around the world.
These displays however, include limitations, which can reduce usability when being used with certain applications. One such limitation is the position and/or orientation of the display. Since electronic data often replaces the physical presentation of data, there is a concern that users will have difficulty adapting to the electronic presentation of data. Therefore, one design criteria of such displays is to provide users with the feeling of working in a traditional work environment, thereby increasing usability.
Another display device developed by EVL is the LambdaTable 24-Megapixel table-oriented LCD display. This device employs a horizontal display and presents a more natural working environment that encourages visualizations and collaborations as it replicates common human practices of working with whiteboards, printouts, blue prints, etc. where multiple people gather around a table to view data and/or documents.
Users of the LambdaTable interact with “pucks”, which are used to control the display. Special purpose pucks, for example, permit moving, shrinking, selecting, and magnifying a portion of data being displayed. Users can select and manipulate data with the pucks, and the table-sized display allows multiple users to view and interact with the data simultaneously. Displays using pucks however, have several limitations. An example limitation is that the number of users interacting with a display is limited by the number of available pucks. Pucks are also costly and subject to loss or damage, thereby requiring replacement.
One display device that avoids use of such pucks is the projector-based Microsoft® Surface display. The Surface employs a multi-touch interface, which allows a user to interact with the display by touching it with one or more fingers, thereby forgoing the need for pucks.
To further enhance usability and more closely resemble a user's natural working environment, some developers have introduced displays capable of producing three-dimensional (“3D” or “stereoscopic”) images, instead of the traditional two-dimensional (“2D” or monoscopic) images as provided in the examples above. Examples of stereoscopic displays include the Philips® MultiSync non-interactive LCD display product line. However, since these displays are configured specifically to display stereoscopic images, they are greatly limited in their ability to display monoscopic images. While such monoscopic images can be displayed, the quality/resolution is considerably poor when compared to traditional monoscopic displays.
An interactive display device of a preferred embodiment includes a display surface and a touch interface associated with the display surface. A first LCD layer generates a dynamic parallax barrier and a second LCD layer generates stereoscopic images in cooperation with the first LCD layer. A light source backlights the first and second LCD layers.
A preferred method for controlling an interactive stereoscopic display device provides two-dimensional content to be displayed by the second LCD layer, three-dimensional content to be displayed by the second LCD layer, and generates a description that overlays two-dimensional content to be rendered monoscopically onto three-dimensional content to be rendered stereoscopically. Parallax barrier content is displayed on the first LCD layer.
A preferred method for displaying images overlays monoscopic and stereoscopic image data into combined image data and displays the combined image data to the second LCD layer. Dynamic parallax barriers are displayed on the first LCD layer in cooperation with the second LCD layer. User interaction with the display device is sensed.
Preferred embodiments of the invention provide an interactive display device, which is capable of displaying both monoscopic and stereoscopic images. However, unlike other stereoscopic displays known in the art (e.g., the Philips® Multi-sync), the device displays monoscopic images as resolutions comparable to traditional monoscopic displays (i.e., full native resolution). In other words, monoscopic image resolution is not compromised by the device's ability to display stereoscopic images. The display device is also capable of displaying both monoscopic images and stereoscopic images simultaneously. That is, users can view both monoscopic windows and stereoscopic windows side by side without having to wear specialized 3D glasses or having to switch between modes. This minimizes physical encumbrances associated with the current commercial instruments. The device also provides the user with either touch or gesture-based interaction.
The usability of a display system is limited by human factors, such as cognition and/or attention-span. Usability is enhanced when a user is provided with the feeling of working in a traditional work environment. However, depending on the application being run, users will have different expectations regarding the working environment. The ability to display both monoscopic and stereoscopic images allows for greater flexibility in representing a traditional work environment for a given application.
For example, consider a group of individuals using the device to collaborate and research geographical information. A traditional working environment may include information likes maps, pictures, statistical data, etc. While some of this information lends itself to being displayed as a stereoscopic image (e.g., maps), other information is traditionally presented as a monoscopic image (e.g., statistical data). Consider for example, a spreadsheet of data concerning a particular region on a map. In a traditional working environment, geologists would likely use a map or a globe to view this particular region along with a standard spreadsheet of data. Using devices of the invention, users can view a stereoscopic representation of the map and simultaneously view a monoscopic representation of the statistical data alongside the map. Similarly, general purpose computing, gaming, communication and many other systems can benefit from the simultaneous clear display of monoscopic data along with stereoscopic data presented by a display of the invention.
An interactive display device of a preferred embodiment includes a display surface and a touch interface associated with the display surface. A first LCD layer generates a dynamic parallax barrier and a second LCD layer generates stereoscopic images in cooperation with the first LCD layer. A light source backlights the first and second LCD layers.
A preferred method for controlling an interactive stereoscopic display device provides two-dimensional content to be displayed by the second LCD layer, three-dimensional content to be displayed by the second LCD layer, and generates a description that overlays two-dimensional content to be rendered monoscopically onto three-dimensional content to be rendered stereoscopically. Parallax barrier content is displayed on the first LCD layer.
A preferred method for displaying images overlays monoscopic and stereoscopic image data into combined image data and displays the combined image data to the second LCD layer. Dynamic parallax barriers are displayed on the first LCD layer in cooperation with the second LCD layer. User interaction with the display device is sensed
Preferred embodiments of the invention will now be discussed with respect to the drawings. The drawings may include schematic representations, which will be understood by artisans in view of the general knowledge in the art and the description that follows. Features may be exaggerated in the drawings for emphasis, and features may not be to scale.
A preferred embodiment of the invention is a large format high resolution interactive display device 10 configured as desk, which is likely to replace a desk and computer workstation. It is noted that such devices can also be configured in other forms (e.g., a table or portable case) as desired by the user. The preferred embodiment display 10 is sized similarly to a traditional desk, providing a generous display and workspace. The desk is a single example, and the invention is not limited thereto. The display 10 can be configured in other arrangements, for example on stands or mounts to present a vertical display and workspace or a horizontal display and workspace when the device 10 is configured as a desk.
Referring now to
Referring back to
Users interact with the device 10 via a touch interface, which preferably provides multi-touch interaction with the device. The preferred touch interface renders external devices, e.g., pucks, unnecessary, though they may be applied if desired. The touch interface permits users to use their own hands to interact with the device 10. An example touch interface utilizes infrared LEDs 22 that are embedded in or around the clear sheet 12 to sense human contact. Preferably, there is also a gesture interface. An infrared camera 24 below the diffuser 20 cooperates with the infrared LEDs 22 to provide the touch interface. The diffuser 20 can have an opening or clear section to provide the camera 24 with a view through the diffuser to the clear sheet 12 for determining user interaction. The camera 24 and light source 18 are preferably disposed in a box 26 having a generally uniform glossy white interior.
The touch sensing used in the preferred embodiment device 10 is based upon use of Frustrated Total Internal Reflection (“FTIR”). See, Han, “Low-cost multi-touch sensing through frustrated total internal reflection.” 2005 Proceedings of the 18th annual ACM symposium on User interface software and technology. Seattle, Wash., USA, ACM. By this technique, the infrared LEDs 22 are embedded at the edges of the clear sheet 12. When an object is brought within several wavelengths' distance, the internally reflected infrared light is able to pass through the acrylic sheet where it is detected by an infrared camera. Han's original implementation was applied to projection-based screens, but has been adapted in the invention to work with LCD panels. Even when an LCD panel is completely opaque to visible light, the infrared light is able to pass through and user interaction can be detected with the camera 24 that senses the internally reflected light caused by user interaction with the screen. An advantage of using LCD panels is that they can be viewed in a normally lit room.
Tiled-display surfaces, present a unique issue with respect to constructing FTIR touch screens in that the mullions (i.e., borders) can occlude the infrared camera's view of portions of the FTIR screen. This can overcome this by first building the FTIR screen as a single large acrylic sheet rather than as a tiling of screens, and raising it some distance (between 0.5″ and 1″) above the dynamic parallax barrier panel depending on the field of view of the camera.
The device 10 also preferably includes a gesture interface. A gesture interface uses sensors or cameras to detect gestures made by a user without requiring the user to touch the clear sheet 12. The gesture interface can detect a hand or other an object in close proximity to the sheet 12, and permit gesture interaction with the device 10. Preferably, infrared cameras 25 equipped with infrared illuminators are used for gesture tracking (
The device's ability to display images and interact with one or more users is preferably managed by a software system. A preferred software architecture and method 100 for the device is shown in
In the finger tracker module 102, a noise filter component 110 causes the infrared cameras 24 to take raw images of the user's fingers or other objects as he interacts with the device, which are smoothed with various filters to reduce noise levels. Next, a finger extractor 110 examiners the contours and position of “blobs” found in the images. This will identify the finger locations of the user 108 on the clear sheet 12 of the device (
Next, a touch acquisition component 122 in the display application manager 104 acquires the touch and gesture interface data provided by the finger tracker module 102 such that an environmental interaction component 124 can manipulate and update the virtual environment and/or any object it contains. Thereafter, a three-dimensional screen descriptor 126 generates a high level description of the three-dimensional scene based on the current state of the environment. This description includes the contained three-dimensional objects, their positions, and their surface material properties. Similarly, a two-dimensional content generator 128 generates all two-dimensional content including for example, overlay images. Next, a view layout manager 130 generates a description of the screen and specifies the portions of the screen that have two-dimensional content, which should be rendered monoscopically, and the portions of the screen that have three-dimensional content, which should be rendered stereoscopically. As will be descried in further detail below, the dynamic parallax barrier driver module 106 affords for this configuration to be dynamic, and thus, the number, position, and size of two-dimensional content can be changed by the display application manager module 104 in real-time. A user configuration component 132 then generates a description of the number of autostereoscopic views to be generated and their corresponding vantage point in three-dimensional space. This vantage point can able be modified in real-time to support a variable number of users.
The last module is the dynamic parallax barrier driver module 106, which has a parallax barrier generation component 134 for generating a barrier by either drawing alternating opaque and transparent lines over three-dimensional content, or leaving areas over two-dimensional content transparent. Parameters of the parallax barrier are altered depending on the user configuration component 132 in the display application manager module 104. The resulting barrier image is then displayed on the first LCD layer 14 on the device 10. A view rendering component 136 provides for each user, a pair of images (one for the left eye and one for the right eye) that are rendered based on the scene information generated by the three-dimensional scene descriptor component 126. The total resulting number of images equals the number of users multiplied by two. Next, a three-dimensional image combination component 138 electronically slices the rendered images into a plurality of thin pieces, which are combined to form a single image. Finally, a two-dimensional image overlay component 140 overlays the two-dimensional images onto the single image to create an image which is displayed on the second LCD panel 16 of the device 10.
The dynamic parallax barrier autostereoscopic technique used in devices of the invention enables an LCD display to support viewing in several simultaneous modes, with the viewing mode selectable on a per-pixel basis. A single-viewer tracked autostereo mode enables a high-resolution virtual-reality experience with first-person perspective, giving ideal viewing of stereoscopic polygonal and volumetric data. Dual-viewer tracked autostereo mode enables a shared virtual-reality experience, with a first-person perspective for each user. Panoramic autostereo mode provides a shared stereoscopic perspective to multiple users, enabling group collaboration with stereoscopic data. Monoscopic display at the LCD's full native resolution allows for the normal viewing of fine text and high-resolution monoscopic digital imagery on both instruments.
The dynamic parallax barrier technology enables these modes and utilizes a parallax barrier, which is an alternating sequence of opaque and transparent regions. An example is shown in
Parallax barrier autostereo displays follow one of two design paradigms. Tracked systems produce a stereo pair of views that follow the user in space, given the location of the user's eyes or head from the tracking system, these are strictly single-user systems
Another option is the untracked panoramagram where a sequence of perspective views is displayed from slightly varying vantage points. An example is shown in
An example is shown in
For very large displays of the invention, particularly in the table embodiments, until large enough high resolution displays are available, some embodiments that exceed the size of currently available LCD panels may require tiling of multiple LCD panels in each of the first and second layers. An example embodiment of a tiled device is shown in
While tiling LCDs introduces mullions, the increased resolution provided is more important. The effect of the mullions can be minimized by rendering graphics is rendered in such a way as to take them into account (e.g., by placing virtual pixels behind them so the effect is like looking out of a window). The need for mullions will disappear when LCD display technology (or another type of comparable display) can make completely seamless and scalable flat-panel displays of desirable size and necessary resolution. For comparison, an example of a non-tiled device 10b is shown in
A preferred embodiment device provides 24-Megapixel resolution, and generates 9 fixed views. The preferred embodiment device also provides 8-Megapixel resolution, and generates user-centered-perspective autostereoscopic views. However, as larger LCD displays become available with high resolution, the need, for example, for multiple LCD panels in layers of a preferred embodiment table of the invention may be alleviated.
Preferred embodiment displays provide resolution that approaches print quality (approximately 72-dpi, or higher). With current LCD technology at a reasonable cost, and example embodiment large format high resolution interactive display table can be built using twelve 30″ (4-Megapixel) LCD panels (6 for image generation, and 6 for stereo separation) providing a total resolution of 24-Megapixels.
As noted, devices of the invention will have many important applications for a variety of users. Some of these users are domain scientists who increasingly rely on digital infrastructure (also known as cyberinfrastructure) and global collaboration to conduct research. Therefore, the device is preferably equipped with 1 to 10 Gigabit/s network interfaces and switches that can enable them to connect to 10-Gigabit national and international high-speed networks, such as National Lambda Rail, Internet2, and the Global Lambda Integrated Facility. As public and private networks evolve to match speeds of these high speed networks, displays of the invention can be configured to communicate with as yet to be developed networks and protocols having suitable data communication speeds.
Preferred display devices of the invention also support life-sized distance collaboration via high-definition videoconferencing with remote participants who want to be part of a meeting, and to leverage high speed networks of National Science Foundation's cyberinfrastructure facilities, such as the TeraGrid and future Petascale Facility, over high-speed networks. Further, the devices provide spatialized audio feedback with the visuals that are presented (e.g., the audio from a videoconference is proximally located with the videoconferencing image.) As shown in
While specific embodiments of the invention have been shown and described, it should be understood that other modifications, substitutions and alternatives are apparent to one of ordinary skill in the art. Such modifications, substitutions and alternatives can be made without departing from the spirit and scope of the invention, which should be determined from the appended claims.
Various features of the invention are set forth in the appended claims.
The application claims priority under 35 U.S.C. §119 from prior provisional application Ser. No. 61/066,188, which was filed Feb. 19, 2008.
This invention was made with government support under Contract No. CNS 0420477 awarded by National Science Foundation. The government has certain rights in the invention.
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/US09/34524 | 2/19/2009 | WO | 00 | 8/18/2010 |
Number | Date | Country | |
---|---|---|---|
61066188 | Feb 2008 | US |