INTERACTIVE THEATER SYSTEM WITH REAL-TIME FEEDBACK AND DYNAMIC SPECIAL EFFECTS

Abstract

The interactive theater system is executed on an interactive gaming computer (IG PC) configured to execute an interactive multi-player video game. The IG PC outputs a video stream to a display system, such as a projector and screen, which allows the players to view the multi-player video game. Each player sits in a multi-sensory seat for providing multi-sensory feedback to each player during execution of the interactive multi-player video game. The player provides input using an input device having an infrared camera and motion sensors. The infrared camera is used to image an infrared light arrangement positioned above-the multi-sensory seats. Infrared camera data and motion sensor data is used by the IG PC to accurately determine the real time position of each input device. Players in a first interactive theater system can compete with players in a second interactive theater system in real time.

Description

FIELD OF THE INVENTION

The present invention is directed to an interactive theater system with real-time feedback and dynamic special effects. In particular, the present invention is directed to an interactive theater system in which each user is presented with personalized real-time feedback and control over the user's interaction with the displayed video image, such as a video game.

BACKGROUND

In recent years, video games have become exponentially more popular. Over two billion people currently play video games on phones, video game consoles, personal computers, tablets, etc. This number is expected to increase.

Electronic sports (eSports) and streaming platforms such as Twitch, Livestream, Mixer, etc., have also become increasingly popular. Video game users either compete against each other in person or stream their video game display over the internet to spectators. However, there is little to no interaction between the video game users and/or spectators other than messaging or audio chat. Spectators, in particular, are observers in a manner similar to attending traditional sporting events.

Therefore, there clearly exists a need for an interactive theater system which allows all users to be active participants which provides an engaging personalized experience while still retaining a feeling of group or team interaction.

SUMMARY

The interactive theater system of the present invention is primarily executed on an interactive gaming system (hereinafter referred to as an “IG PC”) configured to execute an interactive multi-player video game. The IG PC outputs a video stream to a display system, such as a projector and screen, which allows the players to view the multi-player video game. Each player sits in a multi-sensory seat for providing multi-sensory feedback to each player during execution of the interactive multi-player video game. The player provides input using an input device having an infrared camera and motion sensors. The infrared camera is used to image an infrared light arrangement positioned above-the multi-sensory seats. Infrared camera data and motion sensor data is used by the IG PC to accurately determine the real time position of each input device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a schematic diagram of the core components of the interactive theater system.

FIG. 2 depicts a schematic diagram of the audio-visual (AV) system of the interactive theater system.

FIG. 3 depicts a schematic diagram of the interactive system of the interactive theater system.

FIG. 4 depicts a layout view of an interactive theater according to an embodiment of the invention

FIGS. 5-7 depict views of a sample tracking arrangement.

FIG. 8 depicts views of the multi-sensory seats of the interactive theater according to an embodiment of the invention.

FIGS. 9 and 10 depict views of a sample input device.

FIG. 11 depicts a network diagram showing the interconnection of a plurality of interactive theater systems.

DETAILED DESCRIPTION

FIG. 1 depicts a schematic diagram of interactive theater system 100. Interactive theater system 100 provides an interactive experience to users in which multiple participants can engage in simultaneous gameplay in a theater setting. Each user is provided with a multi-sensory seat 122 and one or more associated input devices 104. The input device(s) 104 allows the user to interact with immersive content on the display in real-time. The multi-sensory seats 122 comprise a variety of special effects that are customized for each user or grouping of users in response to the gameplay and the user's input from the input device 104.

IG PC 102 is responsible for receiving and processing all input from input devices 104, rendering video for projectors or displays 106, and outputting show control data to show controller 108. IG PC 102 may be one or more networked or connected computers or servers in accordance with the requirements and demands of the game currently being executed by interactive theater system 100.

Each projector 106 is preferably a pair of stereoscopic projectors which project two images onto one or more curved or flat projection screens 112. Each user is preferably provided with a pair of 3D glasses so that the display from the projectors can be viewed in 3D. However, it should be apparent to one of ordinary skill in the art that projectors 106 can also be video projectors similar to those utilized in movie theaters. In other embodiments, the video output from IG PC 102 can also be shown on one or more active display screens or devices, such as LCD or OLED displays, lenticular displays, light field displays and other autostereoscopic devices. In some embodiments, each user may be provided with an HMD which receives video data from IG PC 102.

Input is received from each user of the interactive theater system 100 via one or more input devices 104. Input device 104 preferably has a form factor allowing it to easily be held or manipulated by a user. Input device 104 can comprise any standard input controls known in the art such as buttons, triggers, directional pads, joysticks, gesture tracking systems, computer vision, tablets, audible inputs (e.g. microphones), etc.

Preferably, each input device 104 is coupled to IG PC 102 via a wired connection or a wireless connection. For example, as depicted in FIG. 1, each input device 104 may be coupled to IG PC 102 via input device switch 110 via an Ethernet connection. A wired connection may be preferable in instances when there are a large number of input devices 104. However, it should be apparent to one of ordinary skill in the art that other connection types (e.g., USB) or even wireless connections may be utilized.

In some embodiments, input device 104 may also receive effect data 114 from IG PC 102. For example, input device 104 may be provided with haptic feedback and/or a speaker to provide individualized effect data 114 to the user in response to their input on input device 104, actions displayed on screen 112, or input from other users in interactive theater system 100. Each screen 112 may be surrounded by decorations or modular panels which depict scenery related to the video game currently being played. For example, if the video game is a shooter set in outer space, the periphery of the screen may be decorated with modular panels having the look of a bridge. The outer periphery of the projector's displayed video could then be designed to blend seamlessly with the modular panels or other decorations. In particular, modular panels or removable decorations allow the theater 400 to easily be repurposed for other video games.

In other embodiments, a second screen may be utilized. The second screen may comprise additional panels, surfaces, flooring or ceiling as a display surface aside from screen 112. A static image, similar to the decoration shown on scenic elements, could then be projected onto the second screen and blended into the main game image shown on screen 112. This allows the viewing area to be increased without requiring constant rendering for the imagery shown on the second screen. In another embodiment, a simple animation (e.g., a looped video) or pre-rendered video could be displayed on the second screen. In another embodiment, the primary game content may be expanded to be displayed on the secondary screen.

IG PC 102 outputs effect data 114 to show controller 108 in synchronization with the video signal output to projectors 106. Effect data 114 may include, but is not limited to, environmental audio, individualized audio, sound effects, digital multiplex (DMX) data, seat control data, and user audio. The environmental audio is output from show controller 108 to an audio digital signal processor (DSP) 116. The audio DSP 116 processes the received environmental audio and outputs it to theater environmental audio system 118.

Theater environmental audio system 118 preferably comprises a plurality of speakers and subwoofers arranged to provide immersive audio and surround sound to the users of interactive theater system 100. The number of speakers and subwoofers and their layout in theater environmental audio system 118 is determined based on the theater configuration as is known in the art.

Show controller 108 outputs the DMX data to lighting system 120. Lighting system 120 may comprise any desired combination of known lighting types such as PAR lights, strip lights, strobe lights, spotlights, moving lights, etc. in accordance with the requirements of the game or program executed by interactive theater system 100 and the configuration of the theater.

Show controller 108 further outputs seat control data and user audio directly to each multi-sensory seat 122. As will be explained in more detail later, each multi-sensory seat 122 provides individualized sensory effects to each user based on their operation of their assigned input device 104, as well as the resultant output of the game to create an individualized and immersive experience for each user.

Each multi-sensory seat 122 is preferably coupled to the show controller 108 via seat switch 124 via an Ethernet connection (e.g., CAT 6). This allows a single cable to simultaneously carry the seat control data and the user audio.

Each multi-sensory seat 122 preferably comprises a reader 126 capable of recognizing a user. For example, the reader 126 may be a NFC reader or RFID reader capable of reading a tag carried by the user sitting in the multi-sensory seat 122 which can be used to identify the user.

Thus, interactive theater system 100 allows for a fully interactive real-time system that gives every user agency to direct the outcome of the action and even affect the narrative unfolding before them on a massive, cinema-style screen 112 with professional level audio and effects.

Preferably, each component of interactive theater system 100 is individually addressable so that control signals can be provided to each component individually. IG PC 102 is able to send separate control signals to projectors 106, input devices 104, theater environmental audio system 118, lighting system 120, and multi-sensory seats 122. The address for each component of interactive theater system 100 is maintained in a database by interactive theater system 100. This allows a programmer of the video game a high level of customization because individual control signals can be sent to each component. This also allows the various components of interactive theater system 100 to be grouped together and provided with grouped control signals in addition to individualized control signals. For example, a plurality of input devices 104 may be assigned a group such that the input devices receive the same haptic feedback in addition to individualized haptic feedback for each user.

This also allows interactive theater system 100 to group any components of interactive theater system 100 into groups of 1 (individual components) up until a group encompassing the every component (e.g., for the entire theater). The groupings can also be rearranged or reevaluated in real time. For example, if a particular user's character dies in a video game, another user in the theater (or another remote theater) may be added to the group. The developer of the video game can thus take advantage of the individual and group control of interactive theater system 100 to provide an experience not possible with other systems that do not have the ability to assign groupings to components.

Referring next to FIG. 2, depicted is a schematic diagram of the AV components of interactive theater system 100 in more detail. Interactive theater system 100 also comprises a variety of facility controls and monitoring tools. For example, each interactive theater system 100 preferably comprises one or more monitoring cameras 202 (e.g., CCTV) which allows for individual monitoring of the interactive theater system 100 via a control panel 204. An operator can provide audio to interactive theater system 100 via a microphone 206 which can be coupled to the theater environmental audio system 118. The microphone 206 can be used to provide the users with procedural instructions, evacuation instructions, warnings, or any other necessary information.

Control panel 204 can also be utilized to override or independently control lighting system 120 or theater environmental audio system 118 via show controller 108. Control panel 204 can also be utilized to control show door access, show action effects, e-stop, or any other systems of interactive theater system 100.

In the depicted embodiment, the audio DSP 116 processes the received audio signal and outputs it to theater environmental audio system 118. The signal for each audio component is preferably passed through audio amplifier 208 before being output to loudspeakers 210 (1×), subwoofers 212 (2×), side surround speakers 214 (4×), or rear surround speakers 216 (4×). As already explained, each audio component of interactive theater system 100 is individually addressable and separate control signals can be sent by IG PC 102 to any component of environmental audio system 118 in any grouping.

Each multi-sensory seat 122 may be coupled to a compressed air system 218 for providing a forced air effect to each seat (neck, legs, arms, etc.). Each multi-sensory seat 122 is capable of controlling the received compressed air in accordance with the effect data received from show controller 108. The effect data may be for system wide effects, effects for particular groupings of components, or for individual components.

FIG. 3 depicts a schematic diagram showing the interconnections between the IG PC 102, input devices 104, and projectors 106. Depicted in this embodiment are a total of 24 input devices 104 arranged into six banks, with four input devices 104 per bank. Each input device 104 is assigned to a single multi-sensory seat 122. As previously discussed, input devices 104 are coupled to seat switch 124 via USB over IP. The seat switch 124 is preferably a power over Ethernet (POE) gigabit switch to avoid having to run power lines to input devices 104. The video signal output to projectors 106 is preferably over Display port or similar output interfaces.

This figure illustrates example groupings that may be employed by interactive theater system 100. For example, the input devices 104 and multi-sensory seats 122 of bank 1 may be assigned a different group than that of bank 2. The users of bank 1 would receive effect data 114 that are for every seat and other event data 114 for each individual user of bank 1. The shared effect data 114 for bank 1 provides the users with a shared experience while still allowing for each input device 104 and multi-sensory seat 122 to be provided with additional customized feedback.

FIG. 4 depicts an overhead view of a theater 400 outfitted with interactive theater system 100. As shown, projectors 106 are positioned at a rear-center of the theater 400 and a curved screen 112 is positioned at a front of the theater 300. In this embodiment, two sets of projectors 106 are employed to cover the entirety of screen 112. The output from each set of projectors 106 overlap at a center of the screen 112. The IG PC 102 is configured to output a corrected video signals so that this overlap is not detectable to the users and the screen 112 appears as a single, unified display.

The screen 112 may be curved or flat. In some embodiments, a plurality of curved screens 112 can be provided to form a partial or full hemisphere or dome display in the theater. In such embodiments, the arrangement of multi-sensory seats 122 could be arranged in an arc or circular configuration to take advantage of the increased display area (e.g., each bank could be assigned to a particular viewing region). In another embodiment, screens and multi-sensory seats can be arranged in an opposing configuration to provide versus, co-op and other gameplay styles.

A plurality of multi-sensory seats 122 are arranged in banks A-F. Each multi-sensory seat 122 has one or more input device(s) 104. The rear set of banks A-C may be positioned higher than banks D-F so that each user has a clear and unobstructed view of screen 112 as depicted in FIG. 5. It should be obvious to one of ordinary skill in the art that the number of multi-sensory seats can easily be expanded into different tiers and configurations (e.g., 16 seat, 36 seats, etc.). It should be obvious to one of ordinary skill in the art that the number of input devices can easily be expanded.

Directly above the banks A-F is an arrangement of infrared (IR) lights 402 suspended from or integrated in the ceiling of theater 400. In some embodiments, each input device 104 comprises one or more infrared cameras configured to image the infrared light arrangement 402. The infrared camera data can then be used by IG PC 102 to pinpoint the exact spatial location of each input device 104 in the theater in real-time. Further, the received infrared camera data can also be used to identify the orientation of each input device 104, with or without additional positional information received from input device 104.

Other methods of tracking input devices 104 may also be employed. For example, an array of cameras (infrared or visible light) may be arranged throughout the theater and each input device 104 may comprise one or more markers. By imaging the markers on the input device 104, the location of each input device 104 can be tracked in real-time. Each input device 104 may further comprise a plurality of sensors such as an accelerometer, gyroscopic sensor, etc. which can be used to estimate the location of input devices 104 if one or more of the markers is temporarily occluded.

FIG. 6 depicts an overhead view of theater 400 with IR light arrangement 402 superimposed and FIG. 7 depicts IR light arrangement 402 in isolation for clarity. Each dot represents an individual IR light 404. The 60 depicted IR lights 404 are preferably arranged asymmetrically. An asymmetric pattern allows IG PC 102 to more quickly and easily triangulate and identify the position of each input device 104. Further, if the input device 104 is temporarily unable to image the IR light arrangement 402, the motion sensors in input device 104 can be used to provide interim position information for each input device if needed. Then, when the IR light arrangement 402 is visible again, the positioning information can seamlessly transition to again using the IR light arrangement 402 for position tracking.

Further, because the asymmetric pattern can easily be changed, each theater 400 can be assigned a unique asymmetric pattern if needed. This would aid in the identification of the theater 400 in environments where input devices 104 are wireless and potentially used in multiple theaters 400.

It should be obvious to one of ordinary skill in the art that any type of tracking system can be utilized in connection to interactive theater system 100 as long as it is able to provide the real-time location of the input devices 104 within the theater. This allows interactive theater system 100 to easily be upgraded as new tracking technologies are developed because interactive theater system 100 only requires the location of the input device 104 to function and to provide the appropriate feedback.

FIG. 8 depicts multiple views of multi-sensory seats 122. As previously mentioned, each multi-sensory seat is coupled to seat switch 124 and receives seat control data and user audio uniquely tailored for each user. Each seat is preferably capable of multiple movement degrees of freedom in response to the seat control data, including, but not limited to, heave (up & down), roll (tilt side to side), pitch (tilt forward and backward), sway (move left and right), and vibration (e.g., 5-250 Hz). Each multi-sensory seat 122 is also capable of producing a variety of other sensory effects in accordance with the seat control data. Sensory effects can include, but are not limited to, air neck blast, a poke effect (back, legs, etc.), leg tickler, water spray, face blast, wind system (variable intensity), multi-scent system, strobe lights, face lights, face illumination, floor illumination. In some embodiments, each multi-sensory seat 122 is provided with one or more speakers in the vicinity of the head of the user for outputting user audio unique to each user. Further, each multi-sensory seat 122 preferably comprises an occupancy sensor so that IG PC 102 only produces seat control data and user audio for occupied seats.

Each multi-sensory seat 122 preferably comprises an input device holder 802 for holding input device 104 when not in use. An example input device holder 802 is depicted in FIG. 9. As shown, input device holder 802 comprises a bracket 902 which can be used to couple the input device holder 802 to a surface, such as an arm of multi-sensory seat 122. Sheath 906 contains an opening in which input device 104 is placed when not in use. The sheath depicted in FIG. 9 is similar to a holster and input device 104 has a form factor similar to a blaster or a gun.

FIG. 10 depicts an example of an input device 104 in isolation. In this embodiment, input device 104 has the form of a gun with the main control being a trigger 1002. A user can point input device 104 at screen 112 during a shooter or blaster video game being produced, for example. The input device 104 preferably comprises motion sensors which may include, but are not limited to, an accelerometer, a gyroscope, a gravity sensor, rotation vector sensor, step count sensor, depth sensor, GPS, etc. Further a top of the housing of input device 104 comprises infrared camera 1004 for imaging IR light arrangement 402 as previously described. The motion sensor data and infrared camera data from each input device is provided to IG PC 102 in real time. This allows IG PC 102 to be aware of the precise location and orientation of each input device 104 in the theater 400. Further, this allows input device 104 to serve multiple functions (i.e., more than just a screen pointing device).

In particular, IR light arrangement 402 allows interactive theater system 100 to provide a unique, advanced targeting system providing accurate and persistent positional tracking that not only detects where users aim but how they move their controller. Because the tracking is very accurate, the input device can be used as multiple different in-game instruments such as a hammer, wand, shield, sword, gun, etc., in accordance with the story or the user's positioning of the device.

As already described, IG PC 102 is constantly aware of the location and position of each input device 104 being used in theater 400 (i.e., tracking 6DOF data). This allows the IG PC 102 to render the output of each user as if it is coming from their seat location and being directed toward the screen 112. For example, if a user used input device 104 as a rocket launcher, audio could be provided to multi-sensory seat 122 so that it sounds like the rocket originates from the user's seat. Further, IG PC 102 can adjust the trajectory of the rocket as it enters the virtual space shown on the display such that it appears as if the rocket originated at the user's seat instead of the center of the screen as occurs in most traditional shooters. This adds to the sense of realism and immersion for the user.

Other advanced effects are also possible using input device 104 because of its integrated motion sensors. For example, in the same scenario described above, a user could alter the course of the rocket left/right by moving the input device left/right or curving its position in space to represent the desired trajectory. Using this method, the user could hit enemies behind pillars/posts.

Thus, interactive theater system 100 provides an interactive 3D system that only requires 3D glasses for immersion. This allows users to be more aware of the actions and expressions of other users, which increases the feeling of comradery between players while still providing a tailored experience for each user.

The multiple input devices 104 can also be utilized to perform group actions since IG PC 102 has access to the real time movement of each. For example, if a portion of a video game required that a spaceship turn to avoid a collision, a certain predetermined number of users may be required to perform a certain action (e.g., move the input device 104 or rotate it left) to avoid the collision. Thus, input devices 104 can also be used to determine the outcome of group actions in the video game. As already explained, this feature is possible because each input device 104 is separately addressable and can be assigned to groups by IG PC 102.

Interactive theater system 100 can also track how input devices 104 are used or what percentage of people performed group actions and in what manner. This can be used to adjust future sensitivity required. For example, users may be required to tilt the input device 104 more or less to perform certain actions. This can be used to adjust sensitivity for new players or increase it for advanced players. In a gaming session with many participants, this adjustability can be used as a handicap to level the playing field between players. Further, interactive theater system 100 may have machine learning or AI capabilities to dynamically adjust the sensitivity for each player on the fly according to their strengths and weaknesses.

FIG. 11 depicts a network diagram showing how multiple theaters 400a-400d can be in communication via the Internet or LAN. In the depicted embodiment, theaters 400a and 400b are geographically co-located at SITE 1. SITE 1 is in communication with theater 400c at SITE 3 and 400d at SITE 2. Because the program for each theater is executed by an IG PC 102, multiple theaters 400 can communicate in real time to allow the groups in each theater to participate in multi-player. This allows for a new type of gaming in which teams of players (using the same screen) can compete against other similar teams in one-time or league play. Each group can be provided with their own unique perspective inside the same theater 400 or even into another theater 400. In contrast, current gaming systems do not allow for such collaborative and interactive group play, with each user simultaneously and actively competing.

In some embodiments, users may register with interactive theater system 100 in advance and maintain an active user profile. When a user arrives at interactive theater system 100, they enter their information at kiosk 1102 which loads their profile from user database 1104. This allows each user to configure details about their profile or character ahead of time. For example, a user may choose a certain weapon type in advance for use with input device 104. This would then be loaded when the user checks in using kiosk 1102.

The user can also use their user profile in conjunction with other external applications 1106. For example, the interactive theater system 100 may be associated with a particular smartphone app 1106 or PWA app 1108. The user could interact with the smartphone app as their character and acquire unique weapons or other perks which could then be used in the interactive theater system 100 for particular games. Further, the user profile could be used by the user to check their current rankings, status, etc. Exemplary user profile systems compatible with the present invention are disclosed in U.S. Pat. No. 10,857,467, issued Dec. 8, 2020 and U.S. patent application Ser. No. 16/897,181, filed Jun. 9, 2020, the entire contents of which are hereby incorporated by reference in their entirety.

When the user enters the theater, the reader 126 on multi-sensory seat is used to identify the user sitting in the seat and any information from their user profile is loaded. The user's own cell phone with an NFC chip may be used to identify the user or a tag may be provided (e.g., carried on the user's keychain). It should be obvious that any type of technology capable of reading/identifying a user may be employed by interactive theater system 100 (e.g., card reader, RFID reader, NFC reader, barcode scanner, etc.).

However, it should be apparent that interactive theater system 100 can be used by users without user profiles. Those users would instead use kiosk 1102 to register and then be assigned a multi-sensory seat 122 for a gaming session.

SITE 1 may further incorporate a leaderboard 1110 to keep track of user and team statistics daily, weekly, etc. Or, in a tournament mode, leaderboard 1110 can be utilized to track the progress of each team or individual through the tournament. The operator can even use the microphone to act as an announcer or real time commentary, similar to commentators in traditional sporting events.

Claims

1. An interactive theater system comprising: an interactive gaming system configured to output a video stream;a display system for receiving and displaying the video stream received from the interactive gaming system;a plurality of seats;an infrared light arrangement positioned above the plurality of seats; anda first input device associated with a first seat of the plurality of seats;wherein the first input device comprises: an infrared camera for imaging the infrared light arrangement,wherein the interactive gaming system receives infrared camera data from the infrared camera for real-time tracking of the first input device.

2. The interactive theater system according to claim 1, wherein the first input device comprises: a motion sensor for tracking an orientation of the first input device,wherein the interactive gaming system receives the orientation from the first input device.

3. The interactive theater system according to claim 1, further comprising: a second input device associated with a second seat of the plurality of seats.

4. The interactive theater system according to claim 3, further comprising: a tracking system for tracking a real-time location of the first input device and the second input device input device;wherein a first response signal is sent to the first seat; andwherein a second response signal different than the first response signal is sent to the second seat.

5. The interactive theater system according to claim 4, wherein a group response signal is sent to the first seat and the second seat, andwherein the group response signal is different than the first response signal and the second response signal.

6. The interactive theater system according to claim 1, wherein the first seat is a multi-sensory seat capable of providing haptic feedback.

7. The interactive theater system according to claim 6, further comprising: a plurality of modular panels surrounding the display system having decorations related to the video stream.

8. The interactive theater system according to claim 1, further comprising: a decoration projector for projecting a static image onto an area surrounding the display system.

9. The interactive theater system according to claim 1, further comprising: a tag reader for reading a tag of a user sitting in the first seat.

10. The interactive theater system according to claim 9, wherein feedback provided to the first input device is altered based upon the read tag of the user.

11. The interactive theater system according to claim 1, wherein the display system comprises: a first screen;a second screen;a first projector for projecting a first portion of the video stream on the first screen;a second projector for projecting a second portion of the video stream on the second screen.

12. The interactive theater system according to claim 11, wherein the first projector projects a part of the first portion onto the second screen, andwherein the second projector also projects a part of the second portion onto the first screen.

13. The interactive theater system according to claim 1, wherein the infrared light arrangement comprises:a plurality of infrared lights arranged in an asymmetric pattern on a ceiling.

14. The interactive theater system according to claim 1, wherein the first input device has a form factor related to the video stream.