Virtual reality (VR) is an interactive computer-generated experience within a simulated environment that incorporates mainly auditory and visual feedback, but also other types of sensory feedback. This immersive environment can be similar to the real world or it can be based on alternative happenings, thus creating an experience that is not possible in ordinary physical reality. Augmented reality systems may also be considered a form of VR that layers virtual information over a live camera feed into a headset or through a smartphone or tablet device giving the user the ability to view three-dimensional images. Current VR technology most commonly uses virtual reality headsets or multi-projected environments, sometimes in combination with physical environments or props, to generate realistic images, sounds and other sensations that simulate a user's physical presence in a virtual or imaginary environment.
This disclosure relates to streaming images of a participant in a given application to outside audience members that are viewing the participant. As the images are generated and the participant moves according to what is happening in the application, a video capture device such as a camera is automatically adjusted to reposition the camera such that the participant remains in the field of view of the camera and thus can be properly observed by the audience members. The application may be a virtual reality (VR) application where images of a VR participant are monitored and streamed to audience members (e.g., over the Internet) who view the reaction of the VR participant to the respective VR application.
In one example, a system is provided that includes a video capture device (e.g., camera) to broadcast an image of a streaming participant such as a participant in the VR application. A motion platform (e.g., pan, tilt, zoom platform) moves the video capture device in response to a control command. A controller receives location data associated with the participant (e.g., from a VR headset). The controller positions the video capture device such that the participant's location within the streamed image is adjusted by the control command. The control command can be adjusted according to participant movement detected by changes in the location data associated with the participant. In this manner, if the participant moves during a given application, the video capture device can be automatically repositioned based on the location data to provide images of the participant within the field of view of the device. In one example, the participant can be centered within the field of view or positioned within a predetermined area of the image (e.g., at the top or bottom of the image captured within the field of view). A location detector can be provided to determine location data from the participant. In one example, the location detector is provided from a VR headset where coordinates of the participant's location are transmitted as the location data to the controller. In another example, machine learning can be utilized for object detection to recognize the VR headset and thus provide the location data to the controller. In yet another example, sensors such as infrared sensor can be mounted on the headset and tracked to provide the location data to the controller.
In an example streaming application, current virtual reality (VR) systems allow streaming images of participants of VR applications. The images broadcast the participant's reactions to a given application by a video capture device such as a camera to outside audience members who may be viewing over the Internet, for example. Based on participant reactions however, such as movement of the VR participant within a given scene of the VR application, this can cause the participant to move off camera and thus prevent the audience from seeing a given reaction. The system 100 automatically detects movement of the streaming participant 120 and automatically reorients the video capture device 110 based on participant movement. In this manner, even if the streaming participant 120 physically moves their location based on a reaction to a given application, the location data can 144 reflect such movement and thus allow the controller 140 to reposition the motion platform 130 such that the participant remains within the field of view of video capture device 110.
The system 100 automatically moves the video capture device 110 to capture movements of the participant 120 in a given streaming application such as an immersive virtual reality (VR) application. In contrast to a fixed camera position for current applications where the streaming participant 120 may be off the frame or to the far left of right, the system 100 can automatically reorient the participant within the broadcast images 114 based on detected participant movement. This can include centering the participant within the image or images 114 or placing the participant in a predetermined location of the images such as at the top or bottom of the image 114. In one example, a virtual reality headset (see e.g.,
Various video processing methods and devices can be provided to track the headset which are described below. In one example, the headset can be tracked using object detection and tracking where machine learning can be trained to recognize the headset as an object. Active motion tracking is provided in case the user turns thereby occluding the headset from the video capture device 110. In another example, coordinates can be provided from VR systems that track VR controllers in the headset. The VR system can provide feedback regarding where the pan/tilt/zoom camera is and can transform three-dimensional (3D) coordinates of headset into the 2D reference point for the camera view. Depending on this information, the camera can pan, tilt, or zoom as needed. In yet another example, infrared sensing can be provided where the head set is equipped with an infrared emitter. The pan tilt zoom camera (or motion platform) can be equipped with an IR sensor. The camera is then adjusted by a controller to facilitate that beacons transmitted from the headset are centered in its frame. Other parameters can be provided to the controller 140. These parameters can include when a headset is put on and taken off, for example. This game state information (e.g., such as application running or ended) can be sent over to the controller 140 to control video capture and device movements. Also, information about headset height from the ground can be used as another signal to indicate when to track the streaming participant 120 (e.g., participant moves from a standing to a seated position).
In this example, the location data is provided as location coordinates 250 from the VR headset 210 to the location tracker 240. The location coordinates 250 are translated from virtual coordinates of the VR headset 210 to a reference point of the video capture device 220 by the location tracker 240 to provide physical coordinates for movement of the pan/tilt platform 230. For example, the reference point can be set as a point in space where the user is centered in the broadcast image which can be set as a given height and yaw position of the video capture device 220 to center the participant such as when they are seated in and before application movements have begun.
The location coordinates 250 which may be virtual coordinates associated with a virtual video frame of the application can be translated to physical coordinates to position the video capture device 220 such that the participant is centered (or some other predetermined orientation) within the physical frame produced by the video capture device. For instance, the physical coordinates determined by the location tracker 240 can be translated to the control command 234 as a pitch command that controls up and down movement or rotation of the pan/tilt platform 230 and a yaw command that controls side-to-side movement or rotation of the platform which has the effect of positioning the participant within a desired area of the video stream produced by the video capture device 220.
In another example, the pan/tilt platform 230 can be mounted on another movable axis (not shown) to control physical repositioning of the motion platform. For example, lateral (e.g., front-to-back, side-to-side) and/or up/down movement of the platform can be achieved in response to the control command 234 or a separate repositioning command sent to the movable axis. Thus, if the participant were to move greater than a predetermined distance from a given starting point, the movable axis can reposition the pan/tilt platform 230 in response to the control command 230 (or another repositioning command for the movable axis) such that the participant remains within the field of view of the video capture device 220.
In this example, the location tracker 340 a machine learning detector 370 that learns the shape of the VR headset 350 during a training sequence and generates the location data by detecting movement of shape within images captured by the video capture device 310. The machine learning detector can include substantially any type of learning to determine the shape of the head set 350 and thus, determine when the shape has moved to another position within a given image frame captured by the video capture device 310. In one example, the machine learning detector 370 can be implemented as a classifier to determine the shape and subsequent movements of the headset 350. An example of a classifier is a support vector machine (SVM) but other types of classifiers can be implemented.
The location data from the location tracker 440 provides an alignment parameter to the controller 440 to center the VR headset 450 within a field of view of the video capture device 410. For example, during an initial alignment of the VR headset 450, the beacon provided by the infrared emitter 460 can be used to center the video capture device on the participant. As the VR headset moves during a given stream application, movement of the beacon can be tracked by the infrared sensor 450 to cause subsequent adjustment of the pant/tilt platform 420 in response to the control command 424. Beacon movement can be tracked by monitoring movement of the beacon across the sensor 450 or by detecting signal strength of the beacon where maximum received strength, for example, indicates centering of the beacon.
The expression detector 510 can be configured detect a change of expression of the participant (based on a probability threshold), where the zoom command 564 can be adjusted based on the detected change of expression. In one example, the feedback device 540 can include a manual control to allow the participant to snapshot an image of the participant's current expression. In another example, the feedback device can include a muscle sensor worn by the participant to detect the change in expression. In yet another example, the feedback device 540 can include an audible sensor to detect a change in voice inflection of the participant.
As described previously, the location data can be provided as location coordinates from the headset to the location tracker 620. The location coordinates are translated from virtual coordinates of the headset to a reference point of the camera by the location tracker to provide physical coordinates for movement of the motion platform 630. In another example, the location tracker 620 can include an infrared sensor to detect an infrared beacon generated from an infrared emitter in the headset, where the location data from the location tracker provides an alignment parameter to the controller 630 to center the headset within a field of view of the camera.
In view of the foregoing structural and functional features described above, an example method will be better appreciated with reference to
What has been described above are examples. One of ordinary skill in the art will recognize that many further combinations and permutations are possible. Accordingly, this disclosure is intended to embrace all such alterations, modifications, and variations that fall within the scope of this application, including the appended claims. Additionally, where the disclosure or claims recite “a,” “an,” “a first,” or “another” element, or the equivalent thereof, it should be interpreted to include one such element and neither requiring nor excluding two or more such elements. As used herein, the term “includes” means includes but not limited to, and the term “including” means including but not limited to. The term “based on” means based at least in part on.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/US2018/056558 | 10/18/2018 | WO | 00 |