Patent Grant
11949891
In many video distribution systems, a source image is captured at a high bit depth, such as 10-16 bits per sample (bps) for each of three primary colors and is converted to a lower bit depth, such as 8 bits per sample for each of the three primary colors in the source before the source is delivered to a display. At 16 bps, a color sample can have over 65,000 different values, but at 8 bps, the color sample can only have 256 different values. This conversion, or remapping, reduces bitrate but may result in a reduction in video quality. This methodology can cause banding and loss of detail. It would be desirable to improve video quality without unnecessarily increasing the bitrate requirements.
In some scenarios, a camera captures between 8 to 16 bits per pixel (bpp) for each of the three primary colors. The higher the bit depth, the greater the detail or quality of picture that may be produced.
Consumer devices, including HDTVs, mobile devices, or computers, are generally equipped to receive video with bit depth of 8 bpp for each primary color. Current consumer display technology such as LCD, LED, and Plasma, typically display around 8 bpp for each primary color as a balance of visual quality and cost effectiveness.
One TV distribution scenario uses a camera which captures video at a high bit depth (such as 10-16 bpp). This signal is converted to a lower bit depth, such as 8 bpp, for broadcast delivery to a consumer display. At 16 bpp, a pixel (or each component of a pixel such as an R, G, or B component, a Y, U, or V component or a Y, Cr, or Cb component) may be assigned one of 65,536 (216) different values, but at 8 bpp, a pixel (or each component of a pixel) may be assigned one of only 256 (28) different values.
In addition to creating additional streams by using increased native resolutions, viewing quality may be enhanced by creating streams with a higher bit depth. Conversion from a high bit depth to a low bit depth may result in a variety of video quality problems, such as loss of detail, reduced dynamic range, image artifacts, and problems at the extremes of the brightness scale. Loss of detail causes flattening or complete loss of fine textures. Reduced bit depth may result in black colors being less black, and white colors being less white, for example. Image artifacts include banding instead of smooth gradients and stair-step edge artifacts. Brightness problems may include issues with objects in dark or bright sections of the image appearing severely degraded or even washed out.
Tone mapping is a technique used to map the sample values at a particular pixel from one value to another. Tone mapping can be employed for different effects such as giving an image or video an overall lighter or darker appearance or to approximate the appearance of high-dynamic-range (HDR) images on standard display devices.
Tone and/or Bit Depth Remapping.
Systems and methods described herein enable viewers of video streams to select one or more particular regions of interest to undergo a selected tone mapping. The regions of interest may be stationary or may move across the screen. In an exemplary method of providing user selectable tone mapping for regions of interest, a first video stream generated using a first tone mapping is sent to a video client, e.g., by a server. Information identifying at least one available region of interest associated with the first video stream is also sent to the client. A user of the client device may select one or more of these regions of interest, and the client device reports the selection to the server. The server receives an indication of the region or regions selected by the client and sends a second video stream to the client. If the user selected one of the available regions of interest, then in the second video stream, an area of the video outside the selected region of interest is generated using the first tone mapping, and an area of the video within the selected region of interest is generated using a second tone mapping different from the first tone mapping. If the user selected more than one of the available regions of interest, then in the second video stream provided by the server to the client, the area of the video which is outside the selected regions of interest is generated using the first tone mapping, and the areas of the video within the selected regions of interest is generated using tone mappings other than the first tone mapping. For example, the areas corresponding to each of the selected regions of interest in the second video stream may be generated using tone mappings designed for or optimized for the individual regions of interest.
In some embodiments, the tone mapping used in a selected region of interest is selected to generate a highlighting effect to allow the user to more readily follow the region of interest. The selection of the particular tone mapping may be made by the user. For example, the user may select whether the tone mapping results in a highlighting effect. In some embodiments, the user may also select the degree of highlighting. Some embodiments include tracking of a location of at least one object in a video sequence. This may be of particular use where the region of interest is a region occupied by a particular athlete or ball in a sporting event, by a particular horse or car in a race, or in other circumstances where it may otherwise be difficult to follow a particular region of interest. In some embodiments, the location of the selected region of interest changes on a frame-by-frame basis. In some embodiments, the location of the selected region of interest changes on a segment-by-segment basis.
In various embodiments, region of interest may have a rectangular shape, an oval shape, or other regular or irregular shape.
In some embodiments, the second video stream is generated on demand by combining independently-coded slices of the video.
In another exemplary embodiment, a method is provided to enable user-selectable tone mapping for regions of interest in an adaptive bit rate (ABR) video system. The location of one or more objects or other regions of interest is tracked in a video sequence. Information describing the objects may be generated and stored. One or more different tone maps are applied to the different regions of interest. In some embodiments, a user is provided with the ability to select both which region(s) of interest are to be highlighted and which tone map(s) are to be used to effect the highlighting. In response to user selection of a particular region of interest and a particular tone map, a user is sent a video stream that includes the selected region(s) of interest highlighted with the selected tone map(s). The application of various tone maps to various regions of interest may be performed in advance of the request by the user, with a server delivering the appropriate pre-mapped video stream in response to the client request. In some embodiments, the video with applied tone maps is divided into a plurality of slices. Those slices having the appropriate tone mappings may be merged on demand in response to a user request, such that the user's client device receives only a single video stream instead of a plurality of slices. In other embodiments, the appropriate slices may be delivered separately to the client device to be displayed together.
Client-Side Mapping Selection.
Systems and method described herein enable viewers of video streams to select one or more particular regions of interest to undergo a corresponding tone remapping. The regions of interest may be stationary or may move across the screen. In exemplary embodiments, one or more networked servers operate to perform the (potentially computationally expensive) calculations involved in tracking regions of interest (including objects of interest) in a video stream and identifying parameters for one or more tone remapping functions. This information may then be delivered to a video client device. In some embodiments, the selection of parameters for one or more tone remapping functions is based on information from the client device that identifies a display type, reports on ambient illumination at the client location, and/or provides other information regarding viewing conditions. The tone mapping parameters may be selected or calculated with the goal of optimizing visibility of a selected region of interest under the viewing conditions reported by the client device. The region of interest information (which may include the location and dimensions of the region of interest in each video frame) and parametric values of a selected remapping function are passed in-band to the client in some embodiments. The client device may then operate to apply the tone remapping function to one or more selected regions of interest using the parameters received from the server.
In an exemplary embodiment, a high bit depth video (which may be a high dynamic range video) is captured. A server operates to determine the location of regions of interest (which may be regions that include objects of interest) in each frame of the video. The server further generates a parameter set for each of the regions of interest, where the parameter set defines a tone map to be applied if a user selects the corresponding region of interest for highlighting. The parameter set and/or the tone map may be defined for all time, for a certain period of time, or may vary with time (e.g., on a frame-by-frame basis).
The parameter sets may include parameters such as a tone map model identifier, maximum and minimum values (e.g., in the case of a linear map with clipping), a width and midpoint (e.g., in the case of a sigmoid mapping), a gamma value (in the case of gamma mapping) and/or other parameters, such as parameters to define a piecewise linear tone map or parameters in a lookup table. In some embodiments, the server determines the locations of regions of interest by collecting object-tracking sensor data and computing the location of an object in a frame by fusing camera location and the sensor data. In some embodiments, the server receives, from the client device, a report identifying viewing conditions at the client. The viewing conditions may include, for example, information on the ambient illumination level at the client device and/or information on the type of display device being used. In such embodiments, the server may generate a tone map parameter set based on the reported viewing conditions. In some embodiments, the server may generate a plurality of parameter sets, each corresponding to a different viewing condition, and the client may select the parameter set to use based on the viewing conditions.
In another exemplary embodiment, a video client device receives a video stream and receives a description of objects being tracked in the video. For each object being tracked in each frame, the video client device receives tone map cues, such as a tone map parameter set, from a server. The client may receive different tone maps for different viewing conditions such as, for example, ambient illumination and display type. The client device applies the tone map defined by the tone map cues to an area of the video frame that corresponds to a selected object. The selection of tone map may be made based on viewing conditions. In some embodiments, the client device provides feedback to the server, where the feedback may identify the viewing conditions.
Distribution of Streaming Video Content.
An exemplary functional architecture of an adaptive bitrate video distribution system with zoom coding features is illustrated in
An exemplary zoom coding encoder 112, shown in the bottom part of the workflow in
The streaming server 108 is configured to transmit a video stream over a network to different display devices. The network may be a local network, the Internet, or other similar network. The display devices include devices capable of displaying the video, such as a television, computer monitor, laptop, tablet, smartphone, projector, and the like. The video stream may pass through an intermediary device, such as a cable box, a smart video disc player, a dongle, or the like. The client devices 110 may each remap the received video stream to best match the display and viewing conditions.
Bit Depth Conversion.
In converting from high bit depth to low bit depth, the conversion may be linear or non-linear. Non-linear conversions may be used to provide contrast enhancements and other effects. Some embodiments may be described as an “S” shaped or sigmoid conversion curve in which the horizontal axis is the high bit depth input pixel value and vertical axis is the low bit depth output pixel value. Non-linear curves may provide more detail in certain pixel value ranges and less detail in other pixel value ranges. For example, mid-tones may receive a relatively greater proportion of the available dynamic range than dark or light tones. Conversion may be applied to any or all of the available image components (e.g., luminance, chrominance, red, green, blue, or other components).
Converting from high bit depth (e.g., 16 bpp) to low bit depth (e.g., 8 bpp) prior to delivery to a display may cause banding, loss of detail, and other image effects.
Optimal high-bit-depth to low-bit-depth conversion mappings may depend on a video signal's pixel characteristics. Such high-bit-depth to low-bit-depth conversion maps may vary per-pixel, per region, or per frame. Applying a single conversion map to an entire frame may result in some portions of the frame receiving non-optimal mappings. Re-computation of the optimal mapping may be performed when temporal variations with a region change significantly (even when tracked). For example, a given image (or frame of a video sequence) may have one portion in the shadows and another portion in bright light. Such a scenario may benefit from application of different conversion maps in different regions.
By changing the conversion to increase sensitivity for dark areas, textures remain visible after the conversion. This change comes as a tradeoff to the bright areas, which become washed out. Therefore, it can be beneficial to separate the total image into regions of interest with different conversions available for each area. Alternatively, different mapping sets may be created for different viewing device types and different ambient viewing conditions.
Different consumer display devices include controls to vary display parameters, such as contrast and brightness. The contrast, brightness, and other image control parameters available on a user's display operate on the low-bit-depth signal received by the display, not the native high-bit-depth source signal.
Regions of interest may be defined in the original content, and different conversion functions may be designed (e.g., optimized) to distribute pixel values within the region of interest. This process may be repeated for each region of interest in the source material. In some embodiments, regions of interest are highlighted
A suitable (e.g., optimal) conversion function may be defined for the pixel values in a region of interest. For example, a histogram may be generated from the pixel values which fall within a given ROI, and a suitable (or optimal) conversion function may be generated from the histogram. Exemplary techniques that may be used for generation of a conversion function from histogram data are described in, for example, R. Mantuik et al., Display Adaptive Tone Mapping, 27 (3) A
Alternately, a suitable conversion function may be created for user-defined (e.g., artist- or content producer-defined) ROIs. For example, a user interface may display the imagery and indicate the boundaries of the ROI. The user interface may display a prototype conversion function similar to the ones shown in
Tone Mapping.
Bit depth conversions can be used as a type of tone mapping. A tone mapping maps one or more components of a pixel to different values. For example, a tone mapping may be used to increase the overall brightness of an image (e.g., a frame in a video) by increasing the luma component at each pixel, or may decrease the overall brightness of the image by decreasing the luma component at each pixel. Such tone mappings may be implemented using gamma correction, such that an output luma value V′out is generated from an input luma value Y′in according to the function
Y′out=Y′inγ,
where values of γ>1 tend to darken the image (and to make details in bright regions more distinguishable), and values of γ<1 tend to lighten the image (and to make details in dark regions more distinguishable). Tone mappings may also be applied to chroma components or to a combination of luma and chroma components. Tone maps may be global (such that the same function may be applied to every pixel), or they may be local (such that the function applied to a pixel depends in part on what area or region of the image contains the pixel, and/or on the values of nearby pixels). Numerous types of tone mapping are known to those skilled in the art and can be defined on an ad hoc basis (e.g., using control points). In exemplary embodiments, tone maps are generated using the techniques described in R. Mantuik et al., referenced above.
Where the number of bits in each of the input pixel values of a tone mapping is different from the number of bits in each output pixel value, the tone mapping operates as a bit depth conversion. Thus, tone mapping and bit depth conversion may be performed together in a single step, or they may be performed sequentially (in either order).
Overview of Exemplary Embodiment.
Exemplary systems and methods disclosed herein enable the client to request and receive video streams in which one or more selected ROIs may be displayed with a tone mapping that is different from a default tone mapping displayed in other regions. These different tone mappings can be used for various purposes, such as to highlight particular objects (such as people) of interest in a video or to improve visibility of light or dark features in different areas of the video. Systems and methods disclosed herein further operate to manage the transmission bandwidth from the server to the client and to limit the number of pixels that are encoded or re-encoded by the server. In some such embodiments, each video frame is partitioned into independent slices, and the slices that contain pixels within an ROI are provided as alternative encoded slices that the client can receive.
In this embodiment, ROIs in the video frame are identified (step 1010), and metadata is created (step 1012) to notify the client about the availability of different ROIs, and what different tone mappings are available for each ROI. Each ROI may be remapped (steps 1014, 1016) using different bit depth conversion functions (or tone mappings), or the same bit depth remapping may be applied to all regions of interest. These conversions may be used to enhance shadow or bright areas and/or to provide better mappings for different client display types or viewing environments, among other uses.
The ROIs are partitioned into slices (steps 1018a-d) of the same pixel dimensions and coordinates as the default video encoding. For each ROI remapping, the pixels in the ROI and the pixels generated using the default remapping for each slice are merged (steps 1020a-d) so that the ROI pixels are replaced in the slice (e.g., in the image area to be covered by the encoded slice) with the alternative-remapping ROI pixels. Non ROI pixels are not replaced. Each slice is encoded (steps 1022a-d). In an ABR system such as DASH (dynamic adaptive streaming for HTTP), each slice may be ABR encoded into different resolutions and/or bit rates.
In response to a client request for the default video stream, the slices generated using the default mapping are delivered to the client. In some embodiments, the server concatenates the encoded slices into a video frame bit stream so that a single stream is sent for each frame. In alternative embodiments, the individual slices are sent to the client, and the client uses these slices to compose the frame.
In response to the client requesting a specific mapping for a region of interest, the corresponding slices are sent to the client along with non-ROI default mapping slices. In a case where the ROIs move spatially over time, different slices may be used in different frames (e.g., slices which contain all or part of the tone-map enhanced version of a selected ROI may be sent to the client, and the set of such slices may vary frame-by-frame as the ROI size, shape, and/or position vary). The server uses the correct slices for each ROI mapping that is requested by the client.
In some embodiments implemented in ABR systems, the period of the ABR segments encoded determines the time latency when a client can switch from receiving different ROI mappings. That is, once the client requests a different remapped ROI, the client does not receive that new version of the video sequence until the next ABR segment period. In other embodiments, fast stream-switching techniques may be used to enable clients to switch more quickly to a different ROI stream.
An example video frame is shown in
Over time, as the ROIs move within the spatial frame, the server may provide the proper slices to a client requesting specific ROI mappings. Over a typical two-second ABR segment duration, the ROI slice location may change significantly. The server thus provides the proper slices in sequence to the client (composed in a frame).
In the case where more than one ROI is contained in the slice, then the slice merging may merge slices from the default, ROI #1, ROI #2. For example, this may create a slice suitable for the case where the user of the client device selects both ROI #1 and ROI #2 for highlighting. This creates an additional slice to encode, which may be in addition to alternative versions of the slice which highlight only ROI #1 and/or ROI #2.
Metadata describing the ROI may be included in each video frame user data enabling the client to provide user interface information and/or to use the information for display.
With multiple clients requesting different ROI mappings, the server thus provides custom streams to clients based on the client requests.
Each client may decode the slices and compose a frame just as a conventional decoder client. In an exemplary embodiment, the client retrieves metadata (such as a manifest file) that describes the available ROIs and bit depth mappings. In a DASH ABR system, this metadata may be provided in a media presentation description (MPD). The client may continue to receive a default stream or may request streams in which a selected mapping is applied to a selected ROI. Selection may be made automatically by a client playback device (e.g., based on display type, viewing conditions, or the like), or selection may be made manually by a user (e.g., from a menu). The client receives and decodes the selected stream.
Overview of Client-Side Mapping Selection.
In exemplary systems and methods disclosed herein, a client requests and receives a desired video stream. Along with the requested stream, the client also receives (1) information identifying one or more regions of interest with the stream, and (2) information identifying one or more tone mappings to apply to the region or regions of interest. The information identifying the region or regions of interest may be, for example, coordinates demarcating the boundaries of each respective region of interest, among other options described in greater detail below. Information identifying the tone mappings may include information identifying the type of tone mapping as well as information identifying parameters of the tone mapping, as described in greater detail below. In some embodiments, the information on the region(s) of interest and the tone mapping(s) is provided on a frame-by-frame basis and may be provided in-band (e.g., in the user data of the video frame within encoded media segments delivered to the client), or out of band in a separate ROI metadata file retrievable by the client. In some embodiments, the information on the region(s) of interest and the tone mappings may be provided in a manifest file such as a DASH Media Presentation Description (MPD) file. The tone mapping information may be retrieved from the same server as the video stream (e.g., from a server in a content distribution network) or from a different server.
In response to a determination to apply a tone mapping to one or more of the regions of interest, e.g., because of a user selection of that region (or those regions) of interest, the client device applies the tone remapping or remappings identified in the received information to the associated regions of interest. Areas of the video frame outside any selected region of interest are not remapped. The client device displays (or causes to be displayed) the video with remapped region or regions of interest.
In some embodiments, the client device collects information regarding viewing conditions (e.g., the type of display device and/or the ambient illumination level) and sends that information to a server, and the tone mapping information that is sent to the client is generated or selected by the server based at least in part on the reported viewing conditions. The tone mappings can be used for various purposes, such as to highlight particular objects (such as people) of interest in a video or to improve visibility of light or dark features in different areas of the video. Selection of Region of Interest.
Selection of Region of Interest.
The regions of interest may be defined manually. For example, a human content producer or artist may trace the regions of interest on a displayed frame of content in a user interface, and the traced outline may be used to define the region of interest. Alternately, the regions of interest may be identified automatically. For example, an edge detection algorithm may be used to detect strong edges which may represent region boundaries. For another example, the image may be divided into patches (e.g., regular patches or blocks) and a local histogram may be generated from the pixels in each patch. A comparison or clustering algorithm may be used to merge patches with similar histograms. If desired, merging may be restricted to spatially neighboring patches to produce spatially contiguous regions. This technique may be used to identify regions with similar pixel value distributions and regions of interest. Another example uses an object detection/segmentation algorithm to identify objects of interest, and the image region occupied by the object may be used as an ROI. In each case, the ROI may occupy an arbitrary shape, or it may be restricted to a rectangular shape, elliptical shape, or other predefined shape.
An ROI may be generated for a particular image (e.g., at a particular point in time), or it may span a range of time. In the latter case, the boundaries which define the ROI (or alternately, the footprint of the ROI in the image) may vary across time. For example, an ROI may track an object which changes size, changes shape, or moves across the screen.
Another option is to identify regions of interest semi-automatically. For example, an automatic segmentation or clustering algorithm (e.g., as described above) may create an initial segmentation of the imagery into regions of interest, and a user (such as an artist or content producer) interacts with this segmentation via a UI. The user selects which areas to designate as regions of interest and may edit the boundaries of those ROIs (e.g., by adding or moving segmentation boundaries, or by merging multiple regions). Manual editing may be employed, for example to refine or alter the results produced by automatic algorithms.
Selection and Application of Different Tone Maps in Different Regions.
To improve visibility under current viewing conditions, a client may act in response to a cue (e.g., a user input, or automatic detection of an ambient illumination level) to request a representation with one or more different tone mappings (other than the default) applied in different regions of interest. In some embodiments, the client may act in response to the cue to itself apply a tone mapping (other than the default) to one or more regions of interest. The tone mapping may be selected to improve the visibility of the video under current viewing conditions, to improve the visibility within a selected region of interest, or to facilitate identification of the region of interest.
In some embodiments, a user provides input to select a particular region of interest. For example, if a user selects ROI #1, a tone mapping 308 may be selected (or locally applied by the client) that optimizes the visibility of features in the mountaintops. This tone mapping may be, for example, a remapping that provides greater detail in brighter regions. In some embodiments, the tone mapping (e.g., parameters that define the tone mapping) is received by the client from the server, along with the definition of ROI 1, as described herein. In some embodiments, this optimized remapping is applied only to pixels in ROI #1, with the remaining pixels in the frame being remapped to the display bit depth using a default tone mapping (such as a linear function). In other embodiments, this tone mapping may be a mapping that is applied to the entire frame. As a result of applying the mapping to the entire frame, some detail may be lost in shadowed regions (e.g., some banding may be visible in ROI #2), but the mountaintop features in ROI #1 will be visible with higher quality (312).
If a user selects ROI #2, a tone mapping may be selected (or applied by the client) that optimizes the visibility of features in the shaded brook. This mapping may be, for example, a mapping that provides greater detail in darker regions. The tone mapping for ROI #2 (e.g., parameters that define the tone mapping) may be received by the client from the server, along with the definition of ROI #2, as described herein. In some embodiments, this mapping is applied only to pixels in ROI #2, with the remaining pixels in the frame being mapped using a default tone mapping, such as a linear function. In other embodiments, this mapping may be a mapping that is applied to the entire frame. As a result of applying the mapping to the entire frame, some detail may be lost in brighter regions (e.g., some banding may be visible in ROI 1), but the shadowy brook features in ROI 2 will be visible with higher quality (314).
In some embodiments, a user may select both ROI #1 and ROI #2. In this case, separate tone mappings may be applied to those regions (by the client or by the server) as described above. In other embodiments, if a user selects both ROI #1 and ROI #2, a tone mapping may be selected that balances optimization for visibility of features in the brook and the mountaintops. This mapping may be, for example, a mapping that provides greater detail in the brightest and in the darkest regions. This mapping may be a mapping that is applied to the entire frame. As a result, some detail may be lost in mid-tone regions (e.g., some banding may be visible outside ROI #1 and ROI #2), but the mountaintop region of ROI #1 and brook region of ROI #2 will be displayed with higher quality (though perhaps not with the same quality that could be seen if one of those regions was selected individually).
Exemplary Manifest.
Embodiments disclosed herein may be employed in an MPEG-DASH ABR video distribution system. Provided below is an exemplary DASH MPD that may be used in some embodiments to convey information regarding the availability of different ROIs and different tone mappings available for those ROIs. Information analogous to that provided in the example MPD may be provided in-band (e.g., in the user data of the video frame within encoded media segments delivered to the client), or in a separate ROI metadata file retrievable by the client. The ROI parameters described here may be provided periodically or on a frame-by-frame basis, to support the ROI moving spatially over time. Therefore, the client on startup retrieves the metadata on ROIs from the MPD, but once decoding frames, the client may check each frame ROI user data (which may be available in-band or from a separate metadata file, if it is not present in the MPD) to update its user interface or other algorithms related to ROI selection and display.
The following parameters are used in the exemplary MPD:
Object x,y position and size may be provided in pixel units that correspond to the first-listed representation in the appropriate adaption set. For secondary representations (if they exist), the Object x,y size and position values are scaled to the secondary representations picture dimensions with a linear scale factor.
Each representation may be generated with different ROI bit depth mappings. These mappings typically increase detail in dark or light areas compared to the overall intensity of the video frame size.
A client device by receiving an MPD can represent the ROI on the user interface in a variety of ways. The Supplemental Property of the of an adaption set indicates to the client how many ROIs are available. Object_UserData may be used by the client to display information describing aspects of the ROI. For example, in a sporting event this can be specific player information.
If more than one ROI is available with different mappings, then combinations of different ROI mappings may also be available. For example, with two ROIs, this would result in four representations: default (no remappings), only ROI #1 remapped, only ROI #2 remapped, and both ROIs remapped. If each ROI has multiple remappings available, the number of representations increases.
In addition to the different mappings for the ROIs, the overall video frame mapping can be optimized for specific viewing conditions (bright room, dark room, etc.) or for specific viewing devices such as LED, CRT, Plasma, or other screen types. Viewing condition and ViewingDeviceType parameters can be transmitted for each representation and in combination with different ROI mappings.
The following is an example of an MPD that may be used in an embodiment in which there is one ROI with one available remapping:
The following is an example of an MPD that may be used in an embodiment in which Two ROIs are available. In this example, eight total representations are available: four remapping variations (default, remapping of ROI #1 only, remapping of ROI #2 only, and remapping of both ROI #1 and ROI #2) are each available at two different resolutions (1080p and 720p):
Exemplary Region of Interest Information.
Various techniques may be employed to convey information regarding the availability of different regions of interest and different tone mappings available for those regions of interest. the information may be provided in-band (e.g., in the user data of the video frame within encoded media segments delivered to the client), or, e.g., in a separate region of interest metadata file retrievable by the client. The region of interest parameters described here may be provided periodically or on a frame-by-frame basis, to support the region of interest moving spatially over time. The following exemplary parameters may be used to provide data regarding regions of interest.
Object x,y position and size may be provided in pixel units that correspond to the first-listed representation in the appropriate adaption set. For secondary representations (if they exist), the Object x,y size and position values are scaled to the secondary representations picture dimensions with a linear scale factor. In some embodiments, the dimensions and boundaries of the region of interest are determined and delivered to the client in real time.
A client device that receives region of interest data can represent the region of interest on the user interface in a variety of ways. For example, Object_UserData may be used by the client to display information describing aspects of the region of interest. For example, in a sporting event, this can be specific player information.
Server Configuration of Remappings.
In an exemplary embodiment, the server informs the client about the availability of both a standard bit depth (e.g., 8 bit) version and a high bit depth version (e.g., 12 bit) of the video content, e.g., in a manifest such as an MPEG-DASH MPD. The manifest may identify a plurality of representations for both the standard-bit-depth and the high-bit-depth versions of the content (with different representations having, for example, different bitrates). In an exemplary architecture, tracking of a region of interest (including an object of interest) is performed at the server. This information is carried inband to the client. In addition, the server generates tone mapping parameters for the regions of interest. The tone mapping parameters may be generated with the objective of improving visibility under the viewing conditions at the client device and may be based on information regarding the viewing parameters as reported by the client device to the server.
These tone mapping parameters are communicated to the client device. The resources of the client device thus need not be dedicated to functions of object tracking or generation/selection of tone remapping parameters, as these functions are performed at the server. The tone remapping itself (using the parameters received from the server) is performed in some embodiments by the client device.
In an exemplary embodiment, parameters used for the tone mapping of an individual region of interest are communicated from the server to the client using the syntax of an H.264 supplemental enhancement information (SEI) message, such as a “Tone mapping information SEI” message where different types of models and parameters are supported as signaled by the syntax element tone_map_model_id, as described in ITU-T H.264 and ISO/IEC 14496-10, Advanced Video Coding for Generic Audiovisual Services MPEG-4 AVC, 2014. In that syntax element, a tone_map_model_id value of 0 corresponds to a linear mapping with clipping; a tone_map_model_id value of 1 corresponds to a sigmoidal mapping; a tone_map_model_id value of 2 corresponds to a user-defined table mapping, a tone_map_model_id value of 3 corresponds to a piece-wise linear mapping. Each of these models has associated parameters that are also included (e.g., included in the communication from the server to the client) to fully describe a tone map. The linear map with clipping is defined by giving maximum and minimum values. The sigmoidal model, S-curve, uses a midpoint, sigmoid_midpoint, and a width, sigmoid_width, together with the formula:
The model value 2 allows a full user-defined lookup table, while the model value 3 defines a piecewise linear mapping, where the pieces are defined by pivot points which define each linear piece. Other sets of parameters may alternatively be used.
A client device (e.g., an application program running on a client) may receive a manifest (e.g., a DASH MPD) which contains a presentation description advertising one or more alternative bit depth representations. The client may parse the description, and may request, retrieve, and display the content of an available content representation. The client may request and retrieve the full depth version of the stream from the server (using the appropriate MPD description). The server sends to the client (for example on a per-frame basis) information identifying the positions of tracked regions of interest and tone remapping parameters. For example, each tracked region of interest may be associated with one or more tone remapping functions, where each tone remapping function may be represented by a set of tone remapping parameters. The client then performs an appropriate tone remapping function using the remapping parameters on that region of interest. In some embodiments, the tone remapping function may convert the full depth version to a lower bit depth suitable for the bit depth which the client (or a display device connected to the client) is capable of displaying.
In some embodiments, the client requests the full depth version of the content and performs an appropriate bit-depth remapping in response to user input, or based on internal client logic. The bit-depth remapping applied by the client may be suitable for (e.g., optimized for) a region of interest. The bit-depth remapping may be adapted for (e.g., optimized for) a particular display technology. The bit-depth remapping may be adapted for (e.g., optimized for) a set of viewing conditions.
Exemplary Region of Interest (ROI) Tone Remapping System and Method.
An illustration of exemplary operation and use cases for systems and methods disclosed herein is provided with reference to
An exemplary method for generating video representations with tone remapping of ROIs is illustrated in
As illustrated in
As shown in
Data that associates remappings with different slices may take the form of the data in Tables 1-4.
In step 916, when a client request for a particular tone remapping is received, the data that associates slices to remappings is used to reconstruct a full video frame in which the requested regions of interest are remapped. As illustrated in
Video Delivery Method for Server-Side Remapping.
The content source transmits a compressed or uncompressed media stream of the source media to an encoder at a high bit depth (1316). The encoder separately creates ABR streams with default tone mappings (1318) and ABR streams with alternative tone remappings in various regions of interest or combinations thereof (1320). (For a given tone mapping, the various ABR streams may represent different encoding points which may vary in properties such as bit rate and/or resolution). The various ABR streams with both the default tone mapping and alternative ROI tone mappings are transmitted to the transport packager 1306. The transport packager may segment the files and make the files available via an ftp or http download and prepare a manifest.
Note that the content preparation entities and steps shown in
A client device 1314 may transmit a signal (1322) to a web server 1312 requesting to download the media content and may receive a streaming server redirect signal (1324). The client device may request (1326) a manifest which describes the available content files (e.g., media segment files). The request may be sent from the client to a server. The server (e.g., origin server or an edge streaming server) may deliver the manifest file (1328) in response to the client request. The manifest may indicate availability of the various ABR streams with default tone maps and with alternative tone maps in regions of interest.
Initially, the client may request (1330) a default stream from a streaming server, and the streaming server may responsively transmit the default stream (e.g., media segments of that default stream) to the client device (1332). The client device may display the default stream (1334).
The client device may detect a cue to request an alternate representation of the content with highlighting in one or more regions of interest. For example, the cue may be user input wherein the user selects one or more regions of interest for highlighting. The cue may be detection by the client of a viewing condition or a change in viewing conditions for which the client has appropriate logic to convert a higher bit depth representation to a lower bit depth representation using a remapping appropriate for the current viewing conditions. In response to the client device detecting a cue to request a stream with an alternate tone mapping, the client device requests (1336) the appropriately remapped stream from the streaming server, and the streaming server responsively streams the requested stream (1338) to the client device.
The client device may continue to receive and display (1342) the default stream (1340) for a time period to allow additional buffering (1344) of the alternative stream to reduce playback latency. The alternative stream is displayed (1346) when a sufficient amount of the content is received and buffered at the client.
Video Delivery Method for Client-Side Remapping.
In some embodiments, users are provided with the ability to select which type or type of remapping are to be applied to regions of interest selected by the user. For example, users with poor vision may select a tone remapping that leads to an exceptionally high luminance within the region of interest. While this may make particular details difficult to discern, it may be easier for such a user to track, e.g., a game ball as it moves across a playing field. Users with better vision may select a subtler highlighting effect. An overview of one such method for user configuration of a remapping is illustrated in
In the embodiment of
A server receives the video and may, in some embodiments, apply a default tone map 1412 to the video. The default tone map may be a default bit map to map the source video from high bit depth to a standard bit depth (e.g., 8 bit) suitable for use by a display device. The standard bit depth video 1414 is delivered to a client device 1416 (e.g., through a content distribution network), which decodes the video and causes the video to be displayed on a display 1418 (e.g., by sending uncompressed video over an HDMI cable or other connection).
The server may embed the description 1420 of the objects of interest in-band in the video delivered to the client device (e.g., in the user data of the video frame within encoded media segments delivered to the client). The client device may, e.g., in response to user input, select one or more of the regions of interest. In some embodiments, the selection of a region of interest is conveyed by the client device to the server (1422). In such cases, the server may provide the client device with appropriate remapping parameters (1424) for the region of interest in response to the server receiving the client's region of interest selection information. In other embodiments, the client's selection is not conveyed to the server, and the server may provide the client device with appropriate remapping parameters (e.g., for each region of interest) regardless of whether any particular region of interest has been selected to undergo tone remapping. The client device may further convey information regarding viewing conditions (1426) and/or display device type at the client device. Different viewing conditions include the ambient lighting, such as a dark ambient light or a bright ambient light as detected by the client device. Different viewing device types may include a plasma display, an LCD display, an LED display, an OLED display, and the like, and different makes and models of those viewing devices.
Based on the viewing conditions at the client device, the server may generate tone mapping parameters (1428) that may be used by the client device for tone remapping of the region of interest. Various techniques may be used for selecting a suitable (e.g., optimal) tone remapping function for the pixel values in a region of interest. For example, a histogram may be generated from the pixel values which fall within a given region of interest, and a suitable (or optimal) conversion function may be generated based on the histogram and on ambient illumination conditions. Exemplary techniques that may be used for generation of a conversion function from histogram data for different levels of ambient illumination are described in, for example, R. Mantuik et al., Display Adaptive Tone Mapping, 27 (3) A
In some embodiments, the server generates a plurality of tone map parameter sets for each region of interest, with each parameter set being associated with a different viewing condition. The respective parameter sets may be tagged with information identifying each viewing condition. For example, a video stream may be accompanied by a plurality of parameter sets for a particular region of interest, with different parameter sets being tagged for use under different lighting conditions. In such embodiments, in response to a user selecting the region of interest for highlighting, the client device selects the parameter set associated with current local lighting conditions and applies the tone map defined by that parameter set to the selected region of interest. In such embodiments, the client device may not report the local viewing conditions to the server. In such embodiments, the server may apply the ‘generate tone map parameters’ step earlier than shown in
The server delivers the generated tone mapping parameters to the client device (1424). The tone mapping parameters may be provided on a frame-by-frame basis and may be provided in-band (e.g., in the user data of the video frame within encoded media segments delivered to the client), or in a separate ROI metadata file retrievable by the client. Alternately the tone mapping parameters may be provided in a manifest file (e.g., a DASH MPD). The tone mapping parameters may be retrieved from the same server as the video stream (e.g., from a server in a content distribution network) or from a different server. In some embodiments, tone mapping parameters are updated less frequently than with every video frame. In some embodiments, the server generates and delivers to the client only tone mapping parameters for one or more regions of interest that the client has reported to the server as having been selected. In other embodiments, the client may not report to the server whether any particular region of interest has been selected, and the server may provide appropriate parameters for all regions of interest (even though some or all of those parameters may go unused by the client device). The client device receives the tone map parameters from the server and, if one or more regions of interest has been selected, applies the tone map to the appropriate region of interest (1430). The client device then causes the video (1432) with the remapped region (or regions) of interest to be displayed to the user.
Exemplary Client and Server Hardware.
Note that various hardware elements of one or more of the described embodiments are referred to as “modules” that carry out (i.e., perform, execute, and the like) various functions that are described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable by those of skill in the relevant art for a given implementation. Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and it is noted that those instructions may take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and/or the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM, ROM, etc.
Exemplary embodiments disclosed herein are implemented using one or more wired and/or wireless network nodes, such as a wireless transmit/receive unit (WTRU) or other network entity.
The processor 1518 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 1518 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the WTRU 1502 to operate in a wireless environment. The processor 1518 may be coupled to the transceiver 1520, which may be coupled to the transmit/receive element 1522. While
The transmit/receive element 1522 may be configured to transmit signals to, or receive signals from, a base station over the air interface 1516. For example, in one embodiment, the transmit/receive element 1522 may be an antenna configured to transmit and/or receive RF signals. In another embodiment, the transmit/receive element 1522 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, as examples. In yet another embodiment, the transmit/receive element 1522 may be configured to transmit and receive both RF and light signals. It will be appreciated that the transmit/receive element 1522 may be configured to transmit and/or receive any combination of wireless signals.
In addition, although the transmit/receive element 1522 is depicted in
The transceiver 1520 may be configured to modulate the signals that are to be transmitted by the transmit/receive element 1522 and to demodulate the signals that are received by the transmit/receive element 1522. As noted above, the WTRU 1502 may have multi-mode capabilities. Thus, the transceiver 1520 may include multiple transceivers for enabling the WTRU 1502 to communicate via multiple RATs, such as UTRA and IEEE 802.11, as examples.
The processor 1518 of the WTRU 1502 may be coupled to, and may receive user input data from, the speaker/microphone 1524, the keypad 1526, and/or the display/touchpad 1528 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 1518 may also output user data to the speaker/microphone 1524, the keypad 1526, and/or the display/touchpad 1528. In addition, the processor 1518 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 1530 and/or the removable memory 1532. The non-removable memory 1530 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 1532 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 1518 may access information from, and store data in, memory that is not physically located on the WTRU 1502, such as on a server or a home computer (not shown).
The processor 1518 may receive power from the power source 1534, and may be configured to distribute and/or control the power to the other components in the WTRU 1502. The power source 1534 may be any suitable device for powering the WTRU 1502. As examples, the power source 1534 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), and the like), solar cells, fuel cells, and the like.
The processor 1518 may also be coupled to the GPS chipset 1536, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 1502. In addition to, or in lieu of, the information from the GPS chipset 1536, the WTRU 1502 may receive location information over the air interface 1516 from a base station and/or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 1502 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.
The processor 1518 may further be coupled to other peripherals 1538, which may include one or more software and/or hardware modules that provide additional features, functionality and/or wired or wireless connectivity. For example, the peripherals 1538 may include sensors such as an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, and the like.
Communication interface 1692 may include one or more wired communication interfaces and/or one or more wireless-communication interfaces. With respect to wired communication, communication interface 1692 may include one or more interfaces such as Ethernet interfaces, as an example. With respect to wireless communication, communication interface 1692 may include components such as one or more antennae, one or more transceivers/chipsets designed and configured for one or more types of wireless (e.g., LTE) communication, and/or any other components deemed suitable by those of skill in the relevant art. And further with respect to wireless communication, communication interface 1692 may be equipped at a scale and with a configuration appropriate for acting on the network side—as opposed to the client side—of wireless communications (e.g., LTE communications, Wi-Fi communications, and the like). Thus, communication interface 1692 may include the appropriate equipment and circuitry (perhaps including multiple transceivers) for serving multiple mobile stations, UEs, or other access terminals in a coverage area.
Processor 1694 may include one or more processors of any type deemed suitable by those of skill in the relevant art, some examples including a general-purpose microprocessor and a dedicated DSP.
Data storage 1696 may take the form of any non-transitory computer-readable medium or combination of such media, some examples including flash memory, read-only memory (ROM), and random-access memory (RAM) to name but a few, as any one or more types of non-transitory data storage deemed suitable by those of skill in the relevant art could be used. As depicted in
Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.
This application is a Continuation of U.S. application Ser. No. 16/314,846, filed Jan. 2, 2019, which is a National Phase entry under 35 U.S.C. § 371 of International Application No. PCT/US2017/041142, filed Jul. 7, 2017, which claims the benefit of U.S. Provisional Patent Application Ser. No. 62/365,863, filed Jul. 22, 2016, and U.S. Provisional Patent Application Ser. No. 62,360,105, filed Jul. 8, 2016, which are incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 934888 | Charles | Sep 1909 | A |
| 5610653 | Abecassis | Mar 1997 | A |
| 5892554 | Dicicco et al. | Apr 1999 | A |
| 6282713 | Kitsukawa et al. | Aug 2001 | B1 |
| 6750919 | Rosser | Jun 2004 | B1 |
| 6973130 | Wee et al. | Dec 2005 | B1 |
| 7079176 | Freeman et al. | Jul 2006 | B1 |
| 7114174 | Brooks et al. | Sep 2006 | B1 |
| 7117517 | Milazzo et al. | Oct 2006 | B1 |
| 7343617 | Katcher et al. | Mar 2008 | B1 |
| 7577980 | Kienzle et al. | Aug 2009 | B2 |
| 7782344 | Whittaker | Aug 2010 | B2 |
| 7949032 | Frost | May 2011 | B1 |
| 8055785 | Liu et al. | Nov 2011 | B2 |
| 8160077 | Traversat et al. | Apr 2012 | B2 |
| 8243797 | Lin et al. | Aug 2012 | B2 |
| 8331760 | Butcher | Dec 2012 | B2 |
| 8621000 | Adimatyam et al. | Dec 2013 | B2 |
| 8705527 | Addepalli et al. | Apr 2014 | B1 |
| 9047236 | Gigliotti | Jun 2015 | B2 |
| 9060187 | Wu et al. | Jun 2015 | B2 |
| 9064313 | Seshadrinathan et al. | Jun 2015 | B2 |
| 9137027 | Matthews et al. | Sep 2015 | B2 |
| 9137558 | Gibbon et al. | Sep 2015 | B2 |
| 9170707 | Laska et al. | Oct 2015 | B1 |
| 9177225 | Cordova-Diba et al. | Nov 2015 | B1 |
| 9218641 | Ellis | Dec 2015 | B1 |
| 9231838 | Ould-Brahim et al. | Jan 2016 | B2 |
| 9288545 | Hill et al. | Mar 2016 | B2 |
| 9317688 | Yegin et al. | Apr 2016 | B2 |
| 9336060 | Nori et al. | May 2016 | B2 |
| 9344888 | Starsinic et al. | May 2016 | B2 |
| 9380623 | Kim et al. | Jun 2016 | B2 |
| 9515372 | Chang et al. | Dec 2016 | B2 |
| 9992553 | Bennett et al. | Jun 2018 | B2 |
| 10152826 | Saito et al. | Dec 2018 | B2 |
| 20020056136 | Wistendahl et al. | May 2002 | A1 |
| 20020059588 | Huber et al. | May 2002 | A1 |
| 20020065678 | Peliotis et al. | May 2002 | A1 |
| 20020120931 | Huber et al. | Aug 2002 | A1 |
| 20020120934 | Abrahams | Aug 2002 | A1 |
| 20020126990 | Rasmussen et al. | Sep 2002 | A1 |
| 20020131511 | Zenoni | Sep 2002 | A1 |
| 20020133817 | Markel | Sep 2002 | A1 |
| 20020147987 | Reynolds et al. | Oct 2002 | A1 |
| 20020174425 | Markel et al. | Nov 2002 | A1 |
| 20030028873 | Lemmons | Feb 2003 | A1 |
| 20030149983 | Markel | Aug 2003 | A1 |
| 20040031062 | Lemmons | Feb 2004 | A1 |
| 20040032495 | Ortiz | Feb 2004 | A1 |
| 20040190779 | Sarachik et al. | Sep 2004 | A1 |
| 20050093976 | Valleriano et al. | May 2005 | A1 |
| 20050286466 | Tagg et al. | Dec 2005 | A1 |
| 20060193295 | White et al. | Aug 2006 | A1 |
| 20070024706 | Brannon et al. | Feb 2007 | A1 |
| 20070042784 | Anderson et al. | Feb 2007 | A1 |
| 20070077943 | Hamilla | Apr 2007 | A1 |
| 20070086669 | Berger et al. | Apr 2007 | A1 |
| 20070101418 | Wood et al. | May 2007 | A1 |
| 20070104369 | Weatherhead | May 2007 | A1 |
| 20070153739 | Zheng | Jul 2007 | A1 |
| 20070169161 | Kienzle et al. | Jul 2007 | A1 |
| 20070171921 | Wookey et al. | Jul 2007 | A1 |
| 20070226761 | Zalewski et al. | Sep 2007 | A1 |
| 20080077965 | Kamimaki et al. | Mar 2008 | A1 |
| 20080127253 | Zhang et al. | May 2008 | A1 |
| 20080134305 | Hinton et al. | Jun 2008 | A1 |
| 20080225116 | Kang et al. | Sep 2008 | A1 |
| 20080261711 | Tuxen | Oct 2008 | A1 |
| 20090002564 | Barnhoefer et al. | Jan 2009 | A1 |
| 20090003718 | Liu | Jan 2009 | A1 |
| 20090007023 | Sundstrom | Jan 2009 | A1 |
| 20090046152 | Aman et al. | Feb 2009 | A1 |
| 20090047902 | Nory et al. | Feb 2009 | A1 |
| 20090201917 | Maes et al. | Aug 2009 | A1 |
| 20090217339 | Kim et al. | Aug 2009 | A1 |
| 20090220080 | Herne et al. | Sep 2009 | A1 |
| 20090225164 | Renkis | Sep 2009 | A1 |
| 20090276805 | Andrews et al. | Nov 2009 | A1 |
| 20090300692 | Mavlankar et al. | Dec 2009 | A1 |
| 20090305708 | Matsui et al. | Dec 2009 | A1 |
| 20090316795 | Chui et al. | Dec 2009 | A1 |
| 20090322489 | Jones et al. | Dec 2009 | A1 |
| 20100097221 | Kreiner et al. | Apr 2010 | A1 |
| 20100188531 | Cordes et al. | Jul 2010 | A1 |
| 20100232504 | Feng | Sep 2010 | A1 |
| 20100321389 | Gay et al. | Dec 2010 | A1 |
| 20110013836 | Gefen et al. | Jan 2011 | A1 |
| 20110016321 | Sundaram et al. | Jan 2011 | A1 |
| 20110113480 | Ma et al. | May 2011 | A1 |
| 20110145430 | Ha et al. | Jun 2011 | A1 |
| 20110231653 | Bollay et al. | Sep 2011 | A1 |
| 20110299832 | Butcher | Dec 2011 | A1 |
| 20110304772 | Dasher et al. | Dec 2011 | A1 |
| 20120030637 | Dey et al. | Feb 2012 | A1 |
| 20120047551 | Pattar et al. | Feb 2012 | A1 |
| 20120056898 | Tsurumi et al. | Mar 2012 | A1 |
| 20120062732 | Marman et al. | Mar 2012 | A1 |
| 20120072952 | Vaysman et al. | Mar 2012 | A1 |
| 20120078712 | Fontana et al. | Mar 2012 | A1 |
| 20120117140 | Wang et al. | May 2012 | A1 |
| 20120195574 | Wallace et al. | Aug 2012 | A1 |
| 20120308202 | Murata et al. | Dec 2012 | A1 |
| 20130003576 | Segura et al. | Jan 2013 | A1 |
| 20130016910 | Murata et al. | Jan 2013 | A1 |
| 20130031582 | Tinsman et al. | Jan 2013 | A1 |
| 20130036442 | Wingert | Feb 2013 | A1 |
| 20130050268 | Lohrenz et al. | Feb 2013 | A1 |
| 20130061262 | Briggs et al. | Mar 2013 | A1 |
| 20130065622 | Hwang | Mar 2013 | A1 |
| 20130091430 | Zhai et al. | Apr 2013 | A1 |
| 20130091515 | Sakata et al. | Apr 2013 | A1 |
| 20130142118 | Cherian et al. | Jun 2013 | A1 |
| 20130155106 | Rolleston et al. | Jun 2013 | A1 |
| 20130155847 | Li et al. | Jun 2013 | A1 |
| 20130163507 | Hoshino et al. | Jun 2013 | A1 |
| 20130167062 | Herring et al. | Jun 2013 | A1 |
| 20130185353 | Rondao et al. | Jul 2013 | A1 |
| 20130188503 | Anepu et al. | Jul 2013 | A1 |
| 20130188515 | Pinheiro et al. | Jul 2013 | A1 |
| 20130208784 | Pietila | Aug 2013 | A1 |
| 20130223339 | Nakahara | Aug 2013 | A1 |
| 20130246784 | Zhang et al. | Sep 2013 | A1 |
| 20130266286 | Yu et al. | Oct 2013 | A1 |
| 20130294506 | Kang et al. | Nov 2013 | A1 |
| 20130312042 | Shaw et al. | Nov 2013 | A1 |
| 20130336305 | Yan et al. | Dec 2013 | A1 |
| 20130342539 | Khan et al. | Dec 2013 | A1 |
| 20140022074 | Balinski et al. | Jan 2014 | A1 |
| 20140026052 | Thorwirth et al. | Jan 2014 | A1 |
| 20140029462 | Stewart | Jan 2014 | A1 |
| 20140033074 | Thibaux et al. | Jan 2014 | A1 |
| 20140045452 | Ma et al. | Feb 2014 | A1 |
| 20140059457 | Min | Feb 2014 | A1 |
| 20140082054 | Denoual et al. | Mar 2014 | A1 |
| 20140086217 | Park et al. | Mar 2014 | A1 |
| 20140089990 | Van et al. | Mar 2014 | A1 |
| 20140168056 | Swaminathan et al. | Jun 2014 | A1 |
| 20140192204 | Glazer | Jul 2014 | A1 |
| 20140195918 | Friedlander | Jul 2014 | A1 |
| 20140204206 | Itoi et al. | Jul 2014 | A1 |
| 20140204247 | Bilgen et al. | Jul 2014 | A1 |
| 20140210714 | Kang et al. | Jul 2014 | A1 |
| 20140215043 | Ryu et al. | Jul 2014 | A1 |
| 20140245367 | Sasaki et al. | Aug 2014 | A1 |
| 20140269581 | Song et al. | Sep 2014 | A1 |
| 20140269932 | Su et al. | Sep 2014 | A1 |
| 20140279068 | Systrom et al. | Sep 2014 | A1 |
| 20140280781 | Gregotski et al. | Sep 2014 | A1 |
| 20140282751 | Lee et al. | Sep 2014 | A1 |
| 20140282909 | Cherian et al. | Sep 2014 | A1 |
| 20150014773 | Cheng et al. | Jan 2015 | A1 |
| 20150016504 | Auyeung et al. | Jan 2015 | A1 |
| 20150032901 | Wang et al. | Jan 2015 | A1 |
| 20150033312 | Seed et al. | Jan 2015 | A1 |
| 20150062339 | Ostrom | Mar 2015 | A1 |
| 20150070587 | Emeott et al. | Mar 2015 | A1 |
| 20150089023 | Phillips et al. | Mar 2015 | A1 |
| 20150089072 | Phillips et al. | Mar 2015 | A1 |
| 20150104155 | Bloch et al. | Apr 2015 | A1 |
| 20150117229 | Avudainayagam et al. | Apr 2015 | A1 |
| 20150145782 | Brown et al. | May 2015 | A1 |
| 20150172775 | Yee et al. | Jun 2015 | A1 |
| 20150179219 | Gao et al. | Jun 2015 | A1 |
| 20150201222 | Mertens | Jul 2015 | A1 |
| 20150208103 | Guntur et al. | Jul 2015 | A1 |
| 20150222815 | Wang et al. | Aug 2015 | A1 |
| 20150264096 | Swaminathan et al. | Sep 2015 | A1 |
| 20150268822 | Waggoner et al. | Sep 2015 | A1 |
| 20150296250 | Casper | Oct 2015 | A1 |
| 20150304693 | Hwang et al. | Oct 2015 | A1 |
| 20150341091 | Park et al. | Nov 2015 | A1 |
| 20150348247 | Mclaughlin et al. | Dec 2015 | A1 |
| 20150350659 | Auyeung et al. | Dec 2015 | A1 |
| 20150382268 | Hampel et al. | Dec 2015 | A1 |
| 20160042496 | Ichimi et al. | Feb 2016 | A1 |
| 20160057494 | Iwang et al. | Feb 2016 | A1 |
| 20160073106 | Su et al. | Mar 2016 | A1 |
| 20160156949 | Hattori et al. | Jun 2016 | A1 |
| 20160165309 | Van et al. | Jun 2016 | A1 |
| 20160182593 | Denoual et al. | Jun 2016 | A1 |
| 20160192433 | Thangaraj et al. | Jun 2016 | A1 |
| 20160021762 | Singh | Jul 2016 | A1 |
| 20160217623 | Singh | Jul 2016 | A1 |
| 20160227297 | Bennett et al. | Aug 2016 | A1 |
| 20160249269 | Niu et al. | Aug 2016 | A1 |
| 20160249353 | Nakata et al. | Aug 2016 | A1 |
| 20160253240 | Cocagne et al. | Sep 2016 | A1 |
| 20160269771 | Bangma et al. | Sep 2016 | A1 |
| 20160306036 | Johnson | Oct 2016 | A1 |
| 20160337706 | Hwang et al. | Nov 2016 | A1 |
| 20160360172 | Tao et al. | Dec 2016 | A1 |
| 20160360235 | Ramasubramonian et al. | Dec 2016 | A1 |
| 20160364087 | Thompson et al. | Dec 2016 | A1 |
| 20160366454 | Tatourian et al. | Dec 2016 | A1 |
| 20170070735 | Ramasubramonian et al. | Mar 2017 | A1 |
| 20170118540 | Thomas et al. | Apr 2017 | A1 |
| 20170136948 | Sypitkowski et al. | May 2017 | A1 |
| 20170201686 | Choi et al. | Jul 2017 | A1 |
| 20170212583 | Krasadakis | Jul 2017 | A1 |
| 20170236288 | Sundaresan et al. | Aug 2017 | A1 |
| 20170302719 | Chen et al. | Oct 2017 | A1 |
| 20180098063 | Chen et al. | Apr 2018 | A1 |
| 20180332632 | Vikberg et al. | Nov 2018 | A1 |
| 20180343043 | Hakola et al. | Nov 2018 | A1 |
| 20190036910 | Choyi et al. | Jan 2019 | A1 |
| 20190320189 | Cooper et al. | Oct 2019 | A1 |
| 20200014961 | Ramaswamy et al. | Jan 2020 | A1 |
| 20210130629 | Suzuki et al. | May 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| 101313578 | Nov 2008 | CN |
| 101350923 | Jan 2009 | CN |
| 102752877 | Oct 2012 | CN |
| 102907068 | Jan 2013 | CN |
| 103190089 | Jul 2013 | CN |
| 104012106 | Aug 2014 | CN |
| 104052788 | Sep 2014 | CN |
| 105103566 | Nov 2015 | CN |
| 105144727 | Dec 2015 | CN |
| 105264904 | Jan 2016 | CN |
| 106023261 | Oct 2016 | CN |
| 106233745 | Dec 2016 | CN |
| 106416223 | Feb 2017 | CN |
| 1175785 | Jan 2002 | EP |
| 1443768 | Aug 2004 | EP |
| 2408196 | Jan 2012 | EP |
| 2426645 | Mar 2012 | EP |
| 2530675 | Dec 2012 | EP |
| 2824883 | Jan 2015 | EP |
| 2824885 | Jan 2015 | EP |
| 2919471 | Sep 2015 | EP |
| 3076672 | Oct 2016 | EP |
| 3110160 | Dec 2016 | EP |
| 103336663 | Oct 2013 | IN |
| 2004104351 | Apr 2004 | JP |
| 2007192988 | Aug 2007 | JP |
| 2009207114 | Sep 2009 | JP |
| 2013522965 | Jun 2013 | JP |
| 2014525191 | Sep 2014 | JP |
| 6313503 | Apr 2018 | JP |
| 20100065327 | Jun 2010 | KR |
| 20140043532 | Apr 2014 | KR |
| 20140098156 | Aug 2014 | KR |
| 20160021141 | Feb 2016 | KR |
| 20170128515 | Nov 2017 | KR |
| WO0118658 | Mar 2001 | WO |
| WO2011082150 | Jul 2011 | WO |
| WO2012114666 | Aug 2012 | WO |
| WO2012141557 | Oct 2012 | WO |
| WO2013142966 | Oct 2013 | WO |
| WO2014067566 | May 2014 | WO |
| WO2014111423 | Jul 2014 | WO |
| WO2014183803 | Nov 2014 | WO |
| WO2015013645 | Jan 2015 | WO |
| WO2015013685 | Jan 2015 | WO |
| WO2015014773 | Feb 2015 | WO |
| WO2015059194 | Apr 2015 | WO |
| WO2015122737 | Aug 2015 | WO |
| WO2015130796 | Sep 2015 | WO |
| WO2015197815 | Dec 2015 | WO |
| WO2016040833 | Mar 2016 | WO |
| WO2016098056 | Jun 2016 | WO |
| WO2017061297 | Apr 2017 | WO |
| WO2017196670 | Nov 2017 | WO |
| Entry |
|---|
| A-Yong et al., “Research on Tower Crane Automatic Tracking and Zooming Video Monitoring System based on Improved Camshift algorithm”, 2011 International Conference on Electric Information and Control Engineering, Wuhan, 2011, 4 pages. |
| Taylor, T. “NFL using Zebra RFID Chips to Track Player Movements, Gather Data”. Sports Illustrated article,available at: http://web.archive.org/web20160703053654/hllp://www.si.corn:80/, updated Mar. 6, 2015, 4 pages. |
| Zebra, “Zebra and the NFL: Change the Game”. Zebra.com article, available at: http://web.archive.org/web/20160604013821/https://www.zebra.com/us/en/nfl.html, web archive dated Jun. 4, 2016, 4 pages. |
| Sheikh Y., et al., “Geodetic Alignment of Aerial Video Frames”. In Shah M., Kumar R. (eds) Video Registration, Chapter 7, (2003), pp. 141-177. |
| “NASCAR Pointer Screen Capture”, retrieved from video available at: http://www.sportvision.com/media/pointers,on Jul. 8, 2016, 1 page. |
| Aberdeen, “Case Study: FCC Challenges Video Editors to Make Room for Caption Placement”, Aug. 4, 2015, 3 pages. |
| Richardson, Iain E., el. al., “The H.264 Advanced Video Compression Standard, Chapter 10: Extensions and Directions”. John Wiley, Second Edition, (2010), pp. 287-311. |
| Notification of Transmittal of the International Search Report and the Written Opinion of the International Searching Authority, or the Declaration for PCT/US2017/031335 dated Jul. 19, 2017, 16 pages. |
| Giladi, A. et. al., “Use Cases for Conveying Source Information”. International Organization for Standardization Coding of Moving Pictures and Audio, ISO/IEC JTC1/SC29/WG11, m31594, Oct. 2013, 3 pages. |
| Liu S., et al., “SVC Inter-Layer Prediction for SVC Bit-Depth Scalability” _ Joint Video Team Of ISO/IEC JTC1/SC29/ G11 and ITU-T SG.1624, No. JVT-X075, Jun. 2007, 12 pages. |
| International Search Report and Written Opinion of the International Searching Authority for PCT/US2017/041142 dated Oct. 11, 2017. |
| Written Opinion of the International Preliminary Examining Authority for PCT/US2017/031335 dated Apr. 5, 2018. |
| Written Opinion of the International Preliminary Examining Authority for PCT/US2017/041142 dated Jun. 20, 2018. |
| International Preliminary Report on Palenlability for PCT/US2017/031335 completed on Jun. 26, 2018. |
| International Preliminary Report on Patentability for PCT/US2017/041142 completed on Sep. 19, 2018. |
| Mantiuk, R., el. al., “Display Adaptive Tone Mapping” _ ACM Transactions on Graphics (TOG), vol. 27, No. 3, Aug. 2008 at available at: http://resources.mpi-inf.mpg.de/hdr/datmo/mantiuk08datm.pdf, 10 pages. |
| Mavlankar, A. et. al., “Video Streaming with Interactive Pan!Till/Zoom”, Oct. 2009. Available at: http://web. tanford.edu/-bgirod/pdfs/Mavlankar_Girod_Chapter_Oct09.pdf. |
| Unifore Security, “What's ROI (Region of Interest) Video Surveillance Technology?”. Unifore.net, Oct. 23, 015, 4 pages. |
| Mavlankar, A., el. al., “Peer-To-Peer Multicast Live Video Streaming with Interactive Virtual Pan/Tilt/Zoom unctionality” _ In 2008 15th IEEE International Conference on Image Processing, Oct. 2008, pp. 2296-2299. |
| Chen, Y., et. al., “Intra-And-Inter-Constraint-Based Video Enhancement Based On Piecewise Tone Mapping” _ IEEE ransactions on Circuits and Systems for Video Technology, vol. 23, No. 1, Jan. 2013, pp. 74-82. |
| Vehkapera, J., el. al., “Replacing Picture Regions in H. 264/AVC Bitstream By Utilizing Independent Slices”. roceedings of 2010 IEEE 17th International Conference on Image Processing, Sep. 26-29, 2010, pp. 397-3400. |
| Classxteam, “ClassX Mobile”. You Tube Link, published on May 8, 2011, available at: https://www.youtube.com/watch?v=Kul-0Cw4hj8. |
| International Telecommunication Union, “Advanced Video Coding for Generic Audiovisual Services”. Series H: Audiovisual and Multimedia Systems, ITU-T, H.264, Feb. 2014, 790 pages. |
| International Organization for Standardization, “Information Technology—Dynamic Adaptive Streaming Over HTTP {DASH), Part 1: Media Presentation Description and Segment Formats”. International Standard, ISO/IEC 23009-1, Second Edition, May 15, 2014, 152 pages. |
| Lavigne, F., “Automatic Video Zooming for Sport Team Video Broadcasting on Smart Phones”. Available at: http://tcts.fpms.ac.be/publications/papers/2010/visapp2010_flfcxd.pdf, (2010), 7 pages. |
| Malavlankar, A., et al., “An Interactive Region-of-Interest Video Streaming System for Online Lecture Viewing” _ troceedings of 2010 IEEE 18th International Packet Video Workshop, Dec. 13-14, 2010, pp. 64-71. |
| International Search Report and The Written Opinion of the International Searching Authority for PCT/US2016/046317, dated Oct. 19, 2016, 11 pages. |
| International Preliminary Report on Palenlability for PCT/US2016/046317 dated Mar. 1, 2018. |
| Mavlankar, A., el. al., “Optimal Slice Size for Streaming Regions of High Resolution Video with Virtual Pan!Till/ foam Functionality”, Jun. 2007. Available at: http://web.stanford.edu/-bgirod/pdfsMavlankarEUSIPC007.pdf. |
| Written Opinion of the International Preliminary Examining Authority for PCT/US2016/053512 dated Aug. 28, 2017. |
| International Preliminary Report on Patentability, Corrected Version, for PCT/US2016/053512 completed Jan. 15, 018. |
| International Preliminary Report on Patentability for PCT/US2016/053512 completed Jan. 15, 2018. |
| Internatinal Search Report and Written Opinion of the International Searching Authority for PCT/US2016/053512, dated Nov. 28, 2016, 10 pages. |
| Yilmaz, A., el. al., “Object Tracking—A Survey” _ ACM Computing Surveys, vol. 38, No. 4, Article 13, Dec. 2006, pp. 1-45. |
| International Organization for Standardization, “Information Technology—Dynamic adaptive Streaming Over HTTP (DASH)—Part 3: Implementation Guidelines/AMO 1: Alignment with ISO/IEC 23009-1:2014 and Extensibilily” _ International Standard, ISO/IEC JTC 1/SC 29N, ISO/IEC TR 23009-3:2014/PDAM1, ISO/IEC JTC 1/SC 29/WG11, No. N15990, Mar. 2, 2016, 49 pages. |
| International Search Report and Written Opinion of the International Searching Authority for PCT/US2017/040060 dated Sep. 11, 2017, 15 pages. |
| Wang, X., el. al., “Multi-Stream Streaming in DASH {Multi-Stream DASH)”. Huawei Technology, Ltd., Motion Picture Expert Group {MPEG) Meeting, No. m35885, Feb. 2015, 8 pages. |
| International Preliminary Report on Palenlability for PCT/US2017/040060 dated Jan. 1, 2019. |
| International Search Report and Written Opinion of The International Searching Authority for PCT/US2017/026388 dated Jul. 24, 2017, 14 pages. |
| Written Opinion of the International Preliminary Examining Authority for PCT/US2017/026388 dated Mar. 5, 2018. |
| International Preliminary Report on Palenlability for PCT/US2017/026388 completed on Jul. 12, 2018. |
| International Search Report and Written Opinion of the International Searching Authority for PCT/US2018/018540 dated Apr. 6, 2018. |
| Mavlankar, A. et. al., “An Interactive Region-Of-Interest Video Streaming System for Online Lecture Viewing”. Proceedings of 2010 IEEE 18th International Packet Video Workshop Dec. 13-14, 2010, pp. 64-68. |
| Written Opinion of the International Preliminary Examining Authority for PCT/US2018/018540 dated Jan. 18, 2019, 7 pages. |
| International Preliminary Report on Palentability for PCT/US2018/018540 completed Apr. 26, 2019, 7 pages. |
| Ubuntu, “Display Adaptive Tone Mapping” _ Ubuntu Manuals, available at: x<http://manpages.ubuntu.com/manpages/ xenial/en/man1/pfstmo_mantiuk08.1.html, (2010), 7pages. |
| Invitation to pay additional Fees for PCT/US2018/020229, issued by the International Searching Authority, and dated Jun. 8, 2018, 14 pages. |
| International Search Report and Written Opinion of the International Searching Authority for PCT/US2018/020229 dated Jul. 31, 2018, 18 pages. |
| Brandenburg, R., et., al., “Immersive Second-Screen Experiences Using Hybrid Media Synchronization”. Media Synchronization Workshop, (2013), 7 pages. |
| HBB Next, “Immersive Second Screen Synchronization”. HBB Next, Next Generation Hybrid Media, Applications, vailable at: http://www.mediafutureweek.nl/wp-content/uploads/2014/05/poster_IBC_2013_applications.pdf, (2013), 1 page. |
| Niamut, O.A., el. al., “Live Event Experiences—Interactive UHDTV On Mobile Devices”. IBC2014 Conference, Sep. 11-15, 2014, 8 pages. |
| Lindgren, P., el., al., “How True, Synchronized Live OTT Can Change the Second Screen and Social TV Game”. IBC2016 Conference, Sep. 8-12, 2016, pp. 1-8. |
| Foungev Films, “HD 1080p—Nature Scenery Video”. YouTube link available at: www.youtube.com/walch?v=DQuhA5ZCV9M, published on Jan. 13, 2016, 3 pages. |
| Flowplayer—“Bandwidth Detection”. Web Archived dated Dec. 24, 2016. Retrieved from http://web.archive.org/web/20161224034909/hllp://flash.flowplayer.org/plugins/streaming/bwcheck.html. |
| International Preliminary Report on Palenlability for PCT/US2018/020229 dated Sep. 10, 2019, 12 pages. |
| International Organization for Standardization, “Information Technology—Dynamic Adaptive Streaming Over HTTP {DASH), Part 1: Media Presentation Description and Segment Formats/ Amendments 2: Spatia relationship Description, Generalized URL Parameters and other Extensions”. International Standard, ISO/IEC 23009-1:2015/FDAM 2:2015{E), Second Edition {Annex H), Feb. 25, 2015, 23 pages. |
| IBC, “D-Zero—Low Latency ABR Streaming” _ Sky Italia, available at: http://www.ibc.org/ibc-awards/ibc2016-innovation-award-entrants/ibc-innovation-award-for-content-delivery-2016-entrants, Sep. 2016. |
| IBC, “True live OTT Solution Puts F1 in Pole Position”. Tata Communications, available at: http://www.ibc.org/ibc-awards/ibc2016-innovation-award-entrants/ibc-innovation-award-for-content-delivery-2016-entrants, Sep. 2016. |
| SYE, “SYE”. Web Archive dated Sep. 5, 2016, available at: https://netinsight.nel/starl/technical-offer-sye/, 1 page. |
| Sun, Lei, el. al., “The Dynamic Videobook: A Hierarchical Summarization for Surveillance Video”. IEEE International Conference on Image Processing, (2013), pp. 3963-3966. |
| Tatacomms, “Transforming media+ entertainment (subtitles)”. Youtube link dated Sep. 10, 2015, available at: https://w,w.youtube.com/watch?v=KUxxPYocMiw, 19 pages. |
| Tata Communications, “Media and Entertainment Services”. Tata Communications, Archived on Feb. 27, 2017. Retrieved from https://web.archive.org/web/20170217212843/hllps://www.tatacommunications.com/products-services/enterprises/media-entertainment-services, 1 page. |
| Intel Corporation, “Overview of new radio access technology requirements and designs”, 3GPP Tdoc 1-162379, 3GPP TSG RAN WG1 Meeting #84bis, Busan, Korea, Apr. 11-15, 2016, 4 pages. |
| ETSI TS 102690 V1.1.1, Machine-lo-Machine Communications {M2M), “Functional Architecture” Oct. 2011, 280 pages. |
| European Telecommunications Standards Institute (ETSI / TS 102 690 V1.2.1; Machine-lo-Machine Communications M2M); Functional Architecture, Jun. 2013, 279 pages. |
| Qualcomm incorporated S2-162339 “Solution for key issue 1 on network slicing : network slice and network functions election based on evolved eDeCOR model” S2-162339, SA WG2 Meeting #115, May 23-27, 2016, Nanjing, P.R. hina (7 pages). |
| ZTE: “Considerations on channel access procedure for NR-U”, 3GPP Draft; R2-1811281 Considerations on Random Access Procedure for NR-U, 3rd Generation Partnership Project (3GPP), Mobile Competence Centre; 650, Route Des, Lucioles; F-06921 Sophia-Antipolis Cedex; France, vol. RAN WG2, No. Gothenburg, Sweden; Aug. 20, 2018-Aug. 24, 2018 Aug. 9, 2018, Aug. 9, 2018 , XP051520945. |
| Hawei Etal: “Numerology and wideband operation in NR unlicensed”, 3GPP Draft; R1-1803677, 3rd Generation Partnership Project (3GPP), Mobile Competence Centre; 650, Route Des Lucioles; F-06921 Sophia-Antipolis Cedex; France, vol. RAN WG1, No. Sanya, China; Apr. 16, 2018-Apr. 20, 018 Apr. 15, 2018 (Apr. 15, 2018), XP051425974. |
| Nokia et al: “Feature Lead's Summary on Channel Access Procedures” 3GPP Draft; R1-1809801 NRCJ Channel Access Summary, 3rd Generation Partnership Project {3GPP), Mobile Competence Entre ; 650, Route Des Lucioles ; F-06921 Sophia-Antipolis Cedex ; France, vol. RAN WGI , No. Gothenburg, Sweden; Aug. 20, 2018-Aug. 20, 2018 Aug. 23, 2018 {Aug. 23, 2018),XP051517160. |
| Huawei et al: “Initial access in NR unlicensed”, 3GPP Draft;R1-1808062, 3rd Generation Partnership Project (3GPP),Mobile Competence Centre ; 650, Route Des Lucioles ⋅F-06921 Sophia-Antipolis Cedex ; France, vol. RAN W1I , No. Gothenburg, Sweden; Aug. 20, 2018-Aug. 24, 2018 Aug. 10, 2018 (Aug. 10, 2018), XP051515466. |
| European Telecommunications Standards Institute (ETSI), TS 102 690 V2.1.1, Technical Specification, Machine-to-Machine Communications (M2M); Functional Architecture, Oct. 2013, 332 pages. |
| OneM2M Technical Specification, oneM2M-TS-0001 oneM2M Functional Archilecture-V-0.0.7, oneM2M Functional\Archilecture, Aug. 26, 2013, 43 pages. +. |
| Third Generation Partnership Project, “Technical Specification Group Radio Access Network; Feasibility tudy for Further Advancements for E-UTRA {LTE-Advanced) {Release 13),” 3GPP TR 36.912 V13.0.0, Dec. 015 (62 pages). |
| Third Generation Partnership Project, “Technical Specification Group Radio Access Network; Study on Elevation Beamforming / Full-Dimension (FD) Multiple Input Multiple Output (MIMO) for LTE” (Release 13), 3GPP TR 6.897 V13.0.0, Jun. 2015 (58 pages). |
| Third Generation Partnership Project, Technical Specification Group Radio Access Network; Evolved Universal Terrestrial Radio Access {E-UTRA); Study on Latency Reduction Techniques for LTE (Release 14) 3GPP R 36.881 V14.0.0, Jun. 2016 (99 pages). |
| Third Generation Partnership Project, “Technical Specification Group Services and System Aspects; IP Multimedia Substystem (IMS)” Stage 2 (Release 12), 3GPP TS 23.228 V12.1.0, Jun. 2013, (296 pages). |
| Third Generation Partnership Project, “Technical Specification Group Services and System Aspects; Feasibility Study on New Services and Markets Technology Enablers for Critical Communications” Stage 1 (Release 4), 3GPP TR 22.862 V14.1.0, Sep. 2016, (31 pages). |
| Third Generation Partnership Project, “Technical Specification Group Services and System Aspects; Feasibility Study on New Services and Markets Technology Enablers—Network Operation” Stage 1 (Release 14), GPP TR 22.864 V14.1.0, Sep. 2016 (35 pages). |
| Third Generation Partnership Project, “Technical Specification Group Services and System Aspects; Feasibility Study on New Services and Markets Technology Enablers—Enhanced Mobile Broadband” Stage 1 Release 14), 3GPP TR 22.863 V14.1.0, Sep. 2016 (21 pages). |
| IEEE P802.11, Wireless LANs, Proposed TGax draft specification, Comment Resolutions on UL MU Operation, Sep. 13, 2016, (30 pages). |
| Third Generation Partnership Project, “Technical Specification Group GSM/EDGE Radio Access etwork; Cellular system support for ultra-low complexity and low throughput Internet of Things (CIoT)” (Release 13), GPP TR 45.820 V13.1.0, Nov. 2015 (495 pages). |
| Third Generation Partnership Project, “Technical Specification Group Services and System Aspects; Feasibility Study on New Services and Markets Technology Enablers” Stage 1 (Release 14), 3GPP TR 22.891 V1.1.0, ov, 2015 (95 pages). |
| Third Generation Partnership Project, “Technical Specification Group Services and System Aspects; Study on architecture enhancements for Cellular Internet of Things” {Release 13), 3GPP TR 23.720 V13.0.0, Mar. 2016 (94 pages). |
| ITU-R, “IMT Vision—Framework and Overall Objectives of the Future Development of IMT for 2020 and Beyond”, Recommendation ITU-R M.2083-0, M Series, Mobile, Radiodetermination, Amateur and Related Satellite Services, Sep. 2015 (21 pages). |
| Third Generation Partnership Project, “Technical Specification Group Radio Access Network; Evolved Universal Terrestrial Radio Access {E-UTRA); Medium Access Control (MAC) protocol specification” (Release 13), GPP TS 36.321 V13.0.0, Dec. 2015 (82 pages). |
| Third Generation Partnership Project, Technical Specification Group Radio Access Network; Study on Scenarios and Requirements for Next Generation Access Technologies (Release 14), 3GPP TR 38.913 V0.3.0, Jun. 2017 (30 pages). |
| Third Generation Partnership Project, “Technical Specification Group Radio Access Network; Evolved Universal Terrestrial Radio Access (E-UTRA); Radio Resource Control {RRC); Protocol specification” (Release 13), GPP TS 36.331 V13.0.0, Dec. 2015 (507 pages). |
| Third Generation Partnership Project, Technical Specification Group Radio Access Network; Evolved Universal Terrestrial Radio Access {E-UTRA); Physical layer procedures {Release 12), 3GPP TS 36.213 V13.0.0, Dec. 2015 (6 pages). |
| Third Generation Partnership Project, “Way Forward on Frame Structure”, 3GPP TSG RAN WG1 #85, R1-165669, Qualcomm and etc., Nanjing, China, May 23-27, 2016, Agenda Item 7.1.4 (2 pages). |
| Third Generation Partnership Project, “Technical Specification Group Radio Access Network, Study on Scenarios and Requirements for Next Generation Access Technologies, (Release 14)”, (3GPP), TR 38.913 V0.2.0, Feb. 2016 (19 pages). |
| Third Generation Partnership Project, “Technical Specification Group Radio Access Network; Evolved Universal Terrestrial Radio Access (E-UTRA); Physical Channels and Modulation (Release 13)”, 3GPP TS 36.211 V13.1.0, Mar. 2016 (155 pages). |
| Third Generation Partnership Project, 3GPP TSG RAN WG1 #85, R1-164013, “Framework for Beamformed Access” May 2016 (4 pages). |
| Hattachi, R,E,, et al., White Paper NGMN 5G Initiative, v1.0, Feb. 17, 2015 (125 pages). |
| Third Generation Partnership Project, “Frame Structure for NR”, 3GPP TSG-RAN WG1 #85, R1-164628 Ericsson, Nanjing, China, May 23-27, 2016 (3 pages). |
| Third Generation Partnership Project, “Introduction of Virtual Cell” 3GPP TSG RAN WG2 Meeting #93bis R2-162571, Apr. 2016 (3 pages). |
| Third Generation Partnership Project, “Further enhancements on signaling reduction to enable light connection for LTE”, 3GPP RP-160425 TSG RAN Meeting #71, Intel Corporation, Gothenburg, Sweden, Mar. 7-10, 2016 (7 pages). |
| Huawei, “Network slice selection” RAN WG3 Meeting #92, R3-16113, Agenda Item 10.2.3, Nanjing, China, May 23-27, 2016 (4 pages). |
| Third Generation Partnership Project, “Update the network slicing use case in Smarter”, 3GPP 1-152395 Revision of S1-152074, ZTE Corporation et al., ZTE Smarter Update the Network Slicing Use case REV3, vol. SA WG1, No. Belgrade Serbia, Aug. 24, 2015 (3 pages). |
| Chu, D., “Polyphase Codes With Good Periodic Correlation Properties”, IEEE Transactions On Information Theory, Jul. 1972, pp. 531-532 (2 pages). |
| Third Generation Partnership Project, “Technical Specification Group Services and Systems Aspects, Feasibility Study on New Services and Markets Technology Enablers for Massive Internet of Things, Stage 1” (Release 14), 3GPP TR 22.861 V14.1.0, Sep. 2016, (28 pages). |
| European Telecommunications Standards Institute (ETSI TS) 124 302 V13.5.0, Universal Mobile Telecommunications System (UMTS); LTE; Access to the 3GPP Evolved Packet Core (EPC) via non-3GPP access networks; Stage 3 (Release 13), 3GPP TS 24.302 V13.5.0, Apr. 2016 (128 pages). |
| European Telecommunications Standards Institute (ETSI TS) 136 304 V13.0.0, “LTE; Evolved Universal Terrestrial Radio Access (E-UTRA); User Equipment (UE) procedures in idle mode” (Release 13), 3GPP TS 36.304 V13.0.0, Feb. 2016 (44 pages). |
| Third Generation Partnership Project, (3GPP) TSG RAN WG1 #85, R1-164014, Discussion on RS for Beamformed Access, Samsung, Nanjing, China, May 23-27, 2016 (3 pages). |
| Third Generation Partnership Project, {3GPP) S2-161324 SA WG2 Meeting #113, Solution to Key Issue on Mobility Framework, Sophia Antipolis, FR, Feb. 23-26, 2016 (3 pages). |
| European Telecommunications Standards Institute (ETSI TS) Digital cellular telecommunications system (Phase 2+) (GSM); Universal Mobile Telecommunications System (UMTS); General Packet Radio Service (GPRS); Service Description; Stage 2 (3GPP TS 23.060 version 13.6.0 (Release 13) Apr. 2016 (364 pages). |
| European Telecommunications Standards Institute (ETSI TS) LTE; General Packet Radio Service (GPRS) enhancements for Evolved Universal Terrestrial Radio Access Network (E-UTRAN) access (Release 13), 3GPP TS 23.401 V13.6.1, May 2016 (367 pages). |
| European Telecommunications Standards Institute (ETSI TS) Technical Specification LTE; Evolved Universal Terrestrial Radio Access (E-UTRA); Requirements for support of radio resource management (Release 14), 3GPP TS 36.133 V14.7.0, Apr. 2018 (2999 pages). |
| OMA Open Mobile Alliance, “Lightwieght Machine to Machine Technical Specification, Approved Version 1.0”, OMA-TS-LightweightM2M-V1_0-20170208-A, Feb. 8, 2017 (138 pages). |
| Shibata, F., el. al., “A View Management Method for Mobile Mixed Reality Systems”. In Virtual Environments 2008: 14th Eurographics Symposium on Virtual Env., May 29-30, 2008, 8 pages. |
| International Search Report and Written Opinion of the International Searching Authority for PCT/US2017/043248 dated Oct. 5, 2017. |
| Written Opinion of the International Preliminary Examining Authority for PCT/US2017/043248 dated Jun. 28, 2018. |
| International Preliminary Report on Patentability for PCT/US2017/043248 completed Sep. 27, 2018. |
| Number | Date | Country | |
|---|---|---|---|
| 20230039194 A1 | Feb 2023 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62365863 | Jul 2016 | US | |
| 62360105 | Jul 2016 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16314846 | US | |
| Child | 17956223 | US |
