The present disclosure relates to systems and methods for processing and displaying image data, and more specifically, to systems and methods for implementing depth-based effects such as blurring of images such as light-field images.
In conventional photography, the camera must typically be focused at the time the photograph is taken. The resulting image may have only color data for each pixel; accordingly, any object that was not in focus when the photograph was taken cannot be brought into sharper focus because the necessary data does not reside in the image.
By contrast, light-field images typically encode additional data for each pixel related to the trajectory of light rays incident to that pixel when the light-field image was taken. This data can be used to manipulate the light-field image through the use of a wide variety of rendering techniques that are not possible to perform with a conventional photograph. In some implementations, a light-field image may be refocused and/or altered to simulate a change in the center of perspective (CoP) of the camera that received the image. Further, a light-field image may be used to generate an enhanced depth-of-field (EDOF) image in which all parts of the image are in focus. A depth map can typically be obtained from a light-field image, indicating the depth at which objects in the light-field image were disposed, relative to the light-field camera, at the time the light-field image was captured.
Existing techniques for processing conventional images are limited in many respects. Specifically, editing conventional images to provide effects, such as changing colorization, contrast, or objects in the image, can be challenging. Typically, the user must employ careful selection of object boundaries to control how the effects are applied. Accordingly, application of effects in conventional images can be a time-consuming and labor-intensive effort. Existing processing and effect application techniques for light-field images often do not make full use of the information in the light-field image, or utilize a user interface that makes it difficult for the user to locate and apply the desired effect.
According to various embodiments, the system and method described herein facilitate customization of image properties such as depth of field. The customized depth of field may be used to control various image processes, such as blurring and/or bokeh effects in foreground and/or background portions of the image. When applied to an image accompanied by sufficient depth information, the system and method may enable the user to easily designate depth levels within an image, such as background, subject, and foreground levels, based on depth and without the need to manually select the corresponding image portion(s).
An image such as a light-field image may be captured with a light-field image capture device with a microlens array. The image may be received in a data store along with a depth map that indicates depths at which objects in the image are disposed, relative to the camera, at the time of image capture. At an input device, first and second user input may be received to designate a first focus depth and a second focus depth different from the first focus depth, respectively. The first focus depth may indicate the far limit (i.e., maximum depth) of an image foreground, and the second focus depth may indicate the near limit (i.e., the minimum depth) of an image background.
A processor may use the first focus depth to identify the image foreground. The image foreground may have one or more foreground portions that have one or more foreground portion depths, each of which is less than the first focus depth. Similarly, the processor may also use the second focus depth to identify the image background. The image background may have one or more background portions that have one or more background portion depths, each of which is greater than the second focus depth.
Once the image foreground and the image background have been identified, the processor may apply blurring and/or other effects to the one or more foreground portions and/or the one or more background portions to generate a processed image. The processed image may have, for example, an image subject including one or more subject portions that have one or more subject portion depths, each of which is between the first focus depth and the second focus depth. The image subject may optionally be in focus, and blurring may be applied to the image foreground and the image background in proportion to the difference in depth between each foreground portion and each background portion from the subject portion. Thus, a foreground portion with a depth that is much smaller, relative to the image subject, may have more blurring applied to it than a foreground portion with a depth that is nearly as large as the first focus depth. Similarly, a background portion with a depth that is much greater, relative to the image subject, may have more blurring applied to it than a background portion with a depth that is only just larger than the second focus depth.
The processed image may be displayed on a display device. The user may make further adjustments, such as adjusting the first focus depth and/or the second focus depth again, until the desired effect is obtained. Various user interface elements, controls, and the like may be provided to the user to facilitate determination of the first and second focus depths and/or control application of depth-based effects such as blurring. In addition to or in the alternative to application of blurring, the designated first and second focus depths may be used to control application of bokeh effects such as the application of blur effects, which may be circular, noncircular, and/or variable with depth. Further, the blurring and bokeh effects are merely exemplary; the system and method disclosed herein may be used to apply a wide variety of other effects besides blurring and bokeh effects. Such effects may include, but are not limited to, modification of exposure, contrast, saturation, and/or colorization of the image, replacement of a portion of an image with another image or portion thereof, and/or the like. Hence, it will be understood that reference to application of “blurring” in this disclosure is also disclosure of any other depth-based effect.
Application of blurring, bokeh effects, and/or other effects may be varied linearly with depth and/or nonlinearly with depth, as desired. Additionally or alternatively, application of such effects may be varied linearly and/or nonlinearly based on the X coordinate and/or the Y coordinate of the pixel to be blurred. Thus, blurring maybe applied with greater intensity toward the top, bottom, center, and/or edges of an image.
The accompanying drawings illustrate several embodiments. Together with the description, they serve to explain the principles of the embodiments. One skilled in the art will recognize that the particular embodiments illustrated in the drawings are merely exemplary, and are not intended to limit scope.
For purposes of the description provided herein, the following definitions are used:
In addition, for ease of nomenclature, the term “camera” is used herein to refer to an image capture device or other data acquisition device. Such a data acquisition device can be any device or system for acquiring, recording, measuring, estimating, determining and/or computing data representative of a scene, including but not limited to two-dimensional image data, three-dimensional image data, and/or light-field data. Such a data acquisition device may include optics, sensors, and image processing electronics for acquiring data representative of a scene, using techniques that are well known in the art. One skilled in the art will recognize that many types of data acquisition devices can be used in connection with the present disclosure, and that the disclosure is not limited to cameras. Thus, the use of the term “camera” herein is intended to be illustrative and exemplary, but should not be considered to limit the scope of the disclosure. Specifically, any use of such term herein should be considered to refer to any suitable device for acquiring image data.
In the following description, several techniques and methods for processing light-field images are described. One skilled in the art will recognize that these various techniques and methods can be performed singly and/or in any suitable combination with one another.
Architecture
In at least one embodiment, the system and method described herein can be implemented in connection with light-field images captured by light-field capture devices including but not limited to those described in Ng et al., Light-field photography with a hand-held plenoptic capture device, Technical Report CSTR 2005-02, Stanford Computer Science. Referring now to
In at least one embodiment, camera 200 may be a light-field camera that includes light-field image data acquisition device 209 having optics 201, image sensor 203 (including a plurality of individual sensors for capturing pixels), and microlens array 202. Optics 201 may include, for example, aperture 212 for allowing a selectable amount of light into camera 200, and main lens 213 for focusing light toward microlens array 202. In at least one embodiment, microlens array 202 may be disposed and/or incorporated in the optical path of camera 200 (between main lens 213 and sensor 203) so as to facilitate acquisition, capture, sampling of, recording, and/or obtaining light-field image data via sensor 203. Referring now also to
In at least one embodiment, light-field camera 200 may also include a user interface 205 for allowing a user to provide input for controlling the operation of camera 200 for capturing, acquiring, storing, and/or processing image data.
Similarly, in at least one embodiment, post-processing system 300 may include a user interface 305 that allows the user to provide input to control and/or activate depth-based image processing, as set forth in this disclosure. The user interface 305 may facilitate the receipt of user input from the user to establish one or more parameters of the image processing process.
In at least one embodiment, light-field camera 200 may also include control circuitry 210 for facilitating acquisition, sampling, recording, and/or obtaining light-field image data. For example, control circuitry 210 may manage and/or control (automatically or in response to user input) the acquisition timing, rate of acquisition, sampling, capturing, recording, and/or obtaining of light-field image data.
In at least one embodiment, camera 200 may include memory 211 for storing image data, such as output by image sensor 203. Such memory 211 can include external and/or internal memory. In at least one embodiment, memory 211 can be provided at a separate device and/or location from camera 200, such as the post-processing system 300.
For example, camera 200 and/or the post-processing system 300 may store raw light-field image data, as output by sensor 203, and/or a representation thereof, such as a compressed image data file. In addition, as described in related U.S. Utility application Ser. No. 12/703,367 for “Light-field Camera Image, File and Configuration Data, and Method of Using, Storing and Communicating Same,”, filed Feb. 10, 2010, memory 211 can also store data representing the characteristics, parameters, and/or configurations (collectively “configuration data”) of device 209.
In at least one embodiment, captured image data is provided to post-processing circuitry 204. Such circuitry 204 may be disposed in or integrated into light-field image data acquisition device 209, as shown in
The post-processing system 300 may include any of a wide variety of computing devices, including but not limited to computers, smartphones, tablets, cameras, and/or any other device that processes digital information. The post-processing system 300 may include additional features such as a user input 215 and/or a display screen 216. If desired, light-field image data may be displayed for the user on the display screen 216.
Overview
Light-field images often include a plurality of projections (which may be circular or of other shapes) of aperture 212 of camera 200, each projection taken from a different vantage point on the camera's focal plane. The light-field image may be captured on sensor 203. The interposition of microlens array 202 between main lens 213 and sensor 203 causes images of aperture 212 to be formed on sensor 203, each microlens in microlens array 202 projecting a small image of main-lens aperture 212 onto sensor 203. These aperture-shaped projections are referred to herein as disks, although they need not be circular in shape. The term “disk” is not intended to be limited to a circular region, but can refer to a region of any shape.
Light-field images include four dimensions of information describing light rays impinging on the focal plane of camera 200 (or other capture device). Two spatial dimensions (herein referred to as x and y) are represented by the disks themselves. For example, the spatial resolution of a light-field image with 120,000 disks, arranged in a Cartesian pattern 400 wide and 300 high, is 400×300. Two angular dimensions (herein referred to as u and v) are represented as the pixels within an individual disk. For example, the angular resolution of a light-field image with 100 pixels within each disk, arranged as a 10×10 Cartesian pattern, is 10×10. This light-field image has a 4-D (x,y,u,v) resolution of (400,300,10,10). Referring now to
In at least one embodiment, the 4-D light-field representation may be reduced to a 2-D image through a process of projection and reconstruction. As described in more detail in related U.S. Utility application Ser. No. 13/774,971 for “Compensating for Variation in Microlens Position During Light-Field Image Processing,” filed Feb. 22, 2013, the disclosure of which is incorporated herein by reference in its entirety, a virtual surface of projection may be introduced, and the intersections of representative rays with the virtual surface can be computed. The color of each representative ray may be taken to be equal to the color of its corresponding pixel.
Any number of image processing techniques can be used to reduce color artifacts, reduce projection artifacts, increase dynamic range, and/or otherwise improve image quality. In particular, enhanced depth-of-field (EDOF) images may be created, in which all parts of the image are in focus. However, in many instances the user may wish to fine tune the depth of field of the image to keep one or more portions of the image in focus. For example, the user may wish to designate an image subject that is to be in focus, while an image background and/or an image foreground are blurred. Such image processing may be facilitated by the system and method provided herein.
Depth-Based Image Processing
The system and method of the present disclosure may facilitate user control over which portions of the image are in focus, and which are blurred. Further, the system and method of the present disclosure may facilitate user control over the application of other effects besides blurring, which include but are not limited to modification of exposure, contrast, saturation, and/or colorization of the image, replacement of a portion of an image with another image or portion thereof, and/or the like. Further, the user may have greater control over the manner in which blurring is applied. This will be conceptually illustrated in connection with
The graph 500 of
The graph 550 of
The system and method presented herein may advantageously provide the user with the ability to determine which portion of the image is in focus by designating the first focus depth 570 and the second focus depth 580. Further, if desired, the user may designate the manner in which blur is applied to the foreground and/or background portions of the image. One exemplary method will be shown and described in connection with
The method may be performed, for example, with circuitry such as the post-processing circuitry 204 of the camera 200 of
The method may start 600 with a step 610 in which the image (for example, a light-field image) is captured, for example, by the sensor 203 of the camera 200. In a step 620, the image may be received in a computing device, which may be the camera 200 as in
In a step 630, user input may be received (for example, in the camera 200 and/or the post-processing system 300) to designate the first focus depth 570. This designation may be made in a variety of ways, including but not limited to text entry, speech, selection from a menu, clicking or tapping on an indicator such as a slider, selecting a portion of the image, and/or the like.
In a step 640, user input may be received (for example, in the camera 200 and/or the post-processing system 300) to designate the second focus depth 580. This designation may also be made in a variety of ways, including but not limited to text entry, speech, selection from a menu, clicking or tapping on an indicator such as a slider, selecting a portion of the image, and/or the like.
In a step 650, one or more foreground portions of the image may be identified. This may be done, for example, automatically by the camera 200 and/or the post-processing system. This may be accomplished by referencing the depth information corresponding to the image. Such depth information may take the form of a depth map in which intensity levels, colors, and/or other characteristics are used to indicate the depth at which objects in corresponding portions of the image were positioned, relative to the camera, at the time of image capture. Thus, for example, such a depth map may be used to automatically identify all portions of the image having a depth less than that of the first focus depth received in the step 630.
In a step 660, one or more background portions of the image may be identified. This may be done, for example, automatically by the camera 200 and/or the post-processing system. As in the step 650, this may be accomplished by referencing the depth map or other depth information corresponding to the image. For example, such a depth map may be used to automatically identify all portions of the image having a depth greater than that of the second focus depth received in the step 640.
In a step 670, blurring may be applied to the one or more foreground portions identified in the step 650 and/or to the one or more background portions identified in the step 660. Blurring may be applied through the application of one or more known blurring techniques, which may include mixing color and/or intensity values of a pixel to be blurred with those of surrounding pixels.
More specifically, ignoring the sampling issues caused by the fact that there are a finite number of pixel samples, a fully generalized blur function may be implemented. For example, the 2D bokeh of a point light source may be expressed as an arbitrary function of the three-dimensional coordinates of the point. This may make blur a five-dimensional function. During light-field projection, the five-dimensional function may be implemented as an individual three-dimensional function for each microlens of the two-dimensional array of microlenses (the microlens array 202 of
The five-dimensional function may be implemented as an individual three-dimensional function for each pixel of the processed image. Each three-dimensional function may specify the contribution of each pixel in the EDOF source image, as a function of the depth of that pixel. A wide variety of blur-related viewing features, including but not limited to focus distance, depth of field, tilt, focus spread, and additional blur may be implemented as simplified ways to specify the generalized five-dimensional blur function. The five dimensions may be as follows: Point X, Point Y, Point Z, Bokeh X, and Bokeh Y. The blur-related viewing features listed above may be implemented as follows:
As implemented in light-field projection, generalized blur may also be used to implement higher-level parallax effects such as center-of-perspective and stereo projection. These may simply weight the contributions of the samples behind each microlens in a way that is not radially symmetrical.
Once the desired blurring has been applied, the processed image may be displayed, for example, on the display screen 216 of the post-processing system 300, in a step 680. Then, in a query 690, a determination may be made as to whether the user wishes to modify the focus range, for example, by receiving input from the user via an input device such as the user input 215 of the post-processing system 300 or an input device of the camera 200. If so, the method may return to the step 630 and/or the step 640 so that the user can modify the first focus depth 570 and/or the second focus depth 580. The method may proceed through the step 650, the step 660, the step 670, the step 680, and the query 690 until the user is satisfied with the focus range of the image. When the user is satisfied with the focus range of the image, the query 690 may be answered in the negative. The method may then end 698.
The method of
In other examples, the first focus depth 570 and/or the second focus depth 580 may be used to apply different depth-based image processing steps. For example, any known bokeh effect may be applied, including the use of one or more blur shapes in the foreground and/or background. Such shapes need not be circular, and may vary with depth. In yet other examples, linear and/or nonlinear functions may be used to apply blurring and/or other depth-based effects. Examples of such modifications will be shown and described after the description of an exemplary user interface in
The image 710 may have a subject portion 712 that the user may wish to keep focused (the boy's face), a foreground portion 714 (the boy's hand and the plane he is holding) closer to the camera than the subject portion 712, and a background portion 716 (the sky, trees, and ground) further from the camera than the subject portion 712. Notably, in
As shown, the user interface may display the image 710, and may also display an editing pane 720 in which the user can make various modifications to the image 710. The editing pane 720 may include a focus spread toolbox 730 in which the user can customize the manner in which the image 710 is focused. The editing pane 720 may also include various other toolboxes that can be used to modify other aspects of the image 710.
The focus spread toolbox 730 may, for example, have a focus indicator 740 that indicates the depth at which the image 710 is focused. This may be the first focus depth 570, the second focus depth 580, the average of the first focus depth 570 and the second focus depth 580, and/or any other value indicative of focus depth. The focus spread toolbox 730 may also have a spread indicator 742 that indicates the size of the depth of field, or the size of the focus spread of the image 710. Further, the focus spread toolbox 730 may have a mode indicator 744 that indicates a mode in which the user is currently using the focus spread toolbox 730.
Further, the focus spread toolbox 730 may have a focus slider 750 that can be used to adjust the first focus depth 570 and/or the second focus depth 580. The focus slider 750 may have various elements that facilitate this adjustment, such as a first focus marker 752 and a second focus marker 754. The first focus marker 752 may be positioned at the first focus depth 570, and the second focus marker 754 may be positioned at the second focus depth 580.
The first focus marker 752 and the second focus marker 754 may divide the focus slider 750 into a subject portion 762, a foreground portion 764, and a background portion 766. The left-hand side of the focus slider 750 may represent smaller depth (i.e., objects in the image 710 that are closer to the camera), with depth gradually increasing toward the right-hand side of the focus slider 750. The subject portion 762, the foreground portion 764, and the background portion 766 of the focus slider 750 may correspond to the subject portion 712, the foreground portion 714, and the background portion 716 of the image 710, respectively.
The first focus marker 752 and the second focus marker 754 may be used to set the first focus depth 570 and the second focus depth 580, respectively. If desired, the user may click or tap and drag the first focus marker 752 to the left or right to decrease or increase, respectively, the first focus depth 570. Similarly, the user may click or tap and drag the second focus marker 754 to the left or right to decrease or increase, respectively, the second focus depth 580. Moving the first focus marker 752 to the left may cause portions of the foreground portion 714 of the image 710 to be included in the subject portion 712 rather than the foreground portion 714. Similarly, moving the second focus marker 754 to the right may cause portions of the background portion 716 of the image 710 to be included in the subject portion 712 rather than the background portion 716. The focus slider 750 may be manipulated in various other ways, as will be set forth in connection with
If desired, no blurring may be applied to the subject portion 712, designated by the subject portion 762 of the focus slider 750. Blurring may be applied to the foreground portion 714 and the background portion 716, with increasing blur applied to portions of the image 710 that are further displaced from the depth of the subject portion 712. This may cause the largest blur to be applied to objects in the image 710 that are closest to and/or furthest from the camera.
In addition to the focus slider 750 and related elements, the focus spread toolbox 730 may have a foreground dropper 774 and a background dropper 776. The foreground dropper 774 and the background dropper 776 may provide an alternative mechanism to manipulation of the first focus marker 752 and the second focus marker 754 to set the first focus depth 570 and the second focus depth 580, respectively. For example, the user may click on the foreground dropper 774 and then click on a portion of the image 710 that is at the desired depth for the first focus depth 570. The first focus depth 570 may then be set to the depth of the portion of the image 710 that was selected with the foreground dropper 774. Similarly, the user may click on the background dropper 776 and then click on a portion of the image 710 that is at the desired depth for the second focus depth 580. The second focus depth 580 may then be set to the depth of the portion of the image 710 that was selected with the background dropper 776.
If neither the foreground dropper 774 nor the background dropper 776 has been selected, the user may click, tap, or otherwise designate a portion of the image 710 at which the image 710 is to be focused. The result may be movement of the subject portion 762 of the focus slider 750 to center on a depth corresponding to that of the selected location of the image 710. This method of refocusing an image will be shown and described in greater detail subsequently, in connection with
In addition to the focus spread toolbox 730, the editing pane 720 may have other toolboxes such as a depth map toolbox 780 and a tilt toolbox 790. The depth map toolbox 780 may have a depth map button 792 that may be clicked or otherwise selected to initiate display of the depth map corresponding to the image 710. The depth map toolbox 780 may also have an image button 794 that may be clicked or otherwise selected to return to display of the image 710. Display of the depth map will be shown and described in connection with
The depth map 810 may help the user to more easily visualize the depth of each portion of the image 710. Thus, viewing the depth map 810 may help the user to understand the likely effects of manipulating the focus slider 750.
As indicated previously, the user may select the image button 794 to once again display the image 710 as in
The user may follow a similar procedure to expand the background portion 716 of the image 710 to cause blurring of objects at a greater depth from the camera. The user may move the second focus marker 754 to the left to reduce the second focus depth 580, thereby causing objects with a depth greater than the second focus depth 580 (in the depth map 810) to be blurred. As with adjustment of the foreground portion 714, adjustment of the background portion 716 may be visualized through the use of a color pattern to show the background portion 716. This will be shown in
In the alternative to the use of the first focus marker 752 and the second focus marker 754 to obtain the blurring of
The focus spread toolbox 730 may be manipulated in various other ways to further facilitate user designation of how the image 710 is to be focused and/or blurred. Some of these will be shown and described in connection with
Specifically, the mode indicator 744 of
In
If the user again selects the mode indicator 744, the focus spread toolbox 730 may return to the positive-only spread mode. Further, the first focus marker 752 and the second focus marker 754 may move together to eliminate the negative spread portion 1370. The user may then again be unable to move the first focus marker 752 to the right of the second focus marker 754, or move the second focus marker 754 to the left of the first focus marker 752.
The result may be motion of the entire subject portion 762 to the left or right. The subject portion 762 may remain the same size, but the foreground portion 764 may expand while the background portion 766 contracts, or vice versa. The first focus marker 752 and the second focus marker 754 may both move to the left or right, representing simultaneous reduction or increase of the first focus depth 570 and the second focus depth 580.
The user may even move one or both of the first focus marker 752 and the second focus marker 754 to the left or right, off the end of the focus slider 750. The result may be elimination of one of the foreground portion 764 and the background portion 766 (and thus elimination of one of the foreground portion 714 and the background portion 716 of the image 710) if only one of the first focus marker 752 and the second focus marker 754 is moved off the end of the focus slider 750. If the first focus marker 752 and the second focus marker 754 are both moved off the end of the focus slider 750, the result may also be elimination of the subject portion 762 (and thence, the subject portion 712 of the image 710) as will be shown in
In
The result of the configuration of
As shown, the image 1710 is of a chessboard with chess pieces. In
In
Variations
As indicated previously, various bokeh effects may be applied through the selection of a focus spread. The systems and methods set forth above for establishing a focus spread may be used to control the application of such bokeh effects. One exemplary bokeh effect is a blur shape, which may appear as a halo around a bright location in the blurred portion of an image. The system and method of the present disclosure may be used to apply such blur shapes. Exemplary blur shapes will be shown and described in connection with
These are merely examples of blur shapes that may be used for bokeh effects. In some embodiments, other blur shapes may be used, such as other polygonal shapes, ellipses, shapes with mixed curves and straight segments, and other fanciful shapes. If desired, an image may have more than one blur shape applied to it. Such different blur shapes may be applied based on depth (Z coordinate), horizontal or vertical location (X or Y coordinates), or other factors. In some embodiments, variable blur shapes may be used with variation based on distance from the focus spread.
Thus, for example, the near background (the region of the background portion 716 closest to the second focus depth 580) may have one shape, while the far background (the region of the background portion 716 furthest from the second focus depth 580) may have a different shape. The transition between the two shapes may be abrupt or smooth. If desired, a series of intermediate shapes may automatically be generated, and may be applied to provide a gradual transition between shapes with changing depth.
According to other alternatives, other factors such as the size, orientation, and/or positional offset of blur shapes may be varied with changes in depth. For example, a blur shape in the background portion 716 may be applied with a small size at maximum depth; the size of the blur shape may increase as the depth of objects in the image 710 approaches the second focus depth 580.
In
Thus, the transition in application of blurring or other depth-based effects between any two points (for example, a first point 3186 and a second point 3188 on the nonlinear background function 3184) may vary with a linear relationship to depth (distance from the camera). The first point 3186 and the second point 3188 may represent different degrees of blurriness, or in alternative embodiments, different application of depth-based effects such as the use of different sizes or shapes of blur shapes.
In
Thus, the transition in application of blurring or other depth-based effects between any two points may be variable, depending on the points selected. For example, a first point 3196 and a second point 3198 may represent different degrees of blurriness, or in alternative embodiments, different application of depth-based effects such as the use of different sizes or shapes of blur shapes. The transition in application of such effects between the first point 3196 and the second point 3198 may be variable with depth. As shown, the nonlinear background function 3194 need not continuously rise, but may fall as application of an effect is reversed with increasing depth, for some portion of the function.
In various embodiments, the user may have visibility to and/or control over the functions used to apply depth-based processing. A wide variety of user interfaces may be used to accomplish this. The user may select from a variety of pre-established linear and/or nonlinear functions, or may have the option to customize a function.
The above description and referenced drawings set forth particular details with respect to possible embodiments. Those of skill in the art will appreciate that the techniques described herein may be practiced in other embodiments. First, the particular naming of the components, capitalization of terms, the attributes, data structures, or any other programming or structural aspect is not mandatory or significant, and the mechanisms that implement the techniques described herein may have different names, formats, or protocols. Further, the system may be implemented via a combination of hardware and software, as described, or entirely in hardware elements, or entirely in software elements. Also, the particular division of functionality between the various system components described herein is merely exemplary, and not mandatory; functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component.
Reference in the specification to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment.
Some embodiments may include a system or a method for performing the above-described techniques, either singly or in any combination. Other embodiments may include a computer program product comprising a non-transitory computer-readable storage medium and computer program code, encoded on the medium, for causing a processor in a computing device or other electronic device to perform the above-described techniques.
Some portions of the above are presented in terms of algorithms and symbolic representations of operations on data bits within a memory of a computing device. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps (instructions) leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic or optical signals capable of being stored, transferred, combined, compared and otherwise manipulated. It is convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. Furthermore, it is also convenient at times, to refer to certain arrangements of steps requiring physical manipulations of physical quantities as modules or code devices, without loss of generality.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “displaying” or “determining” or the like, refer to the action and processes of a computer system, or similar electronic computing module and/or device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices.
Certain aspects include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of described herein can be embodied in software, firmware and/or hardware, and when embodied in software, can be downloaded to reside on and be operated from different platforms used by a variety of operating systems.
Some embodiments relate to an apparatus for performing the operations described herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computing device. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, flash memory, solid state drives, magnetic or optical cards, application specific integrated circuits (ASICs), and/or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Further, the computing devices referred to herein may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
The algorithms and displays presented herein are not inherently related to any particular computing device, virtualized system, or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the description provided herein. In addition, the techniques set forth herein are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the techniques described herein, and any references above to specific languages are provided for illustrative purposes only.
Accordingly, in various embodiments, the techniques described herein can be implemented as software, hardware, and/or other elements for controlling a computer system, computing device, or other electronic device, or any combination or plurality thereof. Such an electronic device can include, for example, a processor, an input device (such as a keyboard, mouse, touchpad, trackpad, joystick, trackball, microphone, and/or any combination thereof), an output device (such as a screen, speaker, and/or the like), memory, long-term storage (such as magnetic storage, optical storage, and/or the like), and/or network connectivity, according to techniques that are well known in the art. Such an electronic device may be portable or nonportable. Examples of electronic devices that may be used for implementing the techniques described herein include: a mobile phone, personal digital assistant, smartphone, kiosk, server computer, enterprise computing device, desktop computer, laptop computer, tablet computer, consumer electronic device, television, set-top box, or the like. An electronic device for implementing the techniques described herein may use any operating system such as, for example: Linux; Microsoft Windows, available from Microsoft Corporation of Redmond, Wash.; Mac OS X, available from Apple Inc. of Cupertino, Calif.; iOS, available from Apple Inc. of Cupertino, Calif.; Android, available from Google, Inc. of Mountain View, Calif.; and/or any other operating system that is adapted for use on the device.
In various embodiments, the techniques described herein can be implemented in a distributed processing environment, networked computing environment, or web-based computing environment. Elements can be implemented on client computing devices, servers, routers, and/or other network or non-network components. In some embodiments, the techniques described herein are implemented using a client/server architecture, wherein some components are implemented on one or more client computing devices and other components are implemented on one or more servers. In one embodiment, in the course of implementing the techniques of the present disclosure, client(s) request content from server(s), and server(s) return content in response to the requests. A browser may be installed at the client computing device for enabling such requests and responses, and for providing a user interface by which the user can initiate and control such interactions and view the presented content.
Any or all of the network components for implementing the described technology may, in some embodiments, be communicatively coupled with one another using any suitable electronic network, whether wired or wireless or any combination thereof, and using any suitable protocols for enabling such communication. One example of such a network is the Internet, although the techniques described herein can be implemented using other networks as well.
While a limited number of embodiments has been described herein, those skilled in the art, having benefit of the above description, will appreciate that other embodiments may be devised which do not depart from the scope of the claims. In addition, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, the disclosure is intended to be illustrative, but not limiting.
The present application is a continuation of U.S. application Ser. No. 14/871,533 for “Depth-Based Image Blurring”, filed Sep. 30, 2015, the disclosure of which is incorporated herein by reference in its entirety. The present application is a continuation-in-part of U.S. application Ser. No. 15/439,710 for “Depth-Assigned Content for Depth-Enhanced Virtual Reality Images”, filed Feb. 22, 2017, the disclosure of which is incorporated herein by reference in its entirety. U.S. application Ser. No. 15/439,710 is a continuation-in-part of U.S. Utility application Ser. No. 13/533,319 for “Depth-Assigned Content for Depth-Enhanced Pictures,” filed Jun. 26, 2012, the disclosure of which is incorporated herein by reference in its entirety. The present application is related to U.S. application Ser. No. 14/837,465 for “Depth-Based Application of Image Effects”, filed Aug. 27, 2015, the disclosure of which is incorporated herein by reference in its entirety. The present application is related to U.S. application Ser. No. 13/774,925 for “Compensating for Sensor Saturation and Microlens Modulation During Light-Field Image Processing”, filed Feb. 22, 2013, the disclosure of which is incorporated herein by reference in its entirety. The present application is related to U.S. Utility application Ser. No. 13/774,971 for “Compensating for Variation in Microlens Position During Light-Field Image Processing”, filed on Feb. 22, 2013, issued on Sep. 9, 2014 as U.S. Pat. No. 8,831,377, the disclosure of which is incorporated herein by reference in its entirety. The present application is related to U.S. Utility application Ser. No. 13/774,986 for “Light-Field Processing and Analysis, Camera Control, and User Interfaces and Interaction on Light-Field Capture Devices”, filed on Feb. 22, 2013, issued on Mar. 31, 2015 as U.S. Pat. No. 8,995,785, the disclosure of which is incorporated herein by reference in its entirety. The present application is related to U.S. Utility application Ser. No. 13/688,026 for “Extended Depth of Field and Variable Center of Perspective in Light-Field Processing”, filed on Nov. 28, 2012, issued on Aug. 19, 2014 as U.S. Pat. No. 8,811,769, the disclosure of which is incorporated herein by reference in its entirety. The present application is related to U.S. Utility application Ser. No. 11/948,901 for “Interactive Refocusing of Electronic Images,”, filed Nov. 30, 2007, issued on Oct. 15, 2013 as U.S. Pat. No. 8,559,705, the disclosure of which is incorporated herein by reference in its entirety. The present application is related to U.S. Utility application Ser. No. 12/703,367 for “Light-field Camera Image, File and Configuration Data, and Method of Using, Storing and Communicating Same,”, filed Feb. 10, 2010, now abandoned, the disclosure of which is incorporated herein by reference in its entirety. The present application is related to U.S. Utility application Ser. No. 13/027,946 for “3D Light-field Cameras, Images and Files, and Methods of Using, Operating, Processing and Viewing Same”, filed on Feb. 15, 2011, issued on Jun. 10, 2014 as U.S. Pat. No. 8,749,620, the disclosure of which is incorporated herein by reference in its entirety. The present application is related to U.S. Utility application Ser. No. 13/155,882 for “Storage and Transmission of Pictures Including Multiple Frames,”, filed Jun. 8, 2011, issued on Dec. 9, 2014 as U.S. Pat. No. 8,908,058, the disclosure of which is incorporated herein by reference in its entirety.
Number | Name | Date | Kind |
---|---|---|---|
725567 | Ives | Apr 1903 | A |
4383170 | Takagi et al. | May 1983 | A |
4661986 | Adelson | Apr 1987 | A |
4694185 | Weiss | Sep 1987 | A |
4920419 | Easterly | Apr 1990 | A |
5076687 | Adelson | Dec 1991 | A |
5077810 | D'Luna | Dec 1991 | A |
5251019 | Moorman et al. | Oct 1993 | A |
5282045 | Mimura et al. | Jan 1994 | A |
5499069 | Griffith | Mar 1996 | A |
5572034 | Karellas | Nov 1996 | A |
5610390 | Miyano | Mar 1997 | A |
5748371 | Cathey, Jr. et al. | May 1998 | A |
5757423 | Tanaka et al. | May 1998 | A |
5818525 | Elabd | Oct 1998 | A |
5835267 | Mason et al. | Nov 1998 | A |
5907619 | Davis | May 1999 | A |
5949433 | Klotz | Sep 1999 | A |
5974215 | Bilbro et al. | Oct 1999 | A |
6005936 | Shimizu et al. | Dec 1999 | A |
6021241 | Bilbro et al. | Feb 2000 | A |
6023523 | Cohen et al. | Feb 2000 | A |
6028606 | Kolb et al. | Feb 2000 | A |
6034690 | Gallery et al. | Mar 2000 | A |
6061083 | Aritake et al. | May 2000 | A |
6061400 | Pearlstein et al. | May 2000 | A |
6069565 | Stern et al. | May 2000 | A |
6075889 | Hamilton, Jr. et al. | Jun 2000 | A |
6091860 | Dimitri | Jul 2000 | A |
6097394 | Levoy et al. | Aug 2000 | A |
6115556 | Reddington | Sep 2000 | A |
6137100 | Fossum et al. | Oct 2000 | A |
6169285 | Pertrillo et al. | Jan 2001 | B1 |
6201899 | Bergen | Mar 2001 | B1 |
6221687 | Abramovich | Apr 2001 | B1 |
6320979 | Melen | Nov 2001 | B1 |
6424351 | Bishop et al. | Jul 2002 | B1 |
6448544 | Stanton et al. | Sep 2002 | B1 |
6466207 | Gortler et al. | Oct 2002 | B1 |
6476805 | Shum et al. | Nov 2002 | B1 |
6479827 | Hamamoto et al. | Nov 2002 | B1 |
6483535 | Tamburrino et al. | Nov 2002 | B1 |
6529265 | Henningsen | Mar 2003 | B1 |
6577342 | Webster | Jun 2003 | B1 |
6587147 | Li | Jul 2003 | B1 |
6597859 | Leinhardt et al. | Jul 2003 | B1 |
6606099 | Yamada | Aug 2003 | B2 |
6674430 | Kaufman et al. | Jan 2004 | B1 |
6687419 | Atkin | Feb 2004 | B1 |
6768980 | Meyer et al. | Jul 2004 | B1 |
6785667 | Orbanes et al. | Aug 2004 | B2 |
6833865 | Fuller et al. | Dec 2004 | B1 |
6842297 | Dowski, Jr. et al. | Jan 2005 | B2 |
6900841 | Mihara | May 2005 | B1 |
6924841 | Jones | Aug 2005 | B2 |
6927922 | George et al. | Aug 2005 | B2 |
7015954 | Foote et al. | Mar 2006 | B1 |
7025515 | Woods | Apr 2006 | B2 |
7034866 | Colmenarez et al. | Apr 2006 | B1 |
7079698 | Kobayashi | Jul 2006 | B2 |
7102666 | Kanade et al. | Sep 2006 | B2 |
7164807 | Morton | Jan 2007 | B2 |
7206022 | Miller et al. | Apr 2007 | B2 |
7239345 | Rogina | Jul 2007 | B1 |
7286295 | Sweatt et al. | Oct 2007 | B1 |
7304670 | Hussey et al. | Dec 2007 | B1 |
7329856 | Ma et al. | Feb 2008 | B2 |
7336430 | George | Feb 2008 | B2 |
7417670 | Linzer et al. | Aug 2008 | B1 |
7469381 | Ording | Dec 2008 | B2 |
7477304 | Hu | Jan 2009 | B2 |
7587109 | Reininger | Sep 2009 | B1 |
7620309 | Georgiev | Nov 2009 | B2 |
7623726 | Georgiev | Nov 2009 | B1 |
7633513 | Kondo et al. | Dec 2009 | B2 |
7683951 | Aotsuka | Mar 2010 | B2 |
7687757 | Tseng et al. | Mar 2010 | B1 |
7723662 | Levoy et al. | May 2010 | B2 |
7724952 | Shum et al. | May 2010 | B2 |
7748022 | Frazier | Jun 2010 | B1 |
7847825 | Aoki et al. | Dec 2010 | B2 |
7936377 | Friedhoff et al. | May 2011 | B2 |
7936392 | Ng et al. | May 2011 | B2 |
7941634 | Georgi | May 2011 | B2 |
7945653 | Zuckerberg et al. | May 2011 | B2 |
7949252 | Georgiev | May 2011 | B1 |
7982776 | Dunki-Jacobs et al. | Jul 2011 | B2 |
8013904 | Tan et al. | Sep 2011 | B2 |
8085391 | Machida et al. | Dec 2011 | B2 |
8106856 | Matas et al. | Jan 2012 | B2 |
8115814 | Iwase et al. | Feb 2012 | B2 |
8155456 | Babacan | Apr 2012 | B2 |
8155478 | Vitsnudel et al. | Apr 2012 | B2 |
8189089 | Georgiev et al. | May 2012 | B1 |
8228417 | Georgiev et al. | Jul 2012 | B1 |
8248515 | Ng et al. | Aug 2012 | B2 |
8259198 | Cote et al. | Sep 2012 | B2 |
8264546 | Witt | Sep 2012 | B2 |
8279325 | Pitts et al. | Oct 2012 | B2 |
8289440 | Knight et al. | Oct 2012 | B2 |
8290358 | Georgiev | Oct 2012 | B1 |
8310554 | Aggarwal et al. | Nov 2012 | B2 |
8315476 | Georgiev et al. | Nov 2012 | B1 |
8345144 | Georgiev et al. | Jan 2013 | B1 |
8400533 | Szedo | Mar 2013 | B1 |
8400555 | Georgiev et al. | Mar 2013 | B1 |
8427548 | Lim et al. | Apr 2013 | B2 |
8442397 | Kang et al. | May 2013 | B2 |
8446516 | Pitts et al. | May 2013 | B2 |
8494304 | Venable et al. | Jul 2013 | B2 |
8531581 | Shroff | Sep 2013 | B2 |
8542933 | Venkataraman et al. | Sep 2013 | B2 |
8559705 | Ng | Oct 2013 | B2 |
8570426 | Pitts | Oct 2013 | B2 |
8577216 | Li et al. | Nov 2013 | B2 |
8581998 | Dhno | Nov 2013 | B2 |
8589374 | Chaudhri | Nov 2013 | B2 |
8593564 | Border et al. | Nov 2013 | B2 |
8605199 | Imai | Dec 2013 | B2 |
8614764 | Pitts et al. | Dec 2013 | B2 |
8619082 | Ciurea et al. | Dec 2013 | B1 |
8629930 | Brueckner et al. | Jan 2014 | B2 |
8665440 | Kompaniets et al. | Mar 2014 | B1 |
8675073 | Aagaard et al. | Mar 2014 | B2 |
8724014 | Ng et al. | May 2014 | B2 |
8736710 | Spielberg | May 2014 | B2 |
8736751 | Yun | May 2014 | B2 |
8749620 | Pitts et al. | Jun 2014 | B1 |
8750509 | Renkis | Jun 2014 | B2 |
8754829 | Lapstun | Jun 2014 | B2 |
8760566 | Pitts et al. | Jun 2014 | B2 |
8768102 | Ng et al. | Jul 2014 | B1 |
8797321 | Bertolami et al. | Aug 2014 | B1 |
8811769 | Pitts et al. | Aug 2014 | B1 |
8831377 | Pitts et al. | Sep 2014 | B2 |
8860856 | Wetsztein et al. | Oct 2014 | B2 |
8879901 | Caldwell et al. | Nov 2014 | B2 |
8903232 | Caldwell | Dec 2014 | B1 |
8908058 | Akeley et al. | Dec 2014 | B2 |
8948545 | Akeley et al. | Feb 2015 | B2 |
8953882 | Lim et al. | Feb 2015 | B2 |
8971625 | Pitts et al. | Mar 2015 | B2 |
8976288 | Ng et al. | Mar 2015 | B2 |
8988317 | Liang et al. | Mar 2015 | B1 |
8995785 | Knight et al. | Mar 2015 | B2 |
8997021 | Liang et al. | Mar 2015 | B2 |
9001226 | Ng et al. | Apr 2015 | B1 |
9013611 | Szedo | Apr 2015 | B1 |
9106914 | Doser | Aug 2015 | B2 |
9172853 | Pitts et al. | Oct 2015 | B2 |
9184199 | Pitts et al. | Nov 2015 | B2 |
9201193 | Smith | Dec 2015 | B1 |
9210391 | Mills | Dec 2015 | B1 |
9214013 | Venkataraman et al. | Dec 2015 | B2 |
9294662 | Vondran, Jr. et al. | Mar 2016 | B2 |
9300932 | Knight et al. | Mar 2016 | B2 |
9305375 | Akeley | Apr 2016 | B2 |
9305956 | Pittes et al. | Apr 2016 | B2 |
9386288 | Akeley et al. | Jul 2016 | B2 |
9392153 | Myhre et al. | Jul 2016 | B2 |
9419049 | Pitts et al. | Aug 2016 | B2 |
9467607 | Ng et al. | Oct 2016 | B2 |
9497380 | Jannard et al. | Nov 2016 | B1 |
9628684 | Liang et al. | Apr 2017 | B2 |
9635332 | Carroll et al. | Apr 2017 | B2 |
9639945 | Oberheu et al. | May 2017 | B2 |
9647150 | Blasco Claret | May 2017 | B2 |
9681069 | El-Ghoroury et al. | Jun 2017 | B2 |
9774800 | El-Ghoroury et al. | Sep 2017 | B2 |
9866810 | Knight et al. | Jan 2018 | B2 |
9900510 | Karafin et al. | Feb 2018 | B1 |
9979909 | Kuang et al. | May 2018 | B2 |
20010048968 | Cox et al. | Dec 2001 | A1 |
20010053202 | Mazess et al. | Dec 2001 | A1 |
20020001395 | Davis et al. | Jan 2002 | A1 |
20020015048 | Nister | Feb 2002 | A1 |
20020061131 | Sawhney | May 2002 | A1 |
20020109783 | Hayashi et al. | Aug 2002 | A1 |
20020159030 | Frey et al. | Oct 2002 | A1 |
20020199106 | Hayashi | Dec 2002 | A1 |
20030081145 | Seaman et al. | May 2003 | A1 |
20030103670 | Schoelkopf et al. | Jun 2003 | A1 |
20030117511 | Belz et al. | Jun 2003 | A1 |
20030123700 | Wakao | Jul 2003 | A1 |
20030133018 | Ziemkowski | Jul 2003 | A1 |
20030147252 | Fioravanti | Aug 2003 | A1 |
20030156077 | Balogh | Aug 2003 | A1 |
20040002179 | Barton et al. | Jan 2004 | A1 |
20040012688 | Tinnerinno et al. | Jan 2004 | A1 |
20040012689 | Tinnerinno et al. | Jan 2004 | A1 |
20040101166 | Williams et al. | May 2004 | A1 |
20040114176 | Bodin et al. | Jun 2004 | A1 |
20040135780 | Nims | Jul 2004 | A1 |
20040189686 | Tanguay et al. | Sep 2004 | A1 |
20040257360 | Sieckmann | Dec 2004 | A1 |
20050031203 | Fukuda | Feb 2005 | A1 |
20050049500 | Babu et al. | Mar 2005 | A1 |
20050052543 | Li et al. | Mar 2005 | A1 |
20050080602 | Snyder et al. | Apr 2005 | A1 |
20050162540 | Yata | Jul 2005 | A1 |
20050212918 | Serra et al. | Sep 2005 | A1 |
20050276441 | Debevec | Dec 2005 | A1 |
20060023066 | Li et al. | Feb 2006 | A1 |
20060050170 | Tanaka | Mar 2006 | A1 |
20060056040 | Lan | Mar 2006 | A1 |
20060056604 | Sylthe et al. | Mar 2006 | A1 |
20060072175 | Oshino | Apr 2006 | A1 |
20060082879 | Miyoshi et al. | Apr 2006 | A1 |
20060130017 | Cohen et al. | Jun 2006 | A1 |
20060208259 | Jeon | Sep 2006 | A1 |
20060248348 | Wakao et al. | Nov 2006 | A1 |
20060256226 | Alon et al. | Nov 2006 | A1 |
20060274210 | Kim | Dec 2006 | A1 |
20060285741 | Subbarao | Dec 2006 | A1 |
20070008317 | Lundstrom | Jan 2007 | A1 |
20070019883 | Wong et al. | Jan 2007 | A1 |
20070030357 | Levien et al. | Feb 2007 | A1 |
20070033588 | Landsman | Feb 2007 | A1 |
20070052810 | Monroe | Mar 2007 | A1 |
20070071316 | Kubo | Mar 2007 | A1 |
20070081081 | Cheng | Apr 2007 | A1 |
20070097206 | Houvener | May 2007 | A1 |
20070103558 | Cai et al. | May 2007 | A1 |
20070113198 | Robertson et al. | May 2007 | A1 |
20070140676 | Nakahara | Jun 2007 | A1 |
20070188613 | Nobori et al. | Aug 2007 | A1 |
20070201853 | Petschnigg | Aug 2007 | A1 |
20070229653 | Matusik et al. | Oct 2007 | A1 |
20070230944 | Georgiev | Oct 2007 | A1 |
20070269108 | Steinberg et al. | Nov 2007 | A1 |
20080007626 | Wernersson | Jan 2008 | A1 |
20080012988 | Baharav et al. | Jan 2008 | A1 |
20080018668 | Yamauchi | Jan 2008 | A1 |
20080031537 | Gutkowicz-Krusin et al. | Feb 2008 | A1 |
20080049113 | Hirai | Feb 2008 | A1 |
20080056569 | Williams et al. | Mar 2008 | A1 |
20080122940 | Mori | May 2008 | A1 |
20080129728 | Satoshi | Jun 2008 | A1 |
20080144952 | Chen et al. | Jun 2008 | A1 |
20080152215 | Horie et al. | Jun 2008 | A1 |
20080168404 | Ording | Jul 2008 | A1 |
20080180792 | Georgiev | Jul 2008 | A1 |
20080187305 | Raskar et al. | Aug 2008 | A1 |
20080193026 | Horie et al. | Aug 2008 | A1 |
20080205871 | Utagawa | Aug 2008 | A1 |
20080226274 | Spielberg | Sep 2008 | A1 |
20080232680 | Berestov et al. | Sep 2008 | A1 |
20080253652 | Gupta et al. | Oct 2008 | A1 |
20080260291 | Alakarhu et al. | Oct 2008 | A1 |
20080266688 | Errando Smet et al. | Oct 2008 | A1 |
20080277566 | Utagawa | Nov 2008 | A1 |
20080309813 | Watanabe | Dec 2008 | A1 |
20080316301 | Givon | Dec 2008 | A1 |
20090002365 | Kurabayashi | Jan 2009 | A1 |
20090027542 | Yamamoto et al. | Jan 2009 | A1 |
20090041381 | Georgiev et al. | Feb 2009 | A1 |
20090041448 | Georgiev et al. | Feb 2009 | A1 |
20090070710 | Kagaya | Mar 2009 | A1 |
20090128658 | Hayasaka et al. | May 2009 | A1 |
20090128669 | Ng et al. | May 2009 | A1 |
20090135258 | Nozaki | May 2009 | A1 |
20090140131 | Utagawa | Jun 2009 | A1 |
20090102956 | Georgiev | Jul 2009 | A1 |
20090185051 | Sano | Jul 2009 | A1 |
20090185801 | Georgiev et al. | Jul 2009 | A1 |
20090190022 | Ichimura | Jul 2009 | A1 |
20090190024 | Hayasaka et al. | Jul 2009 | A1 |
20090195689 | Hwang et al. | Aug 2009 | A1 |
20090202235 | Li et al. | Aug 2009 | A1 |
20090204813 | Kwan | Aug 2009 | A1 |
20090273843 | Raskar et al. | Nov 2009 | A1 |
20090295829 | Georgiev et al. | Dec 2009 | A1 |
20090309973 | Kogane | Dec 2009 | A1 |
20090310885 | Tamaru | Dec 2009 | A1 |
20090321861 | Oliver et al. | Dec 2009 | A1 |
20100003024 | Agrawal et al. | Jan 2010 | A1 |
20100021001 | Honsinger et al. | Jan 2010 | A1 |
20100026852 | Ng et al. | Feb 2010 | A1 |
20100050120 | Ohazama et al. | Feb 2010 | A1 |
20100060727 | Steinberg et al. | Mar 2010 | A1 |
20100097444 | Lablans | Apr 2010 | A1 |
20100103311 | Makii | Apr 2010 | A1 |
20100107068 | Butcher et al. | Apr 2010 | A1 |
20100111489 | Presler | May 2010 | A1 |
20100123784 | Ding et al. | May 2010 | A1 |
20100141780 | Tan et al. | Jun 2010 | A1 |
20100141802 | Knight | Jun 2010 | A1 |
20100142839 | Lakus-Becker | Jun 2010 | A1 |
20100201789 | Yahagi | Aug 2010 | A1 |
20100253782 | Elazary | Oct 2010 | A1 |
20100265385 | Knight et al. | Oct 2010 | A1 |
20100277629 | Tanaka | Nov 2010 | A1 |
20100303288 | Malone | Dec 2010 | A1 |
20100328485 | Imamura et al. | Dec 2010 | A1 |
20110018903 | Lapstun et al. | Jan 2011 | A1 |
20110019056 | Hirsch et al. | Jan 2011 | A1 |
20110025827 | Shpunt et al. | Feb 2011 | A1 |
20110050864 | Bond | Mar 2011 | A1 |
20110050909 | Ellenby | Mar 2011 | A1 |
20110069175 | Mistretta et al. | Mar 2011 | A1 |
20110075729 | Dane et al. | Mar 2011 | A1 |
20110090255 | Wilson et al. | Apr 2011 | A1 |
20110123183 | Adelsberger et al. | May 2011 | A1 |
20110129120 | Chan | Jun 2011 | A1 |
20110129165 | Lim et al. | Jun 2011 | A1 |
20110148764 | Gao | Jun 2011 | A1 |
20110149074 | Lee et al. | Jun 2011 | A1 |
20110169994 | DiFrancesco et al. | Jul 2011 | A1 |
20110205384 | Zamowski et al. | Aug 2011 | A1 |
20110221947 | Awazu | Sep 2011 | A1 |
20110242334 | Wilburn et al. | Oct 2011 | A1 |
20110242352 | Hikosaka | Oct 2011 | A1 |
20110261164 | Olesen et al. | Oct 2011 | A1 |
20110261205 | Sun | Oct 2011 | A1 |
20110267263 | Hinckley | Nov 2011 | A1 |
20110273466 | Imai et al. | Nov 2011 | A1 |
20110133649 | Bales et al. | Dec 2011 | A1 |
20110292258 | Adler | Dec 2011 | A1 |
20110298960 | Tan et al. | Dec 2011 | A1 |
20110304745 | Wang et al. | Dec 2011 | A1 |
20110311046 | Oka | Dec 2011 | A1 |
20110316968 | Taguchi et al. | Dec 2011 | A1 |
20120014837 | Fehr et al. | Jan 2012 | A1 |
20120050562 | Perwass et al. | Mar 2012 | A1 |
20120056889 | Carter et al. | Mar 2012 | A1 |
20120057040 | Park et al. | Mar 2012 | A1 |
20120057806 | Backlund et al. | Mar 2012 | A1 |
20120062755 | Takahashi et al. | Mar 2012 | A1 |
20120132803 | Hirato et al. | May 2012 | A1 |
20120133746 | Bigioi et al. | May 2012 | A1 |
20120147205 | Lelescu et al. | Jun 2012 | A1 |
20120176481 | Lukk et al. | Jul 2012 | A1 |
20120188344 | Imai | Jul 2012 | A1 |
20120201475 | Carmel et al. | Aug 2012 | A1 |
20120206574 | Shikata et al. | Aug 2012 | A1 |
20120218463 | Benezra et al. | Aug 2012 | A1 |
20120224787 | Imai | Sep 2012 | A1 |
20120229691 | Hiasa et al. | Sep 2012 | A1 |
20120249529 | Matsumoto et al. | Oct 2012 | A1 |
20120249550 | Akeley | Oct 2012 | A1 |
20120249819 | Imai | Oct 2012 | A1 |
20120251131 | Henderson et al. | Oct 2012 | A1 |
20120257065 | Velarde et al. | Oct 2012 | A1 |
20120257795 | Kim et al. | Oct 2012 | A1 |
20120272271 | Nishizawa et al. | Oct 2012 | A1 |
20120287246 | Katayama | Nov 2012 | A1 |
20120287296 | Fukui | Nov 2012 | A1 |
20120287329 | Yahata | Nov 2012 | A1 |
20120293075 | Engelen et al. | Nov 2012 | A1 |
20120300091 | Shroff et al. | Nov 2012 | A1 |
20120237222 | Ng et al. | Dec 2012 | A9 |
20120320239 | Uehara | Dec 2012 | A1 |
20130002902 | Ito | Jan 2013 | A1 |
20130002936 | Hirama et al. | Jan 2013 | A1 |
20130021486 | Richardson | Jan 2013 | A1 |
20130038696 | Ding et al. | Feb 2013 | A1 |
20130041215 | McDowall | Feb 2013 | A1 |
20130044290 | Kawamura | Feb 2013 | A1 |
20130050546 | Kano | Feb 2013 | A1 |
20130064453 | Nagasaka et al. | Mar 2013 | A1 |
20130064532 | Caldwell et al. | Mar 2013 | A1 |
20130070059 | Kushida | Mar 2013 | A1 |
20130070060 | Chatterjee et al. | Mar 2013 | A1 |
20130077880 | Venkataraman et al. | Mar 2013 | A1 |
20130082905 | Ranieri et al. | Apr 2013 | A1 |
20130088616 | Ingrassia, Jr. | Apr 2013 | A1 |
20130093844 | Shuto | Apr 2013 | A1 |
20130093859 | Nakamura | Apr 2013 | A1 |
20130094101 | Oguchi | Apr 2013 | A1 |
20130107085 | Ng et al. | May 2013 | A1 |
20130113981 | Knight et al. | May 2013 | A1 |
20130120356 | Georgiev et al. | May 2013 | A1 |
20130120605 | Georgiev et al. | May 2013 | A1 |
20130120636 | Baer | May 2013 | A1 |
20130127901 | Georgiev et al. | May 2013 | A1 |
20130128052 | Catrein et al. | May 2013 | A1 |
20130128081 | Georgiev et al. | May 2013 | A1 |
20130128087 | Georgiev et al. | May 2013 | A1 |
20130135448 | Nagumo et al. | May 2013 | A1 |
20130176481 | Holmes et al. | Jul 2013 | A1 |
20130188068 | Said | Jul 2013 | A1 |
20130215108 | McMahon et al. | Aug 2013 | A1 |
20130215226 | Chauvier et al. | Aug 2013 | A1 |
20130222656 | Kaneko | Aug 2013 | A1 |
20130234935 | Griffith | Sep 2013 | A1 |
20130242137 | Kirkland | Sep 2013 | A1 |
20130258451 | El-Ghoroury et al. | Oct 2013 | A1 |
20130262511 | Kuffner et al. | Oct 2013 | A1 |
20130286236 | Mankowski | Oct 2013 | A1 |
20130321574 | Zhang et al. | Dec 2013 | A1 |
20130321581 | El-Ghoroury | Dec 2013 | A1 |
20130321677 | Cote et al. | Dec 2013 | A1 |
20130329068 | Hamanaka | Dec 2013 | A1 |
20130329107 | Burley et al. | Dec 2013 | A1 |
20130329132 | Tico et al. | Dec 2013 | A1 |
20130335596 | Demandoix et al. | Dec 2013 | A1 |
20130342526 | Ng et al. | Dec 2013 | A1 |
20130342700 | Kass | Dec 2013 | A1 |
20140002502 | Han | Jan 2014 | A1 |
20140002699 | Guan | Jan 2014 | A1 |
20140003719 | Bai et al. | Jan 2014 | A1 |
20140009585 | Campbell | Jan 2014 | A1 |
20140013273 | Ng | Jan 2014 | A1 |
20140035959 | Lapstun | Feb 2014 | A1 |
20140037280 | Shirakawa | Feb 2014 | A1 |
20140049663 | Ng et al. | Feb 2014 | A1 |
20140059462 | Wernersson | Feb 2014 | A1 |
20140085282 | Luebke et al. | Mar 2014 | A1 |
20140092424 | Grosz | Apr 2014 | A1 |
20140098191 | Rime et al. | Apr 2014 | A1 |
20140132741 | Aagaard et al. | May 2014 | A1 |
20140133749 | Kuo et al. | May 2014 | A1 |
20140139538 | Barber et al. | May 2014 | A1 |
20140167196 | Heimgartner et al. | Jun 2014 | A1 |
20140176540 | Tosic et al. | Jun 2014 | A1 |
20140176592 | Wilburn et al. | Jun 2014 | A1 |
20140176710 | Brady | Jun 2014 | A1 |
20140177905 | Grefalda | Jun 2014 | A1 |
20140184885 | Tanaka et al. | Jul 2014 | A1 |
20140192208 | Okincha | Jul 2014 | A1 |
20140193047 | Grosz | Jul 2014 | A1 |
20140195921 | Grosz | Jul 2014 | A1 |
20140204111 | Vaidyanathan et al. | Jul 2014 | A1 |
20140211077 | Ng et al. | Jul 2014 | A1 |
20140218540 | Geiss et al. | Aug 2014 | A1 |
20140226038 | Kimura | Aug 2014 | A1 |
20140240463 | Pitts et al. | Aug 2014 | A1 |
20140240578 | Fishman et al. | Aug 2014 | A1 |
20140267243 | Venkataraman et al. | Sep 2014 | A1 |
20140267639 | Tatsuta | Sep 2014 | A1 |
20140300753 | Yin | Oct 2014 | A1 |
20140313350 | Keelan | Oct 2014 | A1 |
20140313375 | Milnar | Oct 2014 | A1 |
20140340390 | Lanman et al. | Nov 2014 | A1 |
20140347540 | Kang | Nov 2014 | A1 |
20140354863 | Ahn et al. | Dec 2014 | A1 |
20140368494 | Sakharnykh et al. | Dec 2014 | A1 |
20140368640 | Strandemar et al. | Dec 2014 | A1 |
20150062178 | Matas et al. | Mar 2015 | A1 |
20150062386 | Sugawara | Mar 2015 | A1 |
20150092071 | Meng et al. | Apr 2015 | A1 |
20150097985 | Akeley | Apr 2015 | A1 |
20150104101 | Bryant et al. | Apr 2015 | A1 |
20150189154 | Laroia | Jul 2015 | A1 |
20150193937 | Georgiev et al. | Jul 2015 | A1 |
20150206340 | Munkberg et al. | Jul 2015 | A1 |
20150207990 | Ford et al. | Jul 2015 | A1 |
20150237273 | Sawadaishi | Aug 2015 | A1 |
20150279056 | Akeley | Oct 2015 | A1 |
20150310592 | Kano | Oct 2015 | A1 |
20150312553 | Ng et al. | Oct 2015 | A1 |
20150312593 | Akeley et al. | Oct 2015 | A1 |
20150370011 | Ishihara | Dec 2015 | A1 |
20150370012 | Ishihara | Dec 2015 | A1 |
20160029017 | Liang | Jan 2016 | A1 |
20160142615 | Liang | May 2016 | A1 |
20160155215 | Suzuki | Jun 2016 | A1 |
20160165206 | Huang et al. | Jun 2016 | A1 |
20160173844 | Knight et al. | Jun 2016 | A1 |
20160191823 | El-Ghoroury | Jun 2016 | A1 |
20160253837 | Zhu et al. | Sep 2016 | A1 |
20160269620 | Romanenko et al. | Sep 2016 | A1 |
20160307368 | Akeley | Oct 2016 | A1 |
20160307372 | Pitts et al. | Oct 2016 | A1 |
20160309065 | Karafin et al. | Oct 2016 | A1 |
20160353026 | Blonde et al. | Dec 2016 | A1 |
20160381348 | Hayasaka | Dec 2016 | A1 |
20170059305 | Nonn et al. | Mar 2017 | A1 |
20170067832 | Ferrara, Jr. et al. | Mar 2017 | A1 |
20170091906 | Liang et al. | Mar 2017 | A1 |
20170134639 | Pitts | May 2017 | A1 |
20170139131 | Karafin et al. | May 2017 | A1 |
20170237971 | Pitts et al. | Aug 2017 | A1 |
20170243373 | Bevensee et al. | Aug 2017 | A1 |
20170244948 | Pang et al. | Aug 2017 | A1 |
20170256036 | Song et al. | Sep 2017 | A1 |
20170263012 | Sabater et al. | Sep 2017 | A1 |
20170318226 | Jung | Nov 2017 | A1 |
20170358092 | Bleibel et al. | Dec 2017 | A1 |
20170365068 | Tan et al. | Dec 2017 | A1 |
20180012397 | Carothers | Jan 2018 | A1 |
20180020204 | Pang et al. | Jan 2018 | A1 |
20180033209 | Akeley et al. | Feb 2018 | A1 |
20180034134 | Pang et al. | Feb 2018 | A1 |
20180070066 | Knight et al. | Mar 2018 | A1 |
20180070067 | Knight et al. | Mar 2018 | A1 |
20180089903 | Pang et al. | Mar 2018 | A1 |
20180097867 | Pang et al. | Apr 2018 | A1 |
20180158198 | Karnad | Jun 2018 | A1 |
Number | Date | Country |
---|---|---|
101226292 | Jul 2008 | CN |
101309359 | Nov 2008 | CN |
19624421 | Jan 1997 | DE |
2010020100 | Jan 2010 | JP |
2011135170 | Jul 2011 | JP |
2003052465 | Jun 2003 | WO |
2006039486 | Apr 2006 | WO |
2007092545 | Aug 2007 | WO |
2007092581 | Aug 2007 | WO |
2011010234 | Mar 2011 | WO |
2011029209 | Mar 2011 | WO |
2011081187 | Jul 2011 | WO |
Entry |
---|
Agarwala, A., et al., “Interactive Digital Photomontage,” ACM Transactions on Graphics, Proceedings of SIGGRAPH 2004, vol. 32, No. 3, 2004. |
Nguyen, Hubert. “Practical Post-Process Depth of Field.” GPU Gems 3. Upper Saddle River, NJ: Addison-Wesley, 2008. |
Shade, Jonathan, et al., “Layered Depth Images”, SIGGRAPH 98, pp. 1-2. |
Wikipedia—Lazy loading of image data: http://en.wikipedia.org/wiki/Lazy_loading. Retrieved Jan. 2013. |
Wikipedia—Methods of Variable Bitrate Encoding: http://en.wikipedia.org/wiki/Variable_bitrate#Methods_of_VBR_encoding. Retrieved Jan. 2013. |
Wikipedia—Portable Network Graphics format: http://en.wikipedia.org/wiki/Portable_Network_Graphics. Retrieved Jan. 2013. |
Wikipedia—Unsharp Mask Technique: https://en.wikipedia.org/wiki/Unsharp_masking. Retrieved May 3, 2016. |
Wilburn et al., “High Performance Imaging using Large Camera Arrays”, ACM Transactions on Graphics (TOG), vol. 24, Issue 3 (Jul. 2005), Proceedings of ACM SIGGRAPH 2005, pp. 765-776. |
Wilburn, Bennett, et al., “High Speed Video Using a Dense Camera Array”, 2004. |
Wilburn, Bennett, et al., “The Light Field Video Camera”, Proceedings of Media Processors 2002. |
Williams, L. “Pyramidal Parametrics,” Computer Graphic (1983). |
Winnemoller, H., et al., “Light Waving: Estimating Light Positions From Photographs Alone”, Eurographics 2005. |
Wippermann, F. “Chirped Refractive Microlens Array,” Dissertation 2007. |
Wuu, S., et al., “A Manufacturable Back-Side Illumination Technology Using Bulk Si Substrate for Advanced CMOS Image Sensors”, 2009 International Image Sensor Workshop, Bergen, Norway. |
Wuu, S., et al., “BSI Technology with Bulk Si Wafer”, 2009 International Image Sensor Workshop, Bergen, Norway. |
Xiao, Z. et al., “Aliasing Detection and Reduction in Plenoptic Imaging,” IEEE Conference on Computer Vision and Pattern Recognition; 2014. |
Xu, Xin et al., “Robust Automatic Focus Algorithm for Low Contrast Images Using a New Contrast Measure,” Sensors 2011; 14 pages. |
Zheng, C. et al., “Parallax Photography: Creating 3D Cinematic Effects from Stills”, Proceedings of Graphic Interface, 2009. |
Zitnick, L. et al., “High-Quality Video View Interpolation Using a Layered Representation,” Aug. 2004; ACM Transactions on Graphics (TOG), Proceedings of ACM SIGGRAPH 2004; vol. 23, Issue 3; pp. 600-608. |
Zoberbier, M., et al., “Wafer Cameras—Novel Fabrication and Packaging Technologies”, 2009 International Image Senor Workshop, Bergen, Norway, 5 pages. |
Schirmacher, H. et al., “High-Quality Interactive Lumigraph Rendering Through Warping,” May 2000, Graphics Interface 2000. |
Shreiner, OpenGL Programming Guide, 7th edition, Chapter 8, 2010. |
Simpleviewer, “Tiltview”, http://simpleviewer.net/tiltviewer. Retrieved Jan. 2013. |
Skodras, A. et al., “The JPEG 2000 Still Image Compression Standard,” Sep. 2001, IEEE Signal Processing Magazine, pp. 36-58. |
Sloan, P., et al., “Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments”, ACM Transactions on Graphics 21, 3, 527-536, 2002. |
Snavely, Noah, et al., “Photo-tourism: Exploring Photo collections in 3D”, ACM Transactions on Graphics (SIGGRAPH Proceedings), 2006. |
Sokolov, “Autostereoscopy and Integral Photography by Professor Lippmann's Method”, 1911, pp. 23-29. |
Sony Corp, “Interchangeable Lens Digital Camera Handbook”, 2011. |
Sony, Sony's First Curved Sensor Photo: http://www.engadget.com; Jul. 2014. |
Stensvold, M., “Hybrid AF: A New Approach to Autofocus is Emerging for both Still and Video”, Digital Photo Magazine, Nov. 13, 2012. |
Story, D., “The Future of Photography”, Optics Electronics, Oct. 2008. |
Sun, Jian, et al., “Stereo Matching Using Belief Propagation”, 2002. |
Tagging photos on Flickr, Facebook and other online photo sharing sites (see, for example, http://support.gnip.com/customer/portal/articles/809309-flickr-geo-photos-tag-search). Retrieved Jan. 2013. |
Takahashi, Keita, et al., “All in-focus View Synthesis from Under-Sampled Light Fields”, ICAT 2003, Tokyo, Japan. |
Tanida et al., “Thin observation module by bound optics (TOMBO): concept and experimental verification” Applied Optics 40, 11 (Apr. 10, 2001), pp. 1806-1813. |
Tao, Michael, et al., “Depth from Combining Defocus and Correspondence Using Light-Field Cameras”, Dec. 2013. |
Techcrunch, “Coolinis”, Retrieved Jan. 2013. |
Teo, P., et al., “Efficient linear rendering for interactive light design”, Tech. Rep. STAN-CS-TN-97-60, 1998, Stanford University. |
Teranishi, N. “Evolution of Optical Structure in Images Sensors,” Electron Devices Meeting (IEDM) 2012 IEEE International; Dec. 10-13, 2012. |
U.S. Appl. No. 15/590,808, filed May 9, 2017 listing Alex Song et al. as inventors, entitled “Adaptive Control for Immersive Experience Delivery”. |
U.S. Appl. No. 15/590,841, filed May 9, 2017 listing Kurt Akeley et al. as inventors, entitled “Vantage Generation and Interactive Playback”. |
U.S. Appl. No. 15/590,951, filed May 9, 2017 listing Alex Song et al. as inventors, entitled “Wedge-Based Light-Field Video Capture”. |
U.S. Appl. No. 15/605,037, filed May 25, 2017 listing Zejing Wang et al. as inventors, entitled “Multl-View Back-Projection to a Light-Field”. |
U.S. Appl. No. 15/666,298, filed Aug. 1, 2017 listing Yonggang Ha et al. as inventors, entitled “Focal Reducer With Controlled Optical Properties for Interchangeable Lens Light-Field Camera”. |
U.S. Appl. No. 15/703,553, filed Sep. 13, 2017 listing Jon Karafin et al. as inventors, entitled “4D Camera Tracking and Optical Stabilization”. |
U.S. Appl. No. 15/864,938, filed Jan. 8, 2018 listing Jon Karafin et al. as inventors, entitled “Motion Blur for Light-Field Images”. |
U.S. Appl. No. 15/874,723, filed Jan. 18, 2018 listing Mark Weir et al. as inventors, entitled “Multi-Camera Navigation Interface”. |
U.S. Appl. No. 15/897,836, filed Feb. 15, 2018 listing Francois Bleibel et al. as inventors, entitled “Multi-View Contour Tracking”. |
U.S. Appl. No. 15/897,942, filed Feb. 15, 2018 listing Francois Bleibel et al. as inventors, entitled “Multi-View Contour Tracking With Grabcut”. |
U.S. Appl. No. 15/897,994, filed Feb. 15, 2018 listing Trevor Carothers et al. as inventors, entitled “Generation of Virtual Reality With 6 Degrees of Freesom From Limited Viewer Data”. |
U.S. Appl. No. 15/944,551, filed Apr. 3, 2018 listing Zejing Wang et al. as inventors, entitled “Generating Dolly Zoom Effect Using Light Field Image Data”. |
U.S. Appl. No. 15/967,076, filed Apr. 30, 2018 listing Jiantao Kuang et al. as inventors, entitled “Automatic Lens Flare Detection and Correction for Light-Field Images”. |
Vaish et al., “Using plane + parallax for calibrating dense camera arrays”, In Proceedings CVPR 2004, pp. 2-9. |
Vaish, V., et al., “Synthetic Aperture Focusing Using a Shear-Warp Factorization of the Viewing Transform,” Workshop on Advanced 3D Imaging for Safety and Security (in conjunction with CVPR 2005), 2005. |
VR Playhouse, “The Surrogate,” http://www.vrplayhouse.com/the-surrogate. |
Wanner, S. et al., “Globally Consistent Depth Labeling of 4D Light Fields,” IEEE Conference on Computer Vision and Pattern Recognition, 2012. |
Wanner, S. et al., “Variational Light Field Analysis for Disparity Estimation and Super-Resolution,” IEEE Transacations on Pattern Analysis and Machine Intellegence, 2013. |
Wenger, et al, “Performance Relighting and Reflectance Transformation with Time-Multiplexed Illumination”, Institute for Creative Technologies, SIGGRAPH 2005. |
Wetzstein, Gordon, et al., “Sensor Saturation in Fourier Multiplexed Imaging”, IEEE Conference on Computer Vision and Pattern Recognition (2010). |
Wikipedia—Adaptive Optics: http://en.wikipedia.org/wiki/adaptive_optics. Retrieved Feb. 2014. |
Wikipedia—Autofocus systems and methods: http://en.wikipedia.org/wiki/Autofocus. Retrieved Jan. 2013. |
Wikipedia—Bayer Filter: http:/en.wikipedia.org/wiki/Bayer_filter. Retrieved Jun. 20, 2013. |
Wikipedia—Color Image Pipeline: http://en.wikipedia.org/wiki/color_image_pipeline. Retrieved Jan. 15, 2014. |
Wikipedia—Compression standard JPEG XR: http://en.wikipedia.org/wiki/JPEG_XR. Retrieved Jan. 2013. |
Wikipedia—CYGM Filter: http://en.wikipedia.org/wiki/CYGM_filter. Retrieved Jun. 20, 2013. |
Wikipedia—Data overlay techniques for real-time visual feed. For example, heads-up displays: http://en.wikipedia.org/wiki/Head-up_display. Retrieved Jan. 2013. |
Wikipedia—Exchangeable image file format: http://en.wikipedia.org/wiki/Exchangeable_image_file_format. Retrieved Jan. 2013. |
Wikipedia—Expeed: http://en.wikipedia.org/wiki/EXPEED. Retrieved Jan. 15, 2014. |
Wikipedia—Extensible Metadata Platform: http://en.wikipedia.org/wiki/Extensible_Metadata_Platform. Retrieved Jan. 2013. |
Wikipedia—Key framing for video animation: http://en.wikipedia.org/wiki/Key_frame. Retrieved Jan. 2013. |
Adelsberger, R. et al., “Spatially Adaptive Photographic Flash,” ETH Zurich, Department of Computer Science, Technical Report 612, 2008, pp. 1-12. |
Adelson et al., “Single Lens Stereo with a Plenoptic Camera” IEEE Translation on Pattern Analysis and Machine Intelligence, Feb. 1992. vol. 14, No. 2, pp. 99-106. |
Adelson, E. H., and Bergen, J. R. 1991. The plenoptic function and the elements of early vision. In Computational Models of Visual Processing, edited by Michael S. Landy and J. Anthony Movshon. Cambridge, Mass.: mit Press. |
Adobe Systems Inc, “XMP Specification”, Sep. 2005. |
Adobe, “Photoshop CS6 / in depth: Digital Negative (DNG)”, http://www.adobe.com/products/photoshop/extend.displayTab2html. Retrieved Jan. 2013. |
Andreas Observatory, Spectrograph Manual: IV. Flat-Field Correction, Jul. 2006. |
Apple, “Apple iPad: Photo Features on the iPad”, Retrieved Jan. 2013. |
Bae, S., et al., “Defocus Magnification”, Computer Graphics Forum, vol. 26, Issue 3 (Proc. of Eurographics 2007), pp. 1-9. |
Belhumeur, Peter et al., “The Bas-Relief Ambiguity”, International Journal of Computer Vision, 1997, pp. 1060-1066. |
Belhumeur, Peter, et al., “The Bas-Relief Ambiguity”, International Journal of Computer Vision, 1999, pp. 33-44, revised version. |
Bhat, P. et al. “GradientShop: A Gradient-Domain Optimization Framework for Image and Video Filtering,” SIGGRAPH 2010; 14 pages. |
Bolles, R., et al., “Epipolar-Plane Image Analysis: An Approach to Determining Structure from Motion”, International Journal of Computer Vision, 1, 7-55 (1987). |
Bourke, Paul, “Image filtering in the Frequency Domain,” pp. 1-9, Jun. 1998. |
Canon, Canon Speedlite wireless flash system, User manual for Model 550EX, Sep. 1998. |
Chai, Jin-Xang et al., “Plenoptic Sampling”, ACM SIGGRAPH 2000, Annual Conference Series, 2000, pp. 307-318. |
Chen, S. et al., “A CMOS Image Sensor with On-Chip Image Compression Based on Predictive Boundary Adaptation and Memoryless QTD Algorithm,” Very Large Scalee Integration (VLSI) Systems, IEEE Transactions, vol. 19, Issue 4; Apr. 2011. |
Chen, W., et al., “Light Field mapping: Efficient representation and hardware rendering of surface light fields”, ACM Transactions on Graphics 21, 3, 447-456, 2002. |
Cohen, Noy et al., “Enhancing the performance of the light field microscope using wavefront coding,” Optics Express, vol. 22, issue 20; 2014. |
Daly, D., “Microlens Arrays” Retrieved Jan. 2013. |
Debevec, et al, “A Lighting Reproduction Approach to Live-Action Compoisting” Proceedings SIGGRAPH 2002. |
Debevec, P., et al., “Acquiring the reflectance field of a human face”, SIGGRAPH 2000. |
Debevec, P., et al., “Recovering high dynamic radiance maps from photographs”, SIGGRAPH 1997, 369-378. |
Design of the xBox menu. Retrieved Jan. 2013. |
Digital Photography Review, “Sony Announce new RGBE CCD,” Jul. 2003. |
Dorsey, J., et al., “Design and simulation of opera light and projection effects”, in Computer Graphics (Proceedings of SIGGRAPH 91), vol. 25, 41-50. |
Dorsey, J., et al., “Interactive design of complex time dependent lighting”, IEEE Computer Graphics and Applications 15, 2 (Mar. 1995), 26-36. |
Dowski et al., “Wavefront coding: a modern method of achieving high performance and/or low cost imaging systems” SPIE Proceedings, vol. 3779, Jul. 1999, pp. 137-145. |
Dowski, Jr. “Extended Depth of Field Through Wave-Front Coding,” Applied Optics, vol. 34, No. 11, Apr. 10, 1995; pp. 1859-1866. |
Duparre, J. et al., “Micro-Optical Artificial Compound Eyes,” Institute of Physics Publishing, Apr. 2006. |
Eisemann, Elmar, et al., “Flash Photography Enhancement via Intrinsic Relighting”, SIGGRAPH 2004. |
Fattal, Raanan, et al., “Multiscale Shape and Detail Enhancement from Multi-light Image Collections”, SIGGRAPH 2007. |
Fernando, Randima, “Depth of Field—A Survey of Techniques,” GPU Gems. Boston, MA; Addison-Wesley, 2004. |
Fitzpatrick, Brad, “Camlistore”, Feb. 1, 2011. |
Fujifilm, Super CCD EXR Sensor by Fujifilm, brochure reference No. EB-807E, 2008. |
Georgiev, T. et al., “Reducing Plenoptic Camera Artifacts,” Computer Graphics Forum, vol. 29, No. 6, pp. 1955-1968; 2010. |
Georgiev, T., et al., “Spatio-Angular Resolution Tradeoff in Integral Photography,” Proceedings of Eurographics Symposium on Rendering, 2006.. |
Georgiev, T., et al., “Suppersolution with Plenoptic 2.0 Cameras,” Optical Society of America 2009; pp. 1-3. |
Georgiev, T., et al., “Unified Frequency Domain Analysis of Lightfield Cameras” (2008). |
Georgiev, T., et al., Plenoptic Camera 2.0 (2008). |
Girod, B., “Mobile Visual Search”, IEEE Signal Processing Magazine, Jul. 2011. |
Gortler et al., “The lumigraph” SIGGRAPH 96, pp. 43-54. |
Groen et al., “A Comparison of Different Focus Functions for Use in Autofocus Algorithms,” Cytometry 6:81-91, 1985. |
Haeberli, Paul “A Multifocus Method for Controlling Depth of Field” GRAPHICA Obscura, 1994, pp. 1-3. |
Heide, F. et al., “High-Quality Computational Imaging Through Simple Lenses,” ACM Transactions on Graphics, SIGGRAPH 2013; pp. 1-7. |
Heidelberg Collaboratory for Image Processing, “Consistent Depth Estimation in a 4D Light Field,” May 2013. |
Hirigoyen, F., et al., “1.1 um Backside Imager vs. Frontside Image: an optics-dedicated FDTD approach”, IEEE 2009 International Image Sensor Workshop. |
Huang, Fu-Chung et al., “Eyeglasses-free Display: Towards Correcting Visual Aberrations with Computational Light Field Displays,” ACM Transaction on Graphics, Aug. 2014, pp. 1-12. |
Isaksen, A., et al., “Dynamically Reparameterized Light Fields,” SIGGRAPH 2000, pp. 297-306. |
Ives H., “Optical properties of a Lippman lenticulated sheet,” J. Opt. Soc. Am. 21, 171 (1931). |
Ives, H. “Parallax Panoramagrams Made with a Large Diameter Lens”, Journal of the Optical Society of America; 1930. |
Jackson et al., “Selection of a Convolution Function for Fourier Inversion Using Gridding” IEEE Transactions on Medical Imaging, Sep. 1991, vol. 10, No. 3, pp. 473-478. |
Kautz, J., et al., “Fast arbitrary BRDF shading for low-frequency lighting using spherical harmonics”, in Eurographic Rendering Workshop 2002, 291-296. |
Koltun, et al., “Virtual Occluders: An Efficient Interediate PVS Representation”, Rendering Techniques 2000: Proc. 11th Eurographics Workshop Rendering, pp. 59-70, Jun. 2000. |
Kopf, J., et al., Deep Photo: Model-Based Photograph Enhancement and Viewing, SIGGRAPH Asia 2008. |
Lehtinen, J., et al. “Matrix radiance transfer”, in Symposium on Interactive 3D Graphics, 59-64, 2003. |
Lesser, Michael, “Back-Side Illumination”, 2009. |
Levin, A., et al., “Image and Depth from a Conventional Camera with a Coded Aperture”, SIGGRAPH 2007, pp. 1-9. |
Levoy et al.,“Light Field Rendering” SIGGRAPH 96 Proceeding, 1996. pp. 31-42. |
Levoy, “Light Fields and Computational Imaging” IEEE Computer Society, Aug. 2006, pp. 46-55. |
Levoy, M. “Light Field Photography and Videography,” Oct. 18, 2005. |
Levoy, M. “Stanford Light Field Microscope Project,” 2008; http://graphics.stanford.edu/projects/lfmicroscope/, 4 pages. |
Levoy, M., “Autofocus: Contrast Detection”, http://graphics.stanford.edu/courses/cs178/applets/autofocusPD.html, pp. 1-3, 2010. |
Levoy, M., “Autofocus: Phase Detection”, http://graphics.stanford.edu/courses/cs178/applets/autofocusPD.html, pp. 1-3, 2010. |
Levoy, M., et al., “Light Field Microscopy,” ACM Transactions on Graphics, vol. 25, No. 3, Proceedings SIGGRAPH 2006. |
Liang, Chia-Kai, et al., “Programmable Aperture Photography: Multiplexed Light Field Acquisition”, ACM SIGGRAPH, 2008. |
Lippmann, “Reversible Prints”, Communication at the French Society of Physics, Journal of Physics, 7 , 4, Mar. 1908, pp. 821-825. |
Lumsdaine et al., “Full Resolution Lightfield Rendering” Adobe Technical Report Jan. 2008, pp. 1-12. |
Maeda, Y. et al., “A CMOS Image Sensor with Pseudorandom Pixel Placement for Clear Imaging,” 2009 International Symposium on Intelligent Signal Processing and Communication Systems, Dec. 2009. |
Magnor, M. et al., “Model-Aided Coding of Multi-Viewpoint Image Data,” Proceedings IEEE Conference on Image Processing, ICIP-2000, Vancouver, Canada, Sep. 2000. https://graphics.tu-bs.de/static/people/magnor/publications/icip00.pdf. |
Mallat, Stephane, “A Wavelet Tour of Signal Processing”, Academic Press 1998. |
Malzbender, et al., “Polynomial Texture Maps”, Proceedings SIGGRAPH 2001. |
Marshall, Richard J. et al., “Improving Depth Estimation from a Plenoptic Camera by Patterned Illumination,” Proc. of SPIE, vol. 9528, 2015, pp. 1-6. |
Masselus, Vincent, et al., “Relighting with 4D Incident Light Fields”, SIGGRAPH 2003. |
Meynants, G., et al., “Pixel Binning in CMOS Image Sensors,” Frontiers in Electronic Imaging Conference, 2009. |
Moreno-Noguer, F. et al., “Active Refocusing of Images and Videos,” ACM Transactions on Graphics, Aug. 2007; pp. 1-9. |
Munkberg, J. et al., “Layered Reconstruction for Defocus and Motion Blur” EGSR 2014, pp. 1-12. |
Naemura et al., “3-D Computer Graphics based on Integral Photography” Optics Express, Feb. 12, 2001. vol. 8, No. 2, pp. 255-262. |
Nakamura, J., “Image Sensors and Signal Processing for Digital Still Cameras” (Optical Science and Engineering), 2005. |
National Instruments, “Anatomy of a Camera,” pp. 1-5, Sep. 6, 2006. |
Nayar, Shree, et al., “Shape from Focus”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 16, No. 8, pp. 824-831, Aug. 1994. |
Ng, R., et al. “Light Field Photography with a Hand-held Plenoptic Camera,” Stanford Technical Report, CSTR 2005-2, 2005. |
Ng, R., et al., “All-Frequency Shadows Using Non-linear Wavelet Lighting Approximation. ACM Transactions on Graphics,” ACM Transactions on Graphics; Proceedings of SIGGRAPH 2003. |
Ng, R., et al., “Triple Product Wavelet Integrals for All-Frequency Relighting”, ACM Transactions on Graphics (Proceedings of SIGGRAPH 2004). |
Ng., R., “Fourier Slice Photography,” ACM Transactions on Graphics, Proceedings of SIGGRAPH 2005, vol. 24, No. 3, 2005, pp. 735-744. |
Nimeroff, J., et al., “Efficient rendering of naturally illuminatied environments” in Fifth Eurographics Workshop on Rendering, 359-373, 1994. |
Nokia, “City Lens”, May 2012. |
Ogden, J., “Pyramid-Based Computer Graphics”, 1985. |
Okano et al., “Three-dimensional video system based on integral photography” Optical Engineering, Jun. 1999. vol. 38, No. 6, pp. 1072-1077. |
Orzan, Alexandrina, et al., “Diffusion Curves: A Vector Representation for Smooth-Shaded Images,” ACM Transactions on Graphics—Proceedings of SIGGRAPH 2008; vol. 27; 2008. |
Pain, B., “Back-Side Illumination Technology for SOI-CMOS Image Sensors”, 2009. |
Perez, Patrick et al., “Poisson Image Editing,” ACM Transactions on Graphics—Proceedings of ACM SIGGRAPH 2003; vol. 22, Issue 3; Jul. 2003; pp. 313-318. |
Petschnigg, George, et al., “Digial Photography with Flash and No-Flash Image Pairs”, SIGGRAPH 2004. |
Primesense, “The Primesense 3D Awareness Sensor”, 2007. |
Ramamoorthi, R., et al, “Frequency space environment map rendering” ACM Transactions on Graphics (SIGGRAPH 2002 proceedings) 21, 3, 517-526. |
Ramamoorthi, R., et al., “An efficient representation for irradiance environment maps”, in Proceedings of SIGGRAPH 2001, 497-500. |
Raskar, Ramesh et al., “Glare Aware Photography: 4D Ray Sampling for Reducing Glare Effects of Camera Lenses,” ACM Transactions on Graphics—Proceedings of ACM SIGGRAPH, Aug. 2008; vol. 27, Issue 3; pp. 1-10. |
Raskar, Ramesh et al., “Non-photorealistic Camera: Depth Edge Detection and Stylized Rendering using Multi-Flash Imaging”, SIGGRAPH 2004. |
Raytrix, “Raytrix Lightfield Camera,” Raytrix GmbH, Germany 2012, pp. 1-35. |
Roper Scientific, Germany “Fiber Optics,” 2012. |
Scharstein, Daniel, et al., “High-Accuracy Stereo Depth Maps Using Structured Light,” CVPR'03 Proceedings of the 2003 IEEE Computer Society, pp. 195-202. |
Number | Date | Country | |
---|---|---|---|
20180082405 A1 | Mar 2018 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 14871533 | Sep 2015 | US |
Child | 15824574 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 15439710 | Feb 2017 | US |
Child | 14871533 | US | |
Parent | 13533319 | Jun 2012 | US |
Child | 15439710 | US |