Patent Application
20120257067
1. Field of the Disclosure
The present disclosure relates generally to adjustment of focus in an image capturing device.
2. Description of the Related Art
A focal plane of an optical system is a plane that lies perpendicular to the optical axis and that passes through a focal point of the optical system. Rays of light that enter the optical system and that are parallel to the optical axis converge on the focal point. Typically, objects within the same focal plane appear equally sharp and objects out of a focal plane appear blurred relative to the objects in the focal plane. Some image capturing devices can capture images with more than one focal plane, for example by capturing light field images or by capturing multiple images, each with a different focal plane.
In one embodiment, a method for setting a focal transition for an image capturing device comprises receiving data including user selections of focal settings for respective areas of interest in an image, generating a first interface that includes indicators of the focal settings for the respective areas of interest and indicators of transitional focal settings for respective transitional areas between the areas of interest, receiving data including user selections of the transitional focal settings for the respective transitional areas between the areas of interest, and generating a second interface, the second interface including indicators of the transitional focal settings for the respective transitional areas between the areas of interest.
In one embodiment, a system for setting a focal transition comprises a computer-readable medium, an input interface configured to receive data indicating selections from a user, and one or more processors configured to cause one or more computing devices to generate a first interface indicating focal settings for respective areas of interest in an image, receive data indicating one or more user selections, the one or more user selections indicating focal settings for areas between the areas of interest in the image, and generate a second interface indicating the focal settings for the areas between the areas of interest in the image.
In one embodiment, one or more computer-readable media store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising receiving one or more first user selections, the one or more first user selections indicating transitional focal settings between a first focal setting and a second focal setting, wherein the first focal setting is associated with a first area of an image, the second focal setting is associated with a second area of the image, and the transitional focal settings are associated with a transitional area of the image between the first area and the second area, and generating a first interface indicating the transitional focal settings between the first focal setting and the second focal setting.
The following description is of certain illustrative embodiments, and the disclosure is not limited to these embodiments, but includes alternatives, equivalents, and modifications such as are included within the scope of the claims. Additionally, the illustrative embodiments may include several novel features, and a particular feature may not be essential to practice the systems and methods described herein
For example, an image may indicate two areas of interest that are in focus. The system 100 generates an interface (e.g., generates data that defines a graphical user interface) that indicates the two areas of interest and their respective focal settings, and the system 100 receives selections of focal settings for the transitional area between the areas of interest. The selections may indicate an abrupt transition, a gradual transition, etc., of the focal settings for the transition between the areas of interest. The system 100 also generates adjustments for image capturing parameters (e.g., parameters for one or more lenses, apertures, shutters, light sensors) that implement the selected focal settings. Thus, the focus and/or blurriness of the space between the areas of interest an image can be selected according to user preferences.
The system 100 includes a lens 10 (which may include a plurality of lenses), an aperture 11 (which may include a plurality of apertures, for example a multi-aperture array), a shutter 12, and a light sensor 14 (which may include a plurality of light sensors) that converts incident electromagnetic radiation into electrical signals. Furthermore, in other embodiments the lens 10, the aperture 11, and the shutter 12 may be arranged differently than is shown in the embodiment of
Electromagnetic radiation (also referred to herein as “light”) reflected from a scene (e.g., an object in the scene) passes through the lens 10, the aperture 11, and the shutter 12 (when open) to the light sensor 14 and may form an optical image on a light sensing surface of the light sensor 14. The light sensor 14 converts the light to analog image signals and outputs the signals to an ND converter 16. The ND converter 16 converts the analog image signals to digital image signals. The light sensor 14 can detect light in the spectrum visible to the human eye and/or in the spectrum that the human eye cannot detect (e.g., infrared, x-ray, ultraviolet, gamma rays). In some embodiments, the light sensor 14 can detect light fields, for example four-dimensional light fields.
The system 100 also includes an image processing unit 20, which applies resize processing, such as predetermined interpolation and reduction, and color conversion processing to data from the ND converter 16 or data from a memory 30. The image processing unit 20 performs predetermined arithmetic operations using the captured image data, and the system 100 performs exposure control and ranging control based on the obtained arithmetic result. The system 100 can perform TTL (through-the-lens) AF (auto focus) processing, AE (auto exposure) processing, and EF (flash pre-emission) processing. The image processing unit 20 further performs TTL AWB (auto white balance) operations based on the obtained arithmetic result.
Output data from the A/D converter 16 is written in the memory 30 via the image processing unit 20 and/or memory control unit 22. The memory 30 stores image data that is captured by the light sensor 14 and converted into digital data by the A/D converter 16. The memory 30 may store images (e.g., still photos, videos) and other data, for example metadata and file headers for captured images. The memory 30 may also serve as an image display memory. A D/A converter 26 converts digital data into an analog signal and supplies that analog signal to an image display unit 28. The image display unit 28 presents images according to the analog signal from the D/A converter 26 on a display (e.g., an LCD, an LED display, an OLED display, a plasma display, a CRT display). The system 100 also includes an optical viewfinder 24 that presents at least part of the view detected by the light sensor 14.
An exposure controller 40 controls the shutter 12 and controls the size of the aperture(s) 11. The exposure controller 40 may also have a flash exposure compensation function that links with a flash 48 (e.g., a flash emission device). A focusing controller 42 controls the size of the aperture 11 and/or the position of the lens 10, and a zoom controller 44 controls the angle of view of the lens 10. The exposure controller 40, focusing controller 42, and zoom controller 44 may each partially control the lens 10, aperture 11, and shutter 12, and may also collaborate to calculate parameters for the lens 10, aperture 11, and shutter 12.
The aperture mask generator 46 generates masks that define aperture settings for respective apertures in an array of apertures. Additionally, the light sensor 14 may include super-pixels that each include a group of pixels. Each pixel in the group may be able to be independently associated with an aperture in a multi-aperture array (e.g., one pixel to one aperture, many to pixels to one aperture). The group of pixels may include various numbers of pixels (e.g., 2, 4, 8, 16), however, resolution may decrease as the number of pixels in the group increases. Thus, since light that passes through different apertures may be detected by at least one pixel in a group, a super-pixel may detect rays of light that each passes through a different aperture.
A memory 56 (as well as the memory 30) includes one or more computer readable and/or writable media, and may include, for example, a magnetic disk (e.g., a floppy disk, a hard disk), an optical disc (e.g., a CD, a DVD, a Blu-ray), a magneto-optical disk, a magnetic tape, semiconductor memory (e.g., a non-volatile memory card, flash memory, a solid state drive, SRAM, DRAM), an EEPROM, etc. The memory 56 may store computer-executable instructions and data for the operation of a system controller 50. The system controller 50 includes one or more processors (e.g., microprocessors) and reads and performs computer-executable instructions, such as instructions stored in the memory 56. Note that the computer-executable instructions may include those for the performance of various methods described herein. The memory 56 is an example of a non-transitory computer-readable medium that stores computer-executable instructions.
The memory 56 includes a focal transition module 58. A module includes computer-readable instructions that may be executed by one or more members of the system 100 to cause the system 100 to perform certain operations, though for ease of description a module may be described as performing the operations. Modules may be implemented in software (e.g., JAVA, C, C++, C#, Basic, Assembly), firmware, and/or hardware. In other embodiments, the system 100 may include more modules and/or the module may be divided into more modules. The instructions in the focal transition module 58 may be executed to cause the system 100 to set focal transitions, adjust focal transitions, and generate interfaces that indicate focal transitions and/or perform the methods described herein. Modules may be implemented in any applicable computer-readable storage medium that can be employed as a storage medium for supplying the computer-executable instructions. Furthermore, when the computer-executable instructions are executed, an operating system executing on the system 100 may perform at least part of the operations that implement the instructions.
The system 100 also includes a mode selector 60 that sets the operation mode of the system 100 to still image recording mode, video recording mode, playback mode, etc. A shutter switch 64 may be activated in the middle of operation (half stroke) and generate a first shutter switch signal. Also, the shutter switch 64 may be activated upon a full stroke and generate a second shutter switch signal. The system controller 50 may start one or more operations (e.g., AF processing, AE processing, AWB processing, EF processing) in response to the first shutter switch signal. Also, in response to the second shutter switch signal, the system controller 50 may perform and/or initiate one or more operations, including the following: reading image signals from the light sensor 14, converting image signals into image data by the A/D converter 16, processing of image data by the image processor 20, writing image data to the memory 30, reading image data from the memory 30, compression of the image data, and writing data to the recording medium 96.
A zoom selector 62 is operable by a user to change the angle of view (zooming magnification or shooting magnification). The zoom selector 62 may include, for example, a slide-type member, a lever, switch, a wheel, a knob, and/or a switch.
The operation unit 66 may include various buttons, touch panels and so on. In one embodiment, the operation unit 66 includes one or more of a menu button, a set button, a macro selection button, a multi-image reproduction/repaging button, a single-shot/serial shot/self-timer selection button, a forward (+) menu selection button, a backward (−) menu selection button, etc. The operation unit 66 may also set and change the flash operation mode. The settable modes include for example, auto, flash-on, red-eye reduction auto, and flash-on (red-eye reduction). The operation unit 66 may be used to select a storage format for the captured image information, including JPEG (Joint Photographic Expert Group) and RAW formats. The operation unit 66 may set the system 100 to a plural-image shooting mode, wherein a plurality of images is captured in response to a single shooting instruction (e.g., a signal from the shutter switch 64). This may include auto bracketing, wherein one or more image capturing parameters (e.g., white balance, exposure, aperture settings) are altered in each of the images.
A power supply controller 80 detects the existence/absence of a power source, the type of the power source, and a remaining battery power level, and supplies a necessary voltage and current to other components as required. A power source 82 includes a battery, such as an alkaline battery, a lithium battery, a NiCd battery, a NiMH battery, and an Li battery, an AC adapter, a DC adapter, etc.
The recording media 96 includes a recording unit 94 that is configured with one or more computer-readable and/or computer-writable media. The system 100 and the recording media 96 communicate via an interface 90 of the system 100 and an interface 92 of the recording media. Although the illustrated embodiment of the system 100 includes one pair of interfaces 90, 92 and one recording media 96, other embodiments may include additional recording media and interfaces.
The system 100 also includes a transition selector 68 that sets and/or adjusts focal transitions between areas of interest. The transition selector 68 may include, for example, one or more buttons, touch screens, dials, joysticks, wheels, levers, cameras, microphones, etc. For example, a user may touch a point on a touch screen that corresponds to a focal setting for a respective transitional area or a user may operate a joystick up, down, left, and/or right to select a transitional area and adjust the respective focal setting based on an interface that indicates the area(s) of interest, the transitional area(s), and/or the respective focal settings. The selector may also receive selections via other inputs, including, for example, voice command(s), eye gazing, eye movement, multi-touch, etc.
The transition adjustment device 200 includes one or more processors 201 (also referred to herein as “CPU 201”), which may be conventional or customized central processing units (e.g., microprocessor(s)). The CPU 201 is configured to read and execute computer readable instructions, and the CPU 201 may command/and or control other components of the transition adjustment device 200. The transition adjustment device 200 also includes I/O interfaces 203, which provide communication interfaces to input and output devices, including a keyboard, a display (e.g., the display device 250), a mouse, a printing device, a touch screen, a light pen, an optical storage device, a scanner, a microphone, a camera, a drive, etc.
The transition adjustment device 200 additionally includes a memory 205, which includes one or more computer-readable and/or writable media. The network interface 207 allows the transition adjustment device 200 to communicate with the network 270 and other devices via the network 270, and the network interface 207 may have wired and/or wireless capabilities. The storage device 209 stores data, computer-executable instructions, and/or modules and may include, for example, a magnetic storage device (e.g., a hard drive), an optical storage device, and/or a solid state drive. The components of the transition adjustment device 200 are connected via a bus. Also, the transition adjustment device 200 includes an operating system, which manages one or more of the hardware, the processes, the interrupts, the memory, and the file system.
The transition adjustment device 200 also includes a focal transition module 210, an interface generation module 215, and a mask generation module. The focal transition module 210 determines one or more focal settings for one or more areas (e.g., areas of interest, transitional areas) in an image and receives user selections of focal settings (e.g., via the I/O interface 203) for respective areas in the image.
The interface generation module 215 generates an interface that indicates areas (e.g., areas of interest, transitional areas) in an image and their respective focal settings. The interface generation module 215 may also allow a user to configure the interface, for example by selecting one or more settings that will be included in the interface, the format of the interface, etc. For example, the interface generation module 215 may let a user select an interface to show aperture settings, a three-dimensional representation of the focal settings for the image, a two dimensional representation of the focal settings for the image, etc. The interface generation module 215 may also generate a preview of the image that shows the image with the selected focal settings applied to the image. The interface may then be transmitted to the display device 250 for display.
The mask generation module 220 generates aperture parameters masks and/or electronic aperture masks based on the focal settings, such as masks that implement the focal settings. An electronic aperture may indicate which aperture is selected for each super-pixel in a system that includes super-pixels and multi-aperture arrays. An aperture parameter mask is a spatial mask that indicates which is the aperture for each region or pixel of the image. It could be encoded as fstop (such as 3.5, 4, 4.5, 5, 5.6, 6.3, 7.1, 8, 9, 10, 11, 13, 14, 16, 18, 20, 22) or any other numerical parameter that can be related to an f-number. The aperture parameter mask then could be correlated to the electronic aperture mask that indicates which pixel in each super-pixel is selected for each super-pixel in the image. The mask generation module 220, interface generation module 215, and/or the focal transition module 210 may be configured with the capabilities of the image capturing device 240 (e.g., an aperture array) and not allow selection of focal settings that are not within the capabilities of the image capturing device.
In block 300, areas of interest in an image are identified. The areas of interest may be identified by a user (e.g., by user selection) or by a device (e.g., a computing device). In block 310, selections of transitional focal settings are acquired, for example from a user via an input device. The selections of transitional focal settings may include the selections of focal settings for a discrete number of points in the transitional area, the selection of the shape of a line that corresponds to focal settings, etc. In block 320 an interface is generated that indicates the transitional focal settings. Examples of interfaces are illustrated in
If in block 410 it is determined that the transition selection is not complete, the flow proceeds to block 430, where an area of interest in the image is selected, for example by a user or by a computing device, and the coordinates of the area of interest 435 are determined. In block 440 any adjustment of focus of the area of interest is determined, for example by a user (who may indicate the adjustment via one or more controls and/or interfaces) or by a computing device, and the aperture setting(s) for the area of interest 445 is generated. Next, in block 450, the coordinates of the area of interest and the aperture setting(s) are stored, for example in a computer readable medium. Moving to block 455, it is determined if there are any additional areas of interest in the image. If there are any additional areas of interest, the flow returns to block 430, where an area of interest is selected. If there are not any additional areas of interest, the flow proceeds to block 460.
In block 460, a user interface (e.g., a graphical user interface) for transitional focal setting adjustment is generated. The user interface may include indicators of the areas of interest, the transitional areas, and/or their respective focal settings and may be modified to show changes to the focal settings and/or areas of interest. Moving to block 465, one or more transitional focal settings are adjusted, for example by receiving user selections of transitional focal settings via the user interface. Next, in block 470, it is determined if the adjustments are finished. If the adjustments are not finished, flow returns to block 465. If the adjustments are finished, flow proceeds to block 475, where an aperture parameter mask 480 is updated (or generated, depending on the embodiment) that indicates the settings for respective apertures and/or one or more other settings (e.g., zoom settings, shutter settings), and in block 485 an electronic aperture mask 490 is generated based on the aperture parameter mask 480. Flow then returns to block 400, where the electronic aperture mask 490 is used by the multi-aperture imaging system to detect light. In some embodiments, aperture settings for different image capturing operations are generated, for example for an image capturing device that has a single aperture and that captures images with sequential image capturing operations that have different respective aperture settings. Furthermore, the image preview 405 may be generated to provide immediate feedback regarding changes made to any areas of interest and focal settings for the areas of interest and transitional areas.
The above described systems and methods can be achieved by supplying one or more storage media having stored thereon computer-executable instructions for realizing the above described operations to one or more devices that are configured to read the computer-executable instructions stored in the one or more storage media and execute them. In this case, the devices perform the operations of the above-described embodiments when executing the computer-executable instructions read from the one or more storage media. Also, an operating system on the one or more devices may implement the operations of the above described embodiments. Thus, the computer-executable instructions and/or the one or more storage media storing the computer-executable instructions therein constitute an embodiment.
Any applicable computer-readable storage medium (e.g., a magnetic disk (including a floppy disk and a hard disk), an optical disc (including a CD, a DVD, a Blu-ray disc), a magneto-optical disk, a magnetic tape, and a solid state drive (including flash memory, DRAM, SRAM) can be employed as a storage medium for the computer-executable instructions. The computer-executable instructions may be written to a computer-readable storage medium provided on a function-extension board inserted into the device or on a function-extension unit connected to the device, and a CPU provided on the function-extension board or unit may implement the operations of the above-described embodiments.
While the above disclosure describes illustrative embodiments, it is to be understood that the invention is not limited to the above disclosure. To the contrary, the invention covers various modifications and equivalent arrangements within the spirit and scope of the appended claims.
