Patent Grant
10430075
In photography, depth of field (DOF) refers to the distance between the nearest and furthest objects in an image that appear acceptably sharp. In addition to the distance between the subject and the camera lens, various properties and settings of the lens (e.g., focal length, f-number, and format size) affect the DOF that can be achieved. A shallow DOF tends to emphasize the subject of interest while de-emphasizing the foreground and background. In traditional photography, achieving a shallow DOF typically requires more expensive equipment.
Some existing image processing applications offer a post-processing feature to mimic the effects of a shallow DOF. After a picture is taken and stored to disk, the user selects an area in the image to emphasize (e.g., by drawing a boundary box using the mouse), and the software applies blurring effects to the rest of the image. Such techniques, however, tend to be computationally intensive, and are typically unsuitable for use on mobile devices such as smartphones.
Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.
The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.
An image processing technique for selectively blurring portions of an image is described. In some embodiments, shader code is applied to an original image to generate a blurred image version, and a mask is applied to the original image and the blurred image to render a composite image that includes an unblurred portion and a blurred portion. In some embodiments, the mask is configured by the user via a touchscreen interface. The user can specify configuration parameters including mask location, shape, size, etc.
Processor 102 is coupled bi-directionally with memory 110, which can include, for example, one or more random access memories (RAM) and/or one or more read-only memories (ROM). As is well known in the art, memory 110 can be used as a general storage area, a temporary (e.g., scratch pad) memory, and/or a cache memory. Memory 110 can also be used to store input data and processed data, as well as to store programming instructions and data, in the form of data objects and text objects, in addition to other data and instructions for processes operating on processor 102. Also as is well known in the art, memory 110 typically includes basic operating instructions, program code, data, and objects used by the processor 102 to perform its functions (e.g., programmed instructions). For example, memory 110 can include any suitable computer readable storage media described below, depending on whether, for example, data access needs to be bi-directional or uni-directional. For example, processor 102 can also directly and very rapidly retrieve and store frequently needed data in a cache memory included in memory 110.
A removable mass storage device 112 provides additional data storage capacity for the computer system 100, and is optionally coupled either bi-directionally (read/write) or uni-directionally (read only) to processor 102. A fixed mass storage 120 can also, for example, provide additional data storage capacity. For example, storage devices 112 and/or 120 can include computer readable media such as magnetic tape, flash memory, PC-CARDS, portable mass storage devices such as hard drives (e.g., magnetic, optical, or solid state drives), holographic storage devices, and other storage devices. Mass storages 112 and/or 120 generally store additional programming instructions, data, and the like that typically are not in active use by the processor 102. It will be appreciated that the information retained within mass storages 112 and 120 can be incorporated, if needed, in standard fashion as part of memory 110 (e.g., RAM) as virtual memory.
In addition to providing processor 102 access to storage subsystems, bus 114 can be used to provide access to other subsystems and devices as well. As shown, these can include a display 118, a network interface 116, an input/output (I/O) device interface 104, an image processing device 106, as well as other subsystems and devices. For example, image processing device 106 can include a camera, a scanner, etc.; I/O device interface 104 can include a device interface for interacting with a touchscreen (e.g., a capacitive touch sensitive screen that supports gesture interpretation), a microphone, a sound card, a speaker, a keyboard, a pointing device (e.g., a mouse, a stylus, a human finger), a Global Positioning System (GPS) receiver, an accelerometer, and/or any other appropriate device interface for interacting with system 100. Multiple I/O device interfaces can be used in conjunction with computer system 100. The I/O device interface can include general and customized interfaces that allow the processor 102 to send and, more typically, receive data from other devices such as keyboards, pointing devices, microphones, touchscreens, transducer card readers, tape readers, voice or handwriting recognizers, biometrics readers, cameras, portable mass storage devices, and other computers.
The network interface 116 allows processor 102 to be coupled to another computer, computer network, or telecommunications network using a network connection as shown. For example, through the network interface 116, the processor 102 can receive information (e.g., data objects or program instructions) from another network, or output information to another network in the course of performing method/process steps. Information, often represented as a sequence of instructions to be executed on a processor, can be received from and outputted to another network. An interface card or similar device and appropriate software implemented by (e.g., executed/performed on) processor 102 can be used to connect the computer system 100 to an external network and transfer data according to standard protocols. For example, various process embodiments disclosed herein can be executed on processor 102, or can be performed across a network such as the Internet, intranet networks, or local area networks, in conjunction with a remote processor that shares a portion of the processing. Additional mass storage devices (not shown) can also be connected to processor 102 through network interface 116.
In addition, various embodiments disclosed herein further relate to computer storage products with a computer readable medium that includes program code for performing various computer-implemented operations. The computer readable medium includes any data storage device that can store data which can thereafter be read by a computer system. Examples of computer readable media include, but are not limited to: magnetic media such as disks and magnetic tape; optical media such as CD-ROM disks; magneto-optical media such as optical disks; and specially configured hardware devices such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and ROM and RAM devices. Examples of program code include both machine code as produced, for example, by a compiler, or files containing higher level code (e.g., script) that can be executed using an interpreter.
The computer system shown in
In some embodiments, user interface facilities are provided for the user to input mask configuration information (e.g., via a touchscreen interface), which is received by mask engine 208. Based on the configuration information, mask engine 208 generates a desired mask 210. Effects engine 212 combines original image 202, blurred image version 206, and mask 210 to generate a composite image 214 in which areas are selectively in focus or blurred. As will be described in greater detail below, the area of interest that is covered by the mask will stay in focus and the rest of the image will appear blurry. In some embodiments, system 200 is implemented using a device with a built-in camera and a touchscreen, such as an iPhone® or an Android® phone.
Process 300 may be implemented on systems such as 100 or 200.
Process 300 initiates at 302 when an image is received. The image can be captured live from a built-in camera, imported from a storage location (e.g., a camera roll or photo directory built into the system or accessible via a network), received in memory as a result of other processing operations such as a conventional filtering process, or obtained from any other appropriate source. Optionally, the image is preprocessed. For example, the dimensions of the image can be adjusted to a square image for the ease of processing; the orientation of the image can be adjusted depending on how images are stored on the device, etc.
At 304, shader code is applied to the image to generate a blurred image version (BIV). In some embodiments, a graphics programming language such as OpenGL is used to implement the shader code. In other words, the entire image is preprocessed so that a partially blurred image can be generated rapidly in response to user inputs.
At 306, configuration information pertaining to a mask is received. In some embodiments, the configuration information includes an indication of a position in the image, and optionally the size and shape of the mask. In some embodiments, the indication is received via an I/O device interface, such as a touchscreen (e.g., a capacitive touch sensitive screen), a pointing device, or other appropriate devices. In a preferred embodiment where the device includes a touchscreen, the user indicates the position in the image by placing his finger on the screen to indicate the approximate location in the image that should remain in focus (unblurred). Optionally, the user can use touchscreen gestures such as dragging to move the mask, as well as pinching and spreading of fingers to adjust the size of the unblurred region that should stay in focus, which in turn adjusts the size and shape of the mask.
At 308, a mask is generated that includes an unblurred region and a blurred region, based at least in part on the configuration information. For example, the location, shape, and size of the unblurred region are determined based on the position, shape, and size information included in the configuration information, respectively. In some embodiments, if the user does not specify any of the configuration options, default values are chosen and used. For example, a default mask is a circle located in the center of the image, having a diameter that is ¼ of the image width.
At 310, the mask is applied to combine the image and the BIV to render a composite image that includes an unblurred portion and a blurred portion. The unblurred portion includes a portion of the image corresponding to the unblurred region of the mask, and the blurred portion includes a portion of the BIV corresponding to the blurred region of the mask. Thus, the combined image gives the visual effects of having a shallow DOF.
In some embodiments, steps in the process are repeated. For example, the user can update configuration information pertaining to the mask; thus, steps 306-310 are repeated in response. As another example, images are continuously captured by a built-in camera in a live capture/video mode, and thus steps 302-310 are continuously repeated to provide the user with updated, selectively blurred images based on the current image and mask configuration. In some embodiments, the result image is saved when the user finalizes the mask configuration. In some embodiments, the user is further given the option to apply additional filters, share the final image by emailing, post to websites, etc.
In some embodiments, the shape, size, and position of the mask are configurable by the user.
Pr=Po*Pm+Pb*(1−Pm),
where Pr represents a pixel value in the resulting image, Po represents a pixel value of the corresponding location in the original image, Pm represents a pixel value of the corresponding location in the mask, and Pb represents a pixel value of the corresponding location in the blurred image.
As shown in the resulting image, area of interest 430, which is covered by the mask, is in focus. The rest of the image is blurry. An image that has the effects of a shallow DOF is thus created.
When the user adjusts the mask, the processing application continuously updates the resulting image using the current mask so that the user can see in real-time the effects from applying the mask. In some cases, the effects of the resulting image can be subtle visually. In embodiments where the result is constantly updated while the user adjusts the mask, to provide better feedback to the user during the mask configuration process, the image processing technique further enhances the visual differences of the unblurred region and the blurred region. An additional color is applied to the blurred region in some embodiments. For example, a secondary mask that has the same shape as the current mask but is set to a particular color (e.g., white) in the blurred region can be added to the resulting image to make the blurred region appear to be tinted with the color, giving the blurred region a fog-like, more obscured effect. As another example, the same mask can be applied to the resulting image in a different way (for instance, the inverse of the mask can be added to the resulting image).
In some embodiments, process 300 is implemented on a device with a multi-touch touchscreen supporting gesture interpretation. The user is given the option to use touchscreen gestures to configure the mask.
The user can also resize the mask by using touch screen gestures, such as pinching or spreading two fingers on the touchscreen. The example of
At 602, a first one-dimensional kernel is applied to pixels in the original image to generate an intermediate result, which in this case is an image that is blurred in one direction (e.g., along the x-axis of the original image). As will be described in greater detail below, the first one-dimensional kernel is precomputed based on a blur function (e.g., a Gaussian blur function or box blur function) to achieve the desired blurring effects. Furthermore, the application of the one-dimensional kernel includes subsampling the neighboring pixels of each pixel to be blurred, which further improves efficiency.
At 604, the intermediate result is rotated. In the example illustrated here, the intermediate result image is rotated 90°.
At 606, a second one-dimensional kernel is applied to the rotated intermediate result to obtain a blurred image. Depending on implementation, the second one-dimensional kernel can be the same as the first one-dimensional kernel, or different from the first one-dimensional kernel (e.g., based on a blur function with different coefficients, a different type of blur function, etc.). In the case where the second one-dimensional kernel is the same as the first one-dimensional kernel, the intermediate result is used as the input to the same processing code of 604. The application of the second one-dimensional kernel blurs the image in a second direction (e.g., along the y-axis of the original image).
At 608, the blurred image is rotated again (by 90° in this example). In some embodiments, the rotation is performed because the same code is executed twice (once to blur the image along the x-axis and once to blur the image along the y-axis). Depending on the specific requirements of the system, the blurred image can be stored directly, or be rotated 180° and stored.
While the resulting image in each step can be stored in a separate buffer, in some embodiments, the technique reuses certain image buffers to save memory. For example, in some embodiments, the original image is stored in a first buffer; the intermediate image resulting from step 602 is stored in a second buffer; the rotated intermediate image resulting from step 604 is stored in a third buffer; the blurred image resulting from step 606 is stored into the second buffer; finally, the rotated blurred image resulting from step 608 is stored into the third buffer.
Before the operations of the one-dimensional kernel are explained, the operations of a two-dimensional kernel are first discussed.
P0′=Σi=080ciPi
Accordingly, in this example, 80 neighboring pixels are obtained to carry out the blur operation for each pixel in the original image, which is inefficient and can cause performance degradation.
The Gaussian blur function is represented as follows:
where σ corresponds to the standard deviation and determines the shape of the Gaussian distribution curve.
Rather than computing a sum of coefficients multiplied with the respective neighboring pixels in the form of P0′=Σi=08ciPi, the neighboring pixels are subsampled to further reduce the amount of computation required. Specifically, in some embodiments, a graphics library such as OpenGL is used to implement the shader code that applies the one-dimensional kernel. The library provides function calls for subsampling pixels that are optimized to take advantage of the way pixel arrays are stored in memory. For example, the glReadPixels function receives as inputs the location of a first pixel and an offset from a reference location, and returns a new, interpolated pixel. In some embodiments, the interpolated pixel value corresponds to a weighted sum of the first pixel and its neighbor, where the weight of each contributing pixel is determined based on the offset, as follows:
glReadPixels(Pr,offset)=Pnew=w1Px+w2Px+1 (2)
where w1=1−0.5offset; w2=0.5offset
In some embodiments, once an offset is chosen, a scaling factor is applied to the results to obtain the desired coefficient values. In one example, according to blur function 802, the values of c1 and c2 are 0.8 and 0.6, respectively. An offset (Offset1) of 0.857 and a scaling factor of 1.4 result in the following:
1.4*glReadPixels(P1,0.857)=1.4*(0.571P1+0.429P2)=0.8P1+0.6P2 (3)
glReadPixels subsamples P1 and P2 at Offset1, and the subsampled result is multiplied with the scaling factor. The subsampled and scaled result is approximately the same as the sum of P1 and P2 multiplied with their respective blur function coefficients c1 and c2. The offset (Offset2) for where to subsample P3 and P4, and the scaling factor for coefficients c3 and c4 can be obtained similarly. Thus, by selecting an appropriate offset and multiplying glReadPixels with an appropriate scaling factor, a single call to glReadPixels can produce a sum of two blur coefficients multiplied with two neighboring pixels, which improves computation efficiency. Although the kernel shown in this example includes 8 pixels (4 neighboring pixels on each side of P0), it only requires four calls to obtain the pixel values for computation. In this example, after the one-dimensional kernel is applied to each pixel in the original image, the intermediate result image is rotated, and the one-dimensional kernel is applied again to pixels in the rotated image.
In some embodiments, a single blur function is used, and the corresponding offsets and scaling factors are precomputed and stored in a table or other appropriate data structure, to be used by the shader code to generate the blurred image version. Table 1 shows an example one-dimensional kernel that accounts for four neighboring pixels on one side of the pixel of interest, and takes two samples. The offset and scaling factor associated with the sample taken between P1 and P2, as well as P3 and P4 are precomputed and stored to the table. The values can change depending on the selection of σ.
Since different blur functions (e.g., Gaussian blur functions with different σ values) have different coefficients that map to the one-dimensional kernel, in some embodiments, multiple sets of offsets and scaling factors are precomputed and stored so the user can select different blurring effects. In some embodiments, to achieve approximately the same amount of blurring visually for different original images having different resolutions, different kernel sizes are used. For example, for images that are 320 pixels by 320 pixels (320×320), a one-dimensional kernel similar to the one illustrated in
Image processing to generate a selectively blurred image has been described. The technique is computationally efficient, thus enabling the user to configure a mask interactively and receive instant feedback of the result from application of the mask.
Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.
This application is a continuation of co-pending U.S. patent application Ser. No. 14/656,508, entitled IMAGE PROCESSING FOR INTRODUCING BLURRING EFFECTS TO AN IMAGE filed Mar. 12, 2015 which is incorporated herein by reference for all purposes, which is a continuation of U.S. Pat. No. 9,007,402, entitled IMAGE PROCESSING FOR INTRODUCING BLURRING EFFECTS TO AN IMAGE filed Sep. 18, 2012 which is incorporated herein by reference for all purposes.
| Number | Name | Date | Kind |
|---|---|---|---|
| 9007402 | Systrom | Apr 2015 | B2 |
| 20060181549 | Alkouh | Aug 2006 | A1 |
| 20110185297 | Reid | Jul 2011 | A1 |
| 20110193980 | Ogasawara | Aug 2011 | A1 |
| 20120019688 | Bugnariu | Jan 2012 | A1 |
| Number | Date | Country |
|---|---|---|
| 1353296 | Oct 2003 | EP |
| 1383082 | Jan 2007 | EP |
| 2006072418 | Mar 2006 | JP |
| 2007146574 | Dec 2007 | WO |
| 2009158209 | Dec 2009 | WO |
| 2012073575 | Jun 2012 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20170344252 A1 | Nov 2017 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14656508 | Mar 2015 | US |
| Child | 15679030 | US | |
| Parent | 13622290 | Sep 2012 | US |
| Child | 14656508 | US |
