A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by any-one of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
This application is related to the following co-pending patent applications, which are hereby incorporated by reference in their entirety:
U.S. patent application Ser. No. 11/249,130, entitled Image Capture Using Display Device as Light Source, filed Oct. 11, 2005.
U.S. patent application Ser. No. 10/424,253, entitled METHOD AND APPARATUS FOR MANIPULATING AN IMAGE TO FORM AN ICON, filed Apr. 28, 2003.
U.S. patent application Ser. No. 10/742,957, entitled CREATING A THEME USED BY AN AUTHORING APPLICATION TO PRODUCE A MULTIMEDIA PRESENTATION, filed Dec. 22, 2003.
U.S. patent application Ser. No. 10/337,924, entitled METHOD AND APPARATUS FOR CREATING MULTIMEDIA PRESENTATIONS, filed Jan. 6, 2003.
This specification describes systems and methods for image capture and manipulation.
Systems and methods for providing image capture and manipulation are described. Many software applications now accept image data such as head shots to identify, for example, the sender of an electronic mail message, an Internet chat buddy, or to identify a caller on a voice-over-IP communication link. Novice computer users may not know how to associate their picture with these applications. Furthermore, a user may wish to modify their picture before using it. While many software packages provide very sophisticated tools for image editing, novice users can find these packages too cumbersome, especially if the user wants to quickly experiment with different effects.
This specification describes systems and methods for image capture and manipulation.
In one aspect, an input stream including image data is received from a source. The input stream is displayed in real-time including displaying a plurality of instantiations of the stream at a same time, each stream different, the step of displaying including determining a filter to apply to each instantiation of the input stream and applying the filter to a respective instantiation. A prompt is received to select one of the instantiations of the stream. One frame is selected from the selected stream to display as a still image including applying the selected filter to the still image prior to display.
Implementations may include one or more of the following features. Receiving can be from a camera attached to a computer or from a phone. There can be nine instantiations of the input stream. A filter can be applied to the image data. User selection of a filter to apply to each stream can be received. Two mosaics can be selectively displayed, each with a predetermined number of instantiations of the stream.
In another aspect, input data is received from an image source. Plural variations of the input data are displayed at a same time, each variation a filtered version of the input data. A prompt is received to select one of the plural variations. The image source is prompted to produce new input data. A still image is displayed including applying a filter to the new input data, the filter being associated with the selected variation.
Implementations may include one or more of the following features. Applying can utilize an effect. An effect can include one or more of the following: cropping, flipping, adding a border, adding a backdrop, scaling, color manipulation, blurring, sharpening, distorting, transforming geometry, or pattern generation. A filter can be a programmatic representation of an effect. Displaying the plural variations can include determining a unique cell to display each variation of the input data in. A flash can be provided through a display device attached to a computer. Capturing the new input data can be in coincidence with the flash.
In another aspect, plural variations of an image stream are simultaneously displayed, each having an associated effect, including applying a given effect to the image stream to produce a given variation. Input is received to capture the image stream. Input is received to select a variation to be captured. The captured selection is displayed.
Implementations may include one or more of the following features. The image stream can be a time-ordered sequence of images originating from a digital camera. The captured selection can be flipped. The captured selection can be displayed as collection of captured selections. User selection of an effect to apply to each variation can be received. The captured selection can be provided to one of: an electronic mail message, an address book application, an account profile, an interactive chat profile, or an image organizer application.
In another aspect, an input is received to select an image in a first user interface having a collection of images, the image transformed by a first effect. Multiple variations of the selected image are displayed simultaneously in a second user interface without the transformation of the first effect, each variation having an associated second effect, including applying a given second effect to the selected image to produce a given variation. An input is received to select a variation. The selected image is transformed by the second effect associated with the selected variation.
Implementations may include one or more of the following features. The first user interface and the second user interface can be displayed simultaneously but may not overlap.
In another aspect, a first image data is selected from live image data, the live image data originating from a digital camera. A first image effect is selected from a plurality of image effects, each image effect in the plurality being associated with a unique display region. The first image effect is applied to the first image data, or to a copy of the first image data, to create a first image, the first image displayed on a first display region associated with the first image effect. Selection of the first display region is detected. Second image data is selected from the live image data, the second image data occurring at a point in time later than the first image data. The first image effect is applied to the second image data, or to a copy of the second image data, to create a second image, the second image displayed on a second unassociated display region.
Implementations may include one or more of the following features. The point in time can be an offset from when the input is detected. The digital camera can be a phone, a portable music player, or a portable electronic game playing device. The live image data can be a continuous time-ordered sequence of packets or frames containing image data. The selection of the first display region includes detecting input incident on the first display region.
Implementations of the invention can realize one or more of the following advantages. A user can easily obtain and apply effects to an image for use in various applications. Different effects (e.g., using one or more filters) can simultaneously be applied to an image (or image stream) with the results being displayed side-by-side. This allows a user to immediately compare different effects and choose the most desirable one. Image data can easily be transferred to other applications such as email, an address book, contacts, Internet chat, and image organization applications. In one embodiment, the underlying image data is not changed. An image strip of the pictures, preferably recent pictures, is provided. One button is provided for image capture and selection. And a count-down timer can be provided before an image is captured.
These general and specific aspects may be implemented using a system, a method, or a computer program, or any combination of systems, methods, and computer programs. The details of one or more implementations of the invention are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the invention will be apparent from the description and drawings, and from the claims.
Like reference symbols in the various drawings indicate like elements.
Although a GUI is illustrated, other user interfaces can be used in systems and methods for providing image capture and manipulation, including user interfaces that allow for user interaction by means of sound, voice, gesture, eye movement and/or use of remote control devices. A still or live image is displayed in region 102 of window 100. For example, the image may be obtained from an image capture device (e.g., a digital still camera or a digital video camera) integrated with or mounted in or located in proximity to a user's computer display. A camera button 106 allows the user to switch between a still image and a live image. A live image reflects a continuous stream of images captured by the image capture device. The camera button 106 can also be used to allow the user to capture an image (see below).
Image strip 104 presents optionally scaled versions of previously captured images. In this illustration, there are no images captured and so the image strip 104 displays blank images (110, 116). Alternatively, the image strip 104 is empty if there are no captured images. Alternatively, the image strip is not presented. The image strip 104 can hold more images than can be displayed. A user can scroll to the left or to the right through images using scroll buttons (112 and 114). The images in the image strip can be sorted in order of the date and/or time that they were captured, the date and/or time they were last modified, or other suitable criteria. In one embodiment, the image strip 104 can show more than one row and can scroll vertically. For example, the image strip 104 might show six images across the first row and the next six images in the second row, etc. In another embodiment, the image strip 104 can be located at other locations and in other orientations, e.g. vertical along the left side of the window 100.
An effects button 108 presents a filter mosaic (or “mosaic”) in region 102 (see
A mosaic is displayed in region 102 of window 100. Region 102 is subdivided into cells in which copies or instantiations of a still or live image are simultaneously rendered, each with an applied effect. An effect is a transformation or distortion of a still or live image. An effect can include, without limitation, one or more of the following: cropping, providing a mirror image (“flipping”), adding a border surrounding the image, adding a backdrop, scaling, color manipulation, blurring, sharpening, distorting, transforming geometry, and pattern generation. By way of illustration, instantiation cell 200 shows a “Sepia” effect applied to an image instantiation. Center cell 208 shows the still or live image without applying an effect or by applying a null effect. Other effects are possible as illustrated in instantiation cells 202, 204, 206, 208, 210, 212, 214, and 216 of
When conventional handheld digital cameras (still or video) capture pictures, the pictures as captured are not flipped. An effect that flips an image is useful if any text will be visible in the image since photographed text appears backwards or if the user is taking a self portrait. A user can optionally specify that the “flip” effect always be applied before or after any other effects are applied. In one embodiment, the “live preview” mode (see below) presents a live, flipped image. In this way, the live preview provides the user with intuitive real-time feedback, as if they were looking into a mirror. In one embodiment, the “flip” effect is applied to captured images or image instances by default, whether or not other effects are applied.
In one implementation, an effect can be applied to an instantiation using the Core Image application programming interface (API) which is part of the Quartz Core framework image processing technology available from Apple Computer, Inc. of Cupertino, Calif. The Core Image API provides access to built-in image filters for both video and still images and provides support for creating custom filters.
Additional effects beyond those displayed in region 102 are available by selecting different mosaics, where the different mosaics have different sets of effects associated with the cells. In one implementation, selection of the effects button 108 can allow a user to view different mosaics (e.g.,
Although there are nine instantiation cells in this figure, more or fewer are possible. In one implementation, the number of cells displayed is user configurable. The cells can also be made larger if there are fewer cells, or smaller to accommodate a greater number of cells. Different size cells are possible at a same time. Associated image instantiations can be scaled to accommodate a given cell size. In another implementation, a mosaic can be presented in a scrollable area. For example, more than nine cells could be shown and a scroll bar could be provided. In yet another implementation, cells can be reordered by dragging and dropping the cells.
In one embodiment, a user can define for a given cell what effect or series of effects to apply. In another embodiment, a user can drag and drop cells onto each other to create cells with multiple effects. For example, if a first cell having a first effect is dropped on a second cell having a second effect, the second cell will have the first effect and the second effect, and both effects will be applied to the image instance in the cell. Alternatively, a new cell is created having the effects of both cells. Thus, cells can have an unlimited number of effects.
The user can select a cell (200, 202, 204, 206, 208, 210, 212, 214, and 216) to indicate selection of a desired effect. Then, by selecting the camera button 106, the user can capture a still image and have the selected effect applied to it. Alternatively, the user can select a previously captured image 218 in the image strip 104 to apply the selected effect to. By doing so, the selected effect is applied to image 218. The user can later change the effect applied to the image 218 by selecting a different cell.
In this illustration, newly captured image 508 is displayed along with the earlier captured image 218. A user can transfer any image in the image strip 104 to another application by selecting the image (e.g., image 508) and either selecting an application icon (500, 502, 504, 506) or dragging and dropping the image onto an application icon or onto another application outside of window 100. For example, selection of the application icon 500 will cause the image to be added to an electronic mail message. Selection of icons 502, 504 and 506 add the image to an email message, an account picture and an Internet chat user profile, respectively. The potential number of applications that a captured image can be transferred to is not limited to the four applications discussed here. Any application that can accept image data is suitable (e.g., iPhoto and iChat, available from Apple Computer, Inc.).
In one implementation, an image can be transferred to a target application by saving the image to a file and sending the target application a notification (e.g., a drag and drop GUI event) that refers to the image file. In another implementation, an image can be saved in file at a location where a target application expects to find it and, optionally, with a file name that the target application will recognize.
System 600 includes an image obtainer component 606 which receives input data (singularly or as a stream) containing compressed or uncompressed image data from an image source such as a digital image capture device 602 or persistent storage 604. The image capture device 602 (e.g., digital camera) can be capable of capturing still or live images, can be wired or wireless, and can be integrated into a computer, game system, portable game system, portable media player, cellular telephone, portable music player (e.g., an iPod from Apple Computer, Inc.), or other suitable device. In one implementation, the image capture device 602 is an Apple iSight Video Camera, available from Apple Computer, Inc. The iSight camera can be mounted on or located in proximity to a computer user's display device so that the user is in front of the camera, preferably centered.
The image obtainer 606 can prompt the camera to capture an image at the behest of a user or a process. If the image capture device 602 is only capable of capturing live images (e.g., where the input data is a continuous time-ordered sequence of video packets or frames containing image data), the image obtainer 606 can determine which frame in the sequence corresponds to the time when the user or the process requested that an image be captured. For example, this could be the frame that coincides in time with the display device “flash”. The image obtainer 606 can optionally receive information from the camera in addition to the input data such as a light measurement and provide this information to other components.
The image obtainer 606 can provide input data (singularly or as a stream) I0 to user interface 512, such as in the case where region 102 of window 100 is in preview mode. Input data I0 can also be provided to duplicator component 608 in which one or more copies of I0 (i.e., I1, I2, I3, I4 and I5) are each individually provided to a corresponding filter (i.e., F1, F2, F3, F4, and F5). Each filter operates on its respective input data by applying an effect to the input data to create modified input data (i.e., I′1, I′2, I′3, I′4, I′5). A filter can also apply a null effect, that is, the effect will not modify the input data. For example, the output of a null effect is seen in cell 208 in
Each modified input data (I′1, I′2, I′3, I′4, I′5) can be provided to a mosaic composer component 610. The mosaic composer 610 prepares the modified input data for display in the user interface 612 (e.g., region 102 of window 100). This can entail extracting image data containing an image from each modified input data, scaling each extracted image to fit in a cell, and updating each cell (200, 202, 204, 206, 208, 210, 212, 214 and 216) with its associated extracted image. Alternatively, the composer 610 can arrange each extracted image into a single mosaic which is used to update region 102 wholesale.
The user interface 612 is rendered on a display device 616 which can optionally serve as a photographic “flash” as described above. A prompt component 614 detects when a user has selected an effect. The prompt component 614 also detects when a user has requested (e.g., by selecting the camera button 106) that an image be captured and communicates this to the user interface 612. The user interface 612 can notify the image obtainer 606 so that the image can be captured (e.g., after a count-down timer has completed). The user interface can also notify a flash component 618 of the requested image capture so that the flash component 618 can “flash” the display device at a time coinciding with the image capture.
Input data containing image data is received singularly or as a stream (step 702). Instances of the input data are created (step 704). A filter is applied to each instance (step 706). Each instance is displayed using its associated filter (step 707) in a mosaic. A prompt is received indicating selection of an instance (step 708). A further prompt is received indicating that an image will be captured (step 712). The image is captured (step 712). The filter associated with the selected instance is applied to the captured image (step 714). The captured image is displayed with the applied filter (step 716).
A previously captured image having an associated filter is selected in the image strip 104 (step 802). Instances of the image are created without the associated filter (step 804). Filters are applied to each instance to create modified instances (step '806). Each modified instance is displayed in a unique cell in a mosaic (step 808). A prompt is received indicating selection of a modified instance (step 810). The filter that was applied to the selected modified instance is applied to the image (step 812).
The invention and all of the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Apparatus of the invention can be implemented in a computer program product tangibly embodied in a machine-readable storage device for execution by a programmable processor; and method steps of the invention can be performed by a programmable processor executing a program of instructions to perform functions of the invention by operating on input data and generating output.
The invention can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. Each computer program can be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language if desired; and in any case, the language can be a compiled or interpreted language.
Suitable processors include, by way of example, both general and special purpose microprocessors. Generally, a processor will receive instructions and data from a read-only memory and/or a random access memory. A computer can be any electronic computing device, such as for example and without limitation, a handheld or other portable device, cell phone, consumer electronic device, embedded device, or other suitable device. Generally, a computer will include one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; a magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM disks. Any of the foregoing can be supplemented by, or incorporated in, ASICs (application-specific integrated circuits).
To provide for interaction with a user, the invention can be implemented on a computer system having a display device such as a monitor or LCD screen for displaying information to the user and a keyboard and a pointing device such as a mouse or a trackball by which the user can provide input to the computer system. The computer system can be programmed to provide a graphical user interface through which computer programs interact with users.
A number of embodiment of the invention have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. Accordingly, other embodiments are within the scope of the following claims.
Number | Name | Date | Kind |
---|---|---|---|
4881127 | Isoguchi et al. | Nov 1989 | A |
5262815 | Aumiller | Nov 1993 | A |
5402513 | Schafer | Mar 1995 | A |
5459529 | Searby et al. | Oct 1995 | A |
5519828 | Rayner | May 1996 | A |
5534917 | MacDougall | Jul 1996 | A |
5670985 | Cappels, Sr. et al. | Sep 1997 | A |
5687306 | Blank | Nov 1997 | A |
5726672 | Hernandez et al. | Mar 1998 | A |
5737552 | Lavallee et al. | Apr 1998 | A |
5748775 | Tsuchikawa et al. | May 1998 | A |
5752029 | Wissner | May 1998 | A |
5778108 | Coleman, Jr. | Jul 1998 | A |
5781198 | Korn | Jul 1998 | A |
5812787 | Astle | Sep 1998 | A |
5854893 | Ludwig et al. | Dec 1998 | A |
5914748 | Parulski et al. | Jun 1999 | A |
5923791 | Hanna et al. | Jul 1999 | A |
5956026 | Ratakonda | Sep 1999 | A |
5995095 | Ratakonda | Nov 1999 | A |
6064438 | Miller | May 2000 | A |
6111562 | Downs et al. | Aug 2000 | A |
6148148 | Wain et al. | Nov 2000 | A |
6230172 | Purnaveja et al. | May 2001 | B1 |
6237025 | Ludwig et al. | May 2001 | B1 |
6298197 | Wain et al. | Oct 2001 | B1 |
6307550 | Chen et al. | Oct 2001 | B1 |
6344874 | Helms et al. | Feb 2002 | B1 |
6351762 | Ludwig et al. | Feb 2002 | B1 |
6400374 | Lanier | Jun 2002 | B2 |
6414707 | Agraharam et al. | Jul 2002 | B1 |
6434266 | Kanno et al. | Aug 2002 | B1 |
6507286 | Weindorf et al. | Jan 2003 | B2 |
6583806 | Ludwig et al. | Jun 2003 | B2 |
6594688 | Ludwig et al. | Jul 2003 | B2 |
6624828 | Dresevic et al. | Sep 2003 | B1 |
6714216 | Abe | Mar 2004 | B2 |
6715003 | Safai | Mar 2004 | B1 |
6716103 | Eck et al. | Apr 2004 | B1 |
6760042 | Zetts | Jul 2004 | B2 |
6912327 | Hori et al. | Jun 2005 | B1 |
6933979 | Gonzales et al. | Aug 2005 | B2 |
6954894 | Balnaves et al. | Oct 2005 | B1 |
6988244 | Honda et al. | Jan 2006 | B1 |
6988248 | Tang et al. | Jan 2006 | B1 |
7030872 | Tazaki | Apr 2006 | B2 |
7064492 | Vijeh | Jun 2006 | B1 |
7095392 | Lin | Aug 2006 | B2 |
7095395 | Bu | Aug 2006 | B2 |
7103840 | Ihara et al. | Sep 2006 | B2 |
7124366 | Foreman et al. | Oct 2006 | B2 |
7127149 | Lin | Oct 2006 | B1 |
7152093 | Ludwig et al. | Dec 2006 | B2 |
7165219 | Peters et al. | Jan 2007 | B1 |
7206809 | Ludwig et al. | Apr 2007 | B2 |
7421470 | Ludwig et al. | Sep 2008 | B2 |
7433921 | Ludwig et al. | Oct 2008 | B2 |
7546544 | Weber et al. | Jun 2009 | B1 |
7739599 | Patten et al. | Jun 2010 | B2 |
20020019833 | Hanamoto | Feb 2002 | A1 |
20020105589 | Brandenberger et al. | Aug 2002 | A1 |
20020140740 | Chen | Oct 2002 | A1 |
20020167540 | Dobbelaar | Nov 2002 | A1 |
20020180803 | Kaplan et al. | Dec 2002 | A1 |
20020194195 | Fenton et al. | Dec 2002 | A1 |
20020198909 | Huynh et al. | Dec 2002 | A1 |
20030046348 | Pinto et al. | Mar 2003 | A1 |
20030047099 | Hanyu | Mar 2003 | A1 |
20030067435 | Liu | Apr 2003 | A1 |
20030090507 | Randall et al. | May 2003 | A1 |
20030189588 | Girgensohn et al. | Oct 2003 | A1 |
20030193520 | Oetzel | Oct 2003 | A1 |
20030210261 | Wu et al. | Nov 2003 | A1 |
20040017390 | Knowlton et al. | Jan 2004 | A1 |
20040021684 | Miller | Feb 2004 | A1 |
20040032497 | Ying et al. | Feb 2004 | A1 |
20040085340 | Dimitrova et al. | May 2004 | A1 |
20040125124 | Kim et al. | Jul 2004 | A1 |
20040196250 | Mehrotra et al. | Oct 2004 | A1 |
20040227751 | Anders | Nov 2004 | A1 |
20040233125 | Tanghe et al. | Nov 2004 | A1 |
20040239799 | Suzuki et al. | Dec 2004 | A1 |
20050073575 | Thacher et al. | Apr 2005 | A1 |
20050117811 | Hoshino | Jun 2005 | A1 |
20050174589 | Tokiwa | Aug 2005 | A1 |
20050194447 | He et al. | Sep 2005 | A1 |
20060018653 | Kido | Jan 2006 | A1 |
20060050151 | Fujinawa | Mar 2006 | A1 |
20060055828 | Henry et al. | Mar 2006 | A1 |
20060274161 | Ing et al. | Dec 2006 | A1 |
20060284895 | Marcu et al. | Dec 2006 | A1 |
20070009028 | Lee et al. | Jan 2007 | A1 |
20070081094 | Ciudad et al. | Apr 2007 | A1 |
20080303949 | Ciudad et al. | Dec 2008 | A1 |
20080307307 | Ciudad et al. | Dec 2008 | A1 |
20100118179 | Ciudad et al. | May 2010 | A1 |
Number | Date | Country |
---|---|---|
1217598 | Jun 2002 | EP |
1 775 939 | Apr 2007 | EP |
2002335432 | Nov 2002 | JP |
2004180245 | Jun 2004 | JP |
2004187154 | Jul 2004 | JP |
2004260560 | Sep 2004 | JP |
2004286781 | Oct 2004 | JP |
2004-350208 | Dec 2004 | JP |
2004349843 | Dec 2004 | JP |
2005241814 | Sep 2005 | JP |
2007-110717 | Apr 2007 | JP |
WO2004054239 | Jun 2004 | WO |
Number | Date | Country | |
---|---|---|---|
20070081740 A1 | Apr 2007 | US |