Systems and Methods for Generating an Interactive Mosaic Comprising User Visual Content Data on a Portable Terminal and an Image Sharing Platform

Abstract

There is disclosed an image processing method for generating a mosaic of images. The method includes selecting a cover image from a plurality of images and dividing the cover image into a number of grid blocks. For each grid block, the average color (or other parameter) is calculated and each grid block is assigned a sub-content based on the color matching. Mosaic generation information is transmitted to a server, the mosaic generation information including a list of sub-content to be displayed in the mosaic, each sub-content including corresponding grid block location information. A server verifies whether each of the sub-content exists on the server and transmits sub-content not present on the server, to the server. The mosaic is displayed on a display of a portable terminal the sub-content using the mosaic generation information so that the cover image is reproduced to form of a mosaic of sub-content that represents the cover image and each sub-content includes additional information specific to each sub-content.

Description

The present invention, and various embodiments thereof, relates to image creation, production, manipulation, editing and sharing, and more particularly, but not exclusively, to systems and methods of selectively adjusting visual content data on a user device. Also provided is a plurality of visual content data editing functions and methods thereof.

Also provided is a visual data sharing platform, which allows a user to share an image or other content data with others amongst and within a network of other users.

The present invention allows sharing, visualizing, and collaborating on visual content. One embodiment of the invention comprises a social network configured for the creation, sharing, and viewing of user-generated visual content data.

In one embodiment, a cover image is selected from a database of images on a user device, and is converted to a mosaic image made up of a plurality of sub-images selected from the database of images, which provide a best-fit match in accordance with predetermined criteria wherein the resulting mosaic image approximates the cover image.

Also provided is a method for generating a mosaic image with an appearance that approximates a cover image by utilizing a plurality of content selected from a database on a user device, comprising the steps of: selecting a cover image from the database of images on the user device; dividing the cover image into a (x, y) grid pattern; comparing the selected content to the regions of the grids comprising the cover image in order to approximate visual similarity based on color matching between the selected content and x,y coordinates of the cover image grid, and thereby selecting the image with the best color match to represent a portion of the grid; positioning the selected images in a targeted location of the grid, thereby creating a mosaic image that approximates the cover image; and sharing the resulting mosaic image amongst a network of users.

Also disclosed is a computer implemented method for generating a mosaic image with an appearance that approximates a cover image by utilizing a plurality of content selected from a database on a user device, comprising the steps of: selecting a cover image from the database of images on the user device; dividing the cover image into a (x, y) grid pattern; comparing the selected content to the regions of the grids comprising the cover image in order to approximate visual similarity based on color matching between the selected content and x,y coordinates of the cover image grid, and thereby selecting the image with the best color match to represent a portion of the grid; positioning the selected images in a targeted location of the grid, thereby creating a mosaic image that approximates the cover image; and sharing the resulting mosaic image amongst a network of users.

Also disclosed is a computer program product having a computer-readable storage medium storing computer-executable code for generating a mosaic image with an appearance that approximates a cover image by utilizing a plurality of content selected from a database of content, and sharing the resulting mosaic image amongst a network of users.

Also provided is a computer-readable storage medium storing an executable computer program application for manipulating, creating and editing visual data content.

The system, methods and application disclosed herein are novel for the way the application and social platform visually organizes content on a mobile device as well as the way users of the system are able to link content, and collaborate on content.

DEFINITIONS

The following terms are used in this disclosure to describe different aspects of the invention and the environment in which the invention operates. These terms are used for explanation purposes only and are not intended to limit the scope for any aspect of the invention.

Content as used herein refers to one or more of the following: a digital image, photo, video, video stream, animation, song, music clip, mp3 file, link to other source and/or content, link to social network feed (such as Facebook, Twitter). Sub-content refers to a plurality of content arranged to create a combined content, such as the mosaic image. The term content may at times be used interchangeably with the term fraxel, which describes content selected to be used for producing, thereby becoming part of, the mosaic image.

Cover image refers to a first unaltered image selected as a template for the mosaic image.

Mosaic image refers to an altered image produced by arranging together various content, sub-content in a manner described in more detail herein, wherein in the mosaic image produced resembles an unaltered cover image. The term mosaic image may at times be used interchangeably with the term fractograf, which describes a mosaic image created and displayed via a mobile application on a user device.

SHA refers to one of several cryptographic hash functions, most often used to verify that a file has been unaltered. SHA is short for Secure Hash Algorithm. File verification using SHA is accomplished by comparing the check sums created after running the algorithm on the two files you want to compare.

OVERVIEW OF VARIOUS EMBODIMENTS OF THE INVENTION

In one embodiment, various visual content are combined to form a larger image (photo, video, with or without sound, or other content, and including combinations thereof); the result is a mosaic or collage built from user-selected content. This organization has significant advantages over other organization tools because of the ability for a user to see many images at once and easily scan to find items of interest.

In one particular embodiment, individual content used in the present invention is compressed to be 512×512 pixels; however, other dimensions are contemplated.

In one embodiment, to produce a mosaic image from a cover image, a cover image is first selected and then the cover image is sectioned, or partitioned into a grid. In one embodiment, the standard grid is 80×60, however other dimensions are contemplated. (For example: If W≧H, 80×(80*h/w); if W≦H, (80*w/h)×80. In this example, k=80, however, other values are envisioned.) Dimensions and grid values are, in part, dependent the aspect ratio of the cover image.

For each grid section, the average color is calculated. Each of the 4800 grid sections is then matched amongst a plurality of content that a user has identified to make up the mosaic image. For example, an algorithm compares two images/content and determines good matches; if more than one image/content can both serve as a best match, the image/content used with the less frequency will be placed into the mosaic image to ensure more variety of content in the resulting image. The resulting mosaic image is comprised of all the images selected by the user, however the mosaic image can also include content selected independently or randomly by way of the matching algorithm or by selection based on other criteria such as user targeting, product placement, etc. The user may add captions, links, and other details to each mosaic image created. The user also adds a title, description, and other details to the mosaic image and then posts the mosaic image via a server to a social network platform so that other users may view it. In the case where the image is moving, the mosaic image is generated when the user stops the video (and the image becomes still). As the image becomes still, this case is reduced to the still image case described previously.

In another embodiment, the system is configured with collaborative editing capabilities.

A user can be granted permission to edit a mosaic image created by another user. To edit, the mosaic image is retrieved from the server. The editing approach resembles the mosaic image generation approach with a pre-existing mosaic image (cover image, sub-content, and details about the image and/or content) to which new content is added by the user.

The system is configured with additional features such as visual recognition of a mosaic image, including within another mosaic image, by means of a beacon. For example, the beacon appears as a faint image emanating from the sub-content to which an additional mosaic image is linked. Additionally, other effects, such as a twinkle or flash can indicate a particular user's edited content in collaboration mode.

While mosaic images are viewed and created on a user device such as a tablet or smartphone, alternative viewing methods are anticipated, such as we browser viewing for those who can't view images through the app. The web viewer will have the standard mosaic viewing. An interactive in-app tour mode is anticipated in which mosaic images will be shown in a sequence, with the zoom moving in and out on each sub-content in the mosaic image. Alternatively, an in-app flip mode view is also anticipated, in which mosaic images will be shown in a sequence one-by-one, thereby allowing a user to flip through images via a touch screen.

Linking Content:

The system is configured for linking content in mosaic images to other content, including other mosaic images. It is envisioned that content can be one or more of an image, an audio file, a video clip or stream, or a combination thereof. Additionally, users can add content into other mosaic images. For example, suppose a user creates Mosaic Image A and adds Mosaic Image B into it by selecting Mosaic Image B as the sub-content. Subsequently, a viewer of Mosaic Image A can visually navigate around the mosaic image to find the cover image of Mosaic Image B. Through this cover image, the viewer may enter Mosaic Image B so long as they have permission. Additionally, Mosaic Image B may contain a Mosaic Image C, which the viewer may also enter if granted permission. This linking method is novel and has the benefit that the user may navigate to other content without having to leave their current content or look for a menu. Existing content viewing and sharing platforms deliver content in a manner than can lead to the user needing to ‘go back’ or go to a menu as they reach the end of a collection of content and want to continue viewing. Additionally, because of the visual nature of the linking, the links are more interesting than traditional menus or content albums/collections.

Collaborating on Content:

The system is configured for collaborating on content. Mosaic images are collections of content (up to 1024 individual sub-content). The system makes it possible for users to place content in Mosaic Images they do not own so long as they are given permission by the owner. This collaboration is novel because of the opportunity users are given to co-create Mosaic Images and shape their content. A user's image becomes not just part of a collection of photos, but a part of a Mosaic; the images are interspersed and grouped to create a visually appealing arrangement mosaic. Current content sharing platforms offer the opportunity for users to create ‘shared albums’ or ‘folders.’ These platforms take users photos and place them together a list and show the photos one-by-one. This results in the content being together, but not together in a way that is unique or novel. The existing content sharing platforms methods for sharing photos is very similar to a folder on a computer showing thumbnails of images.

Exemplary Process for Creating a Mosaic Image (or “Fractograf”)

The present invention requires servers, mobile phones and their operating systems, a website for sharing Mosaic Images with people on mobile devices, a software program of instructions for carrying out the method described herein, and access to the Internet to connect the components.

In one embodiment, the invention is carried out by a mobile application configured to carry out the following steps:

• • Identify a cover photo
  • Identify one or more of a sub-image (sub-content)
  • For each sub-image, measure the average color
  • Match each sub-image with each grid cell based on color and other parameters to ensure all sub-images are used at least 4 times
  • When user clicks save/post, verify with a server using SHA and determine if the sub-images already exist of the server
  • Send sub-images/content to the server if not already deposited
  • Send a list of images that make up the mosaic image (fractograf) to the server
  • Save other details about mosaic image (fractograf) including title, description, permissions, beacon etc.Once the User has Created a New Mosaic Image, or Fractograf, and Indicated which Sub-Images and Fractografs it Will Contain, the Following Steps Occur:
  • SHAs for each image are calculated using the original image's bytes and the user's id. This allows the same image to be uploaded and owned by different users.
  • The SHAs are sent to the server. The server will respond Image Ids for all SHAs that it already has stored. This prevents the image from being uploaded multiple time by the same user.
  • All Images that the server does not know about are bulk uploaded to the server. The server responds with new Ids for each of them.
  • The Mosaic image/Fractograf is posted to the server. The Server will respond with a new Id.
  • All the Sub-content/fraxel Ids are then put into a command message and uploaded to the server.
  • Sub-content/fraxels can be assigned various values, such as content type, id, color, and cover image reference, image source (such as User id).

Example

Image Modification

• • As the user ads and removes sub-content/fraxels from an editable Mosaic Image/Fractograf, the client application creates a command list of additions and removals.
  • When a post is triggered by the user, any new images are uploaded as indicated in the Mosaic image/Fractograf Creation process.
  • The command list is then sent to the server. This command list is applied against the current fraxel list that the server holds. This will merge multiple user's commands into the current Fractograf. The oldest fraxels will be removed when the current number reaches 1001. This is giving each Fractograf 1000 user fraxels. (Note, this number may be increased, and is used for illustrative purposes).
  • The server will then respond with the Fractograf state so the client can display it.

Example

Mosaic Image/Fractograf Delivery

• • When a Fractograf is requested from the server, it will assemble the metadata (name, description, owner, etc) and the user supplied Fraxels.
  • The server will also create an optional extended fraxels list. This list will contain fraxels that will be included in the Fractograf separate and distinct from user-initiated selection, such as for product placement or advertising. These might include Advertisements, Interesting Fractografs, Featured Fractografs, etc. Information about the user, and advertiser preferences, or pre-selected advertisement content, and the Fractograf will be used to determine which extended fraxels are used.
  • The client will calculate the fraxel positions for both the user generated and server generated fraxels using a stable algorithm.

Example

Sub-Content/Fraxel Layout

• • The Cover image is divided into an W×H grid, where currently the max value of W or H is 80, although this number is illustrative and other dimensions are envisioned.
  • A Random function is used, seeded with the same value each time, so multiple reruns will produce the same random sequence.
  • The order that each fraxel position (x, y) is visited is randomize to avoid dithering patterns and prevent “forced fraxels” additions to be scattered. About N positions will be preselected for use by the extended fraxels.
  • The extends fraxels are placed in the first N positions.
  • For each fraxel position, a list of fraxels that have good color matches are found. A fraxel is picked based off of the number of times it has been used already and if it's been used nearby the current (x,y)
  • If the number of positions left is equal to the number of fraxels used less than four times, the algorithm will continue to fill the remaining positions with the fraxels used less than 4 times.
  • It is envisioned that additional embodiments will support explicit layout and region based layout that will allow given fraxels to exist only in given regions of the Fractograf.

FIGURES/CHARTS/DRAWINGS

Also included as part of the disclosure of the present invention are drawings, photos and flowcharts that illustrate additional embodiments of the present invention, including but not limited to, aspects of mosaic image creation, editing, sharing and collaboration of content on a social networking platform, as well as embodiments related to targeted content placement for advertisement within a mosaic image. Screen shots are also included which show functionality of the system on a user device, such as a mobile phone. The drawings, photos and flowcharts are illustrative of particular examples and embodiments of particular examples for enabling methods of the present invention, and are not intended to limit the scope of the invention. The drawings are not to scale (unless so stated) and are intended for use in conjunction with the explanations in the written disclosure.

Claims

1. An image processing method for generating a mosaic of images, the method comprising the steps of: selecting a cover image from a plurality of images;dividing the cover image into a predetermined number of grid blocks;calculating a value of each of the grid blocks and storing the value and the corresponding grid block coordinate information that indicates the relative positions of the grid blocks relative to each other within the cover image;selecting, for each grid block, sub-content from a library of sub-content based on an accepted correspondence between the average value of the grid block and an average value of the sub-content, and assigning the sub-content to the grid block;verifying with a server to determine whether each sub-content exists on the server and transmitting sub-content not present on the server, to the server;transmitting mosaic generation information to the server, the mosaic generation information including a list of sub-content to be displayed in the mosaic, each sub-content including corresponding matching value;displaying on a display of a portable terminal the sub-content using the mosaic generation information so that the cover image is reproduced to form of a mosaic of sub-content that represents the cover image;wherein each sub-content includes additional information specific to each sub-content.

2. A method according to claim 1, further comprising: compressing each sub-content by a first compression factor and storing a medium resolution sub-content on the server and/or the portable terminal; andcompressing each medium resolution sub-content by a second compression factor and storing the low resolution sub-content on the server and/or the portable terminal.

3. A method according to claim 2, wherein the mosaic of sub-content is a mosaic of low resolution sub-content.

4. A method according to claim 3, wherein the medium resolution sub-content is transmitted from the server to the portable terminal when a display focal point is hovered over the low resolution sub-content within the mosaic.

5. A method according to claim 1, further comprising predetermined sub-content wherein each sub-content is assigned to a predetermined number or set of grid blocks.

6. A method according to claim 1, wherein the additional information includes image access information and/or image recognition information.

7. A method according to claim 1, wherein the mosaic of sub-content includes collaborative editing information allowing the mosaic of sub-content to be downloaded from the server to another portable terminal for display on the portable terminal, the mosaic of sub-content being editable by adding and/or removing sub-content to the mosaic of sub-content.

8. A method according to claim 1, wherein the library is a user defined library, and optionally wherein the library is a third party defined library.

9. A method according to claim 1, wherein the sub-content is one or more selected from an image, another mosaic, or video file.

10. A method according to claim 9, wherein the image is a single frame of a video, the frame being selected by a user, wherein the frame is selected from the video by pausing the video, or the image is representative of content such as an audio file.

11. A method according to claim 1, further comprising the steps of: synchronizing and reproducing at least one mosaic of sub-content by a first portable terminal in real time with a second portable terminal by transmitting the mosaic of sub-content by a first portable terminal to the server, the server transmitting the mosaic of sub-content to the second portable terminal;receiving a change to the at least one mosaic of sub-content from the first portable terminal and transmitting the sub-content changes to the server; andtransmitting by the server the change to the at least one mosaic of sub-content to the second portable terminal for display by the second portable terminal.

12. A method according to claim 11, further including identifying whether a sub-content is stored on the server, and if the sub-content is not stored on the server, transmitting the sub-content to the server and storing the sub-content on the server.

13. A method according to claim 11 or 12, further comprising: compressing each sub-content by a first compression factor and saving the medium resolution sub-content on the server and/or the portable terminal; andcompressing each medium resolution sub-content by a second compression factor and saving the low resolution sub-content on the server and/or the portable terminal.

14. A method according to claim 1, further comprising the steps of: accessing a first mosaic of sub-content from the server and downloading the first mosaic of sub-content onto a first portable terminal for displaying on a display of the first portable terminal;identifying a link between the first mosaic of sub-content and a second mosaic of sub-content stored on the server using predetermined identification criteria, wherein the link is a visual indicator that is evident to a user and signals an entry point into another mosaic of sub-content when the link is selected by the user;accessing the second mosaic of sub-content from the server and generating and transmitting a sub-content representing the second mosaic of sub-content onto a first portable terminal for displaying on a display of the first portable terminal;embedding within the first mosaic of sub-content, a second cover image, the second cover image representing the second mosaic of sub-content so that the first mosaic of sub-content and a second mosaic of sub-content are linked.

15. A portable device comprising: a display;a communication unit that receives first mosaic generation information to generate a first mosaic of sub-content, wherein each sub-content includes additional information linked to the sub-content; anda controller that controls the display to display the first mosaic of sub-content on the display, the controller receives an input modifying the first mosaic of sub-content displayed on the display, and the controller generates second mosaic generation information to generate a second mosaic of sub-content based on the input, and controls the communication unit to transmit the second mosaic generation information to a server to store the second mosaic generation information.

16. A non-transitory computer-readable medium having recorded thereon a program that causes a portable device to execute a method for generating a mosaic of images, the method comprising: selecting a cover image from a plurality of images;dividing the cover image into a predetermined number of grid blocks;calculating an average color value of each of the grid blocks and storing the average color value and the corresponding grid block coordinate information that indicates the relative positions of the grid blocks relative to each other within the cover image;selecting, for each grid block, sub-content from a library of sub-content based on an accepted correspondence between the average color value of the grid block and an average color value of the sub-content, and assigning the sub-content to the grid block;verifying with a server to determine whether each sub-content exists on the server and transmitting sub-content not present on the server to the server;transmitting mosaic generation information to the server, the mosaic generation information including a list of sub-content to be displayed in the mosaic, each sub-content including corresponding grid block location information;displaying on a display of a portable terminal the sub-content using the mosaic generation information so that the cover image is reproduced to form of a mosaic of sub-content that represents the cover image;wherein each sub-content includes additional information specific to each sub-content.