Patent Application
20150365454
Embodiments of the present subject matter generally relate to the field of network communication, and, more particularly, to access nodes that provides network access for one or more client devices.
In the field of network communication, an access node may be any device that provides access for one or more client devices to communicate via an upstream network. An example of an access node may include a wireless local area network (WLAN) access point (AP). Other examples of an access node may include a cellular base station, a relay station, a home base station (such as a femtocell, picocell, or other small cell appliance), or local area network switch.
One or more client devices may communicate via the access node to a server located in the upstream network. Examples of client devices may include mobile devices, cameras, accessories, tablets, laptops, personal computers, gaming devices, and the like. Client devices may be configured to upload content via an access node to a server in the upstream network. For example, media content, such as photos, videos, audio files, and the like, may be transmitted to the server. The client device may upload media content to a photo sharing website or social network website.
A backhaul network connection from the access node to the upstream network may become saturated or congested during synchronization of media content. Additionally, when more than one client device attempts to utilize an access node concurrently, the client devices may compete for use of the available bandwidth of the backhaul network connection. Synchronization of media content is typically not coordinated among client devices. The size and quantity of content items being uploaded to the server via the access node may impact the upload experience due to congestion of the backhaul network connection.
Various embodiments are described in which media processing services are performed at an access node. In one embodiment, the access node may receive media content from one or more client devices. The media content may include a plurality of content items. The access node may process at least a first portion of the media content to produce processed media content. The media processing is performed at the access node based at least in part on characteristics of the plurality of content items. After processing at least the first portion of the media content to produce processed media content, the access node may transmit the processed media content to a server.
The present embodiments may be better understood, and numerous objects, features, and advantages made apparent to those skilled in the art by referencing the accompanying drawings.
The description that follows includes exemplary systems, methods, techniques, instruction sequences and computer program products that embody techniques of the present subject matter. However, it is understood that the described embodiments may be practiced without these specific details. For instance, although examples may refer to audio, video, or image content items, other types of media content may be processed at an access node. In this disclosure, an access node is any device that provides a wireless or wired connection for the client device to access other networks or servers. The example embodiments described relate to a wireless access node in which the client device(s) are wireless coupled to the access node. In other instances, well-known instruction instances, protocols, structures and techniques have not been shown in detail in order not to obfuscate the description.
Increasingly, client devices provide capability to capture media content. For example, mobile phones may have integrated cameras and microphones. Digital cameras have also been integrated into wearable accessories, such as watches, eyeglasses, or pocket-sized digital video cameras. A wide variety of client devices may be equipped with various media capture capabilities. These client devices may provide an end user with the ability to capture media content, such as audio, video, or image content. The media content may be uploaded from the client device to a server via an access node. The media content may be uploaded from the client device to a server, such as a photo sharing website, social networking website, cloud storage server, and the like.
In accordance with this disclosure, an access node may enhance the upload experience by performing media processing services at the access node. Performing the media processing services at the access node may reduce the bandwidth utilization of a backhaul network connection between the access node and the server. Furthermore, the media processing services may improve the quality and usefulness of the media content that gets uploaded to the server. The access node may have additional processor capability and memory available to perform media processing services that may otherwise be unavailable or limited at the client device. Additionally, the access node may perform the media processing services for a plurality of content items from one or more client devices.
Several examples of media processing services are described in this disclosure. Examples of media processing services may include an image or audio enhancement process, filtering of low quality content items, removal of duplicate content items, categorizing or ranking the content items, filtering based on facial recognition, or other media processing services. The media processing services may also include the generation of a compilation content item, such as a panoramic image or video representation of several photos.
In one embodiment, the access node 110 may comprise a mobile device. For example, mobile device may operate as an access node for another mobile device operating as a client device. In one example, the access node 110 may comprise a mobile device that provides network access for a plurality of accessories associated with the mobile device. The plurality of accessories may utilize BLUETOOTH™, or other short-range radio frequency connection, to communicate via the mobile device to an upstream network. For simplicity, in the example system 100 described in
The access node 110 may utilize a backhaul network connection 142 and one or more networks 135 to communicate with a server 130. In a typical communication from the client device 101 to the server 130, the client device 101 may transmit packets having a destination address associated with the server 130. The access node 110 may receive the packets and forward them based on the destination address and network routing. The network route in
During a conventional upload session, the client device 101 may synchronize media content with the server 130 by transmitting media content and waiting for acknowledgement. If other client device(s) 102 are concurrently utilizing the access node 110, the upload session may take longer and cause more power consumption. For example, during family or group events, multiple client devices may utilize an access node at the same time. Each client device may be capturing media content and uploading content via the access node 110. In a conventional mobile device, a photo sharing application may be configured to upload all unsynchronized media content. However, some content items may be more or less desirable to share than others. Uploading an undesirable low quality image may cause delay for uploading the more desirable high quality image. To improve the upload experience and to enhance the quality and desirability of the uploaded images, the access node 110 may provide media processing service(s) 120.
Several example media processing service(s) 120 are envisioned in this disclosure. In various embodiments, one or more different media processing services may be combined. Although many media processing service(s) 120 are described in
The access node 110 may receive the media content from the client device 101 (and from other client device(s) 102). The media content may include a plurality of content items, such as photographs, videos, sound files, and the like. In some embodiments, the content items may be audio/visual multimedia content items. In one embodiment, the media content may be uploaded from the client device 101 to the access node 110, such that the client device 101 is aware that the access node 110 will process the media content. In another embodiment, the media content is included in messages directed from the client device 101 to the server 130. The access node 110 may intercept the messages and process the media content on behalf of the client device 101 with or without the client device 101 being aware that the access node 110 is intercepting the messages. The access node 110 may identify messages that include media content based on a destination address, website uniform resource location (URL) address, or other field of the messages. Alternatively, or additionally, the access node 110 may perform packet inspection to identify messages that include media content.
One example of the media processing service(s) 120 may include a content processing module 121. The content processing module 121 may enhance or transform content items prior to uploading the content items. For example, the content items may be enhanced by a media enhancement process or compressed to reduce upload size. The media enhancement process may transform the content item. For example, a photograph may be modified by a media enhancement process to adjust color tone, correct for red eye, adjust contrast, etc. In another example, photographs may be modified in accordance with a color scheme or theme (such as sepia, grayscale, or the like). The content processing module 121 may process content items from multiple client devices so that an uploaded collection of content items from multiple client devices have a similar quality.
The content processing module 121 may also compress or otherwise reduce the size of the content item automatically. The amount of compression may depend on the quantity of client devices coupled to the access node 110. Alternatively, the amount of compression may depend on the quantity of content items being uploaded via the backhaul network connection 142 concurrently.
Another example of a media processing service might include a rank module 122 configured to rank content items based on quality. The rank module 122 may determine an order for uploading content items. The rank module 122 may work with the filter module 123 to determine which content items should be uploaded and which content items should be discarded. Removing low quality content items may preserve the backhaul network connection 142 for higher quality content items. For example, a low quality image may be blurry or lacking in recognizable facial features, while a high quality image may be sharp with recognizable faces.
The rank module 122 may determine a quality metric for each of a plurality of content items. Content items having a higher quality may be uploaded to the server on behalf of the client device, while content items having a lower quality may be discarded. For photo or video content items, the quality metric may be based on sharpness, blurriness, contrast, brightness, resolution, color accuracy, distortion, vignetting, dynamic range, lens flare, moiré patterns, or other characteristics of the content item. For audio content items, the quality metric may be based on voice recognition, noise, clipping, speech clarity, or the like.
A threshold may be used by the rank module 122 to determine which quality metric values represent high quality content items and which quality metric values represent low quality items. The threshold may be system-defined, user-defined, or dynamically adjusting. For example, the threshold may dynamically change in relation to the quantity of content items and/or the current bandwidth availability of the backhaul network connection 142. When the backhaul network connection 142 is congested or saturated, the threshold may be increased such that the uploaded content items are of the highest quality. When the backhaul network connection 142 is not congested or saturated, the threshold may be decreased such that more content items may be uploaded.
A filter module 123 may be included in the media processing service(s) 120. In one example, the filter module 123 may filter low quality content items based on the quality metric and/or ranking provided by the rank module 122. The filter module 123 may discard content items below the quality threshold.
In another example, the filter module 123 may discard redundant content items. For example, the filter module 123 may be configured to identify two content items that have redundant content. The redundant content item may be identical, duplicate, or overlapping to another content item. Consider two photographs taken at the same scene by two different client devices. Each client device may attempt to upload a photograph that is redundant to the other. The filter module 123 may identify that the scene matches and discard one of redundant content items. Metadata in the content items may help the filter module 123 to identify redundant content items. For example, the filter module 123 may determine that two content items have common or close timestamp. The filter module 123 may compare the content items with the common or close timestamp to determine if they are directed at the same scene or have the faces of the same group of people. The filter module 123 may discard the lower quality redundant content item. Alternatively, the filter module 123 may keep the first received redundant content item and discard the later received redundant content item.
Another example of a filter may include an event-based filter. For example, the filter module 123 may utilize metadata, such as timestamp or position information, to select or deselect content items for an event. In an embodiment, such as where the media processing service(s) 120 includes a compilation module 124, the event-based filter may be used to discard content items that are not related in time and/or location.
Other examples of filters may be implemented. For example, a filter may be configured to remove photographs that do not have a person's face. Alternatively, a filter may be configured to keep photographs that have a scenic view (such as a skyline or outdoor view) and discard other photographs. Filters may be used to keep or discard content items based on lighting.
Filters may be used to identify persons. For example, the filter module 123 may determine facial recognition patterns among the content items. Determining facial recognition patterns may include counting a first quantity of content items that include a first face of a first person and counting a second quantity of content items that include a second face of a second person. The filter module 123 may discard a content item if the first quantity of content items is above a face count limit or is above a threshold percentage relative to a total quantity of the plurality of content items.
A compilation module 124 may be included in the media processing service(s) 120. The compilation module 124 may be used to collect content items into an album or collection. For example, the compilation module 124 may create a photo album with multiple photographs from several client devices. In one embodiment, the compilation module 124 may create a panoramic view from several image content items. In another embodiment, the compilation module 124 may create a video representation of several image content items. The compilation module 124 may generate a compilation content item that can be uploaded in addition to or in lieu of the multiple content items.
An upload/share module 125 may enhance the uploading and sharing of content items from one or more client devices. For example, the upload/share module 125 may group content items from different client devices to create a more complete media content album. The upload/share module 125 may be configured to upload content items to a photo sharing website or social networking website on behalf of an account owner for the photo sharing website or social networking website. In one embodiment, the upload/share module 125 may be configured to upload content items to multiple photo sharing or social networking websites using a plurality of accounts. The upload/share module 125 may “post” the content items on behalf of one or more users who provide permissions for the upload/share module 125 to post on their behalf. For example, users of a social networking website may grant permission to an “application” that has permission to post on their behalf. The upload/share module 125 may transmit the content items to a server associated with the application, such that the application can then post the content items to multiple accounts.
The upload/share module 125 may utilize facial detection to automatically tag or link persons that are included in the content item(s). The upload/share module 125 may also determine which client device(s) or server(s) should receive the processed content items.
Many other media processing services 120 may be implemented. Several thresholds or configurations may be used with the media processing service(s) 120. For example, information about a user data plan, upstream bandwidth, or cloud storage size may be used to modify a threshold that impacts the quantity of content items that are uploaded. User upload history or user preferences may also be used to dynamically adjust the thresholds.
At first time instant 321, the first client device 301 may begin uploading first media content 331. For example, a synchronization process at the first client device 301 may begin to transmit content items to the access node 310. The content items may be included in messages directed to a server (not shown) accessible by the access node 310. In one embodiment, the access node 310 intercepts the messages and begins processing 340 the media content at the access node. At least a first portion of the media content may be processed to produce processed media content. At some point after processing 340 has begun, the access node 310 may upload or share the processed media content 350. Uploading of a first portion of processed media content may occur while the access node continues processing 340 a second portion of the media content.
At second time instant 322, the second client device 302 may begin uploading second media content 332. Similarly, at third time instant 323, the third client device 303 may begin uploading third media content 333. It is noted that the various client devices may not begin uploading media content at the same time. The access node 310 may be configured to delay the upload of processed media content for a period of time to determine if multiple client devices are uploading related media content. Alternatively, the access node 310 may automatically adjust the media processing service(s) when media content is received from an additional client device.
In one embodiment, processing 340 of the media content may continue even after a last portion of the third media content 333 has been received. The third client device 303 may discontinue transmission and thus save battery resources, while the access node 310 continues to process the media content on behalf of the third client device 303. Similarly, the uploading and sharing of processed media content 350 may continue as media content is processed and as the backhaul network connection allows. This may be useful, for example, when the multiple client devices upload separate content items and the access node 310 prepares a compilation content item to upload and share on behalf of an end user.
In the example of
In
The access node 410 may discard the redundant item 452, the filtered item 451, and low quality items 461, 462. Therefore, the backhaul network connection may be preserved for more quickly upload the high quality items 441, 442, 443, 444 to the server 470.
In one embodiment, the access node 410 may also utilize the quality metric to determine an order for uploading the content item. For example, the high quality item 441 may have the highest quality metric and may be uploaded first to the server 470. As such, the first content item 471 may be uploaded first and may be the “best” content item from among the content items 431, 432, 433 received by the access node 410 from client devices 401, 402, 403. Subsequently, the second content item 472, third content item 473, and fourth content item 474 may be uploaded in order based on the quality metric determined at the access node 410.
In
The access node 510 may upload the compilation content item 530 to a first server 550. In one embodiment, the access node 510 may be configured to share the uploaded compilation content item 552. For example, the access node 510 may tag, share, link, or otherwise, associate the uploaded compilation content item 552 to end users associated with the first client device 401, second client device 402, and third client device 403. In another example, the access node 510 may utilize facial recognition to identify people included in the compilation content item 530 and share the uploaded compilation content item 552 with those people.
The access node 510 may also upload the compilation content item 530 to a second server 560. For example, the access node 510 may cause shared compilation content item 562 to be stored on a social network website, photo sharing website, or cloud server.
In the basic filter 610, the access node may also process content items to improve quality 614. For example, the basic filter 610 may include a contrast, color, tone, brightness, redeye, or other automatic enhancement process that can be applied to content items.
In the quality-based filter 620, the access node may determine a quality metric 622 for each content item. The quality-based filter 620 may discard content items having a quality metric below a quality threshold 624. As described previously, the quality threshold may be user-defined, system-defined, or dynamically determined.
In the content based filter 630, the access node may discard redundant content items based on matching content 632. The access node may also discard content items based on scenic filter settings 634. For example, the scenic filter settings may indicate whether content items should be discarded or kept based on quantity of faces in the content item, or the identification of outdoor scene, landscape, skyline, or the like.
In the facial recognition filter 640, the access node may determine facial recognition patterns 642. For example, the facial recognition filter 640 may identify persons and categorize the content items based on identified persons. The facial recognition filter 640 may discard content items based on facial recognition settings 644. For example, the facial recognition filter 640 may attempt to include only the content items that have a particular person and discard content items that do not include the particular person. Alternatively, the facial recognition filter 640 may attempt to create an even distribution of content items among various persons, such that no one person monopolizes the collection of content items.
At ranking process 650, the access node may determine ordering of the content items based on characteristics of the content items 652. For example, the quality metric 622, facial recognition patterns 642, or other information may be used to rank the content items. Alternatively, the access node may use a timestamp data embedded in the content items to order the content items. In one embodiment, even if content items are uploaded at different times, the access node may rearrange the content items into the proper chronological order.
At compilation process 660, the access node may generate a compilation content item 665. As described previously, the compilation content item 665 may represent a new content item that is derived from a plurality of content items.
The access node may upload 670 the compilation content item and/or the content items that have not been discarded by the basic filter 610, quality-based filter 620, content based filter 630, or facial recognition filter 640. In one embodiment, the ranking process 650 may be used to determine the order in which the access node uploads the content items.
In one embodiment, the access node may select a subset of the plurality of content items 601 to transmit to the server. The subset of content items to upload may be based at least in part a ranking of content items by ranking process 650 or the quality metrics 622. The access node may determine a quantity of content items to include in the subset based at least in part on an upload limit, user data plan, or backhaul bandwidth availability.
As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method, or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, a software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “unit” or “system.” Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
Any combination of one or more computer readable medium(s) may be utilized, with the sole exception being a transitory, propagating signal. The computer readable medium may be a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
Computer program code embodied on a computer readable medium for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
Aspects of the present disclosure are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present disclosure. Each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks. The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
The electronic device 700 may include a media processing unit 730 configured to implement various embodiments described in the previous figures. For example, the media processing unit 730 may provide media processing service(s) 120. The media processing service(s) 120 may include one or more of the content processing module 121, rank module 122, filter module 123, compilation module 124, upload/share module 125 described in
Any one of these functionalities may be partially (or entirely) implemented in hardware and/or on the processor unit 702. For example, the functionality may be implemented with an application specific integrated circuit, in logic implemented in the processor unit 702, in a co-processor on a peripheral device or card, etc. Further, realizations may include fewer or additional components not illustrated in
While the embodiments are described with reference to various implementations and exploitations, these embodiments are illustrative and that the scope of the present disclosure is not limited to them. In general, techniques for providing media processing services on an access node as described herein may be implemented with facilities consistent with any hardware system or hardware systems. Many variations, modifications, additions, and improvements are possible.
Plural instances may be provided for components, operations or structures described herein as a single instance. Finally, boundaries between various components, operations and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of the present disclosure. In general, structures and functionality presented as separate components in the exemplary configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements may fall within the scope of the present disclosure.
