Patent Application
20150254806
This disclosure relates generally to techniques to display a group of media items in an optimal media arrangement. More particularly, the disclosure relates to techniques to efficiently retrieve and load media items for displaying in a media arrangement.
With the rapid increase in the number of devices capable of capturing digital media and the number of repositories for such media, there exists a need for an interface that is capable of aggregating, sorting, and displaying all—or a substantial portion—of the media to which a user has access to in a visually pleasing manner. Because media items may be stored on various sources (e.g., local device storage, remote social networking services, remote storage services, etc.), a user interface that presents multiple media items to the user in a visually pleasing manner should be able to retrieve the media items efficiently. In addition, because many of the devices capable of capturing and displaying such media items have relatively limited memory and processing capabilities (e.g., mobile devices such as phones, tablets, and PDAs), the user interface should be capable of retrieving and storing the media items in an efficient manner such that storing the media items does not result in memory storage constraints, yet allows for the items to be displayed seamlessly and in a manner that improves the user experience.
TO be filled in after finalizing claims.
This disclosure pertains to systems, methods, and computer readable media for displaying selected media items in a manner that enhances the user experience. In general, a set of media items may be matched to, and displayed in accordance with, one of a number of predefined media arrangements as described in the co-pending applications entitled “Semi-Automatic Organic Layout for Media Streams” (Ser. No. 14/080,522), “Multi Source Media Aggregation” (Ser. No. 14/080,553), and “Viewable Frame Identification” (Ser. No. 14/080,591), the contents of which are incorporated herein by reference. Media items that are displayed in a media arrangement may be stored on different sources (e.g., local device storage, remote social networking services, remote storage services, etc.), some of which may be accessed via a network. Retrieving media items that are stored remotely and are accessed over a network may require more time than is available when a user is quickly interacting with (e.g., scrolling through) a media arrangement. This may result in an automatic slowing or stopping of the scrolling operation to wait for all media items to be retrieved, or it could result in having blank media frames in the visible area of the media arrangement, both of which can negatively impact the user experience. It may also not be possible or advisable to retrieve all such media items in advance, because of memory storage and/or processing limitations on the device on which the media arrangement is being displayed. In one embodiment, a multi-level cache system and asynchronously tuned retrieval and loading operations may be used to retrieve and load media items progressively and in an optimized manner as the user views the media arrangement.
In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the inventive concept. As part of this description, some of this disclosure's drawings represent structures and devices in block diagram form in order to avoid obscuring the invention. In the interest of clarity, not all features of an actual implementation are described in this specification. Moreover, the language used in this disclosure has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter, resort to the claims being necessary to determine such inventive subject matter. Reference in this disclosure to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention, and multiple references to “one embodiment” or “an embodiment” should not be understood as necessarily all referring to the same embodiment.
It will be appreciated that in the development of any actual implementation (as in any development project), numerous decisions must be made to achieve the developers' specific goals (e.g., compliance with system- and business-related constraints), and that these goals will vary from one implementation to another. It will also be appreciated that such development efforts might be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the art of data processing having the benefit of this disclosure.
Referring to
Once the media items that satisfy the required criteria are identified, a determination may be made as to how to order the identified media items to achieve optimal media arrangement 105. This may involve evaluating the properties of the ordered media items against the properties of a set of predefined media arrangements to identify the most appropriate media arrangement (e.g., media arrangement 105). Once an optimal media arrangement has been identified, it may be necessary to retrieve the identified media items to be able to present the media arrangement in a visually pleasing manner. However, because the number and/or size of the media items may be large, it may not be feasible to retrieve all of the identified media items in advance, as the time it takes to retrieve all the media items may be too long. For example, it is possible that the user would desire to view the optimal media arrangement as soon as it has been created, thus not leaving enough time for all of the media items to be retrieved and loaded in advance. Moreover, even if there is enough time to retrieve all of the media items in advance, it may still not be desirable or even possible to do so because of memory storage constraints. Thus, a method is needed to efficiently and progressively retrieve media items and load them into their respective media frames in the media arrangement.
As the number of media items identified for an optimal media arrangement may be large, only a subset of the media items may appear within a viewable portion of the media arrangement at any given time. Each media item may be displayed in a media frame in the viewable portion of the media arrangement. Thus, each media frame may be associated with a media item when it is being prepared to be displayed. As the viewable portion changes, the media frame may no longer need to be associated with a previously viewed media item. This allows a presentation engine or a shipping framework (e.g., UICollectionView, in one embodiment—not shown) to use a limited number of media frames by purging and reusing the same media frames for new media items as the viewable portion changes. In this manner, user interface resources involved can be optimized, thus increasing efficiency. As the user interacts with the display of media items in the media arrangement (e.g., scrolls through the media items), the system may identify which portion of the media arrangement, and consequently which media items, are viewable at any given time.
Referring to
As discussed above, media items 210 in each page of the media arrangement 205 may be stored in multiple sources, some of which may be remote. When a user is quickly interacting with media arrangement 205 (e.g., scrolling through the pages), the time it takes to retrieve some of the media items 210 and load them into a media frame for presentation to the user may be longer than the time it takes the user to move through the pages of the media arrangement. This may result in one or more blank media frames in a visible area of the media arrangement or may cause delays in scrolling through the media arrangement, which could negatively impact the user experience.
To resolve these issues, techniques may be utilized that make use of multiple levels of resolution for each media item, multiple levels of memory caches, asynchronous tuned operations (e.g., by thread priority, dependencies, and/or scheduling priorities), image decompression, and/or prefetching procedures to progressively retrieve and load media items. Referring to
Memory cache 305 may include two or more separate memory cache portions. In one embodiment, memory cache 305 may be segmented into a low resolution volatile memory cache 310, and a high resolution volatile memory cache 315. Low resolution LRU memory cache 310 may be used to store low resolution media items and high resolution LRU memory cache 315 may be used to store high resolution media items. This is because two or more levels of resolution may be available for each media item. A lower resolution version of the media item may require less time to retrieve and load and also less storage space, and may thus be used when saving time and/or space is more important than having a higher resolution media item. The low resolution version may also be used for different media frames regardless of the size of the media frame, while the higher resolution media item may selected such that its' size corresponds to the size of the media frame to which it is being loaded. Additionally, the low resolution media item may be usable across different media arrangement layouts and orientations
The media items from persistent disk cache 300 may be loaded into memory cache 305 on demand. In one embodiment, memory cache 310 and memory cache 315 may be administered as least recent used (LRU) caches. That is, after a media item is loaded into memory caches 310 or 315, the media item may be sent back to disk cache 300 upon memory pressure and in accordance with memory cleanup protocols. For example, if a media item corresponding to a certain group path has not been accessed from memory cache 310 or 315 for a specific period of time, the media item may be dropped from memory cache 310 or 315 and sent to disk cache 300. In one embodiment, the memory item dropped from memory cache 310 or 315 is not sent back to disk cache 300. By removing the least recent used memory item(s), LRU memory caches can keep the memory below a certain storage limit and thus prevent memory pressure. In an alternative embodiment, the LRU memory may remove memory items that have not been used in a specific period of time to keep memory below a certain storage limit.
One or more actions 330 such as a media item retrieval request may result in the performance of one or more operations 320. Operations 320 may be queued in operations queue 325, and upon execution of a particular operation, the media item corresponding to the operation may be retrieved from persistent disk cache 300 and loaded into memory 305. All operations 320 may concurrently access memory 305.
Referring to
Regardless of how the operation 400 begins, after it has started first the media frame is associated with the media item that will be loaded into the media frame in accordance with the defined media arrangement (block 404). This may involve updating the media frame data source path index to point to the specific media item.
After the media frame is associated with the new media item, all previously scheduled and/or running operations for that media frame may be canceled (block 406). This step may be needed because as will be described in detail below many operations may be scheduled and/or performed simultaneously, and various user interactions with the media arrangement may result in changing the priority or the need for some of those operations. For example, if the media item associated with the media frame of operation 400 is being retrieved or pre-fetched (or is scheduled to be retrieved or pre-fetched) in other operations, such operations may be canceled at this stage (block 408).
After cancelations steps of blocks 406 and 408, the visual features of the media frame may be updated in accordance with the associated media item (block 408). This may involve updating a title of the media frame, a badge, or other similar features, After updating the look of the media frame, the operation 400 may determine if the visible area of the media arrangement is changing (i.e., there is scrolling) (block 412). In one embodiment, the determination of whether the visible area is changing is made based on a threshold speed. If the visible area is changing at a speed that is lower than a predetermined threshold, then the procedure may consider that as the visible area not changing. If it is determined that the visible area is changing, the procedure moves to operation 420 of
Operation 420 begins by determining if the media frame will be part of a final destination of the media arrangement (i.e., the media frame will be presented on a stationary viewable area of the media arrangement when scrolling stops) (block 424). This may be determined in one embodiment by calculating the rate of deceleration of scrolling, identifying the location of the media frame in the media arrangement and calculating the relation between the location and the rate of deceleration. Alternative embodiments may be used for determining whether the media frame will be part of a statutory viewable area.
If it is determined the media frame will be part of the final destination viewable area (the “YES” prong of block 424), the operation may then determine if a high resolution version of the associated media item is available in high resolution memory cache 315 (block 426). This is because if the media frame will be part of the final destination and as such presented to the user in stationary form, it may be preferable to use the high resolution version of the media frame for better quality viewing. However, if the media frame is being presented to the user during scrolling, it may be more important to quickly retrieve and/or upload the media item than to present a higher quality image, particularly since the media item is only presented to the user in passing.
If the operation determines that a high resolution version of the media item is available (the “YES” prong of block 426), the high resolution media item may be loaded into the media frame (block 428). If it is determined at block 424 that the media frame will not be part of the final destination viewable area (the “NO” prong of block 424) or it is determined at block 426 that a high resolution version is not available, then the operation may check low resolution memory cache 310 to determine if a low resolution media item is available for loading (block 430). When the low resolution version of the media item is available (the “YES” prong of block 430), the media item may be loaded into the media frame (block 428) thus completing operation 420 and consequently operation 400.
When it is determined that a low resolution version of the media frame is not available at the low resolution memory cache 310 (the “NO” prong of block 430), the operation may move to operation 450.
When neither the high resolution nor the low resolution version of the media item is available at the memory cache 305, the procedure may move to operation 450 first and then move to operation 470. This means that both operations 450 and 470 may be scheduled and running at the same time. However, the two operations may have different priorities and may be running in different priority threads. Generally operation 450 which involves retrieving a low resolution version of the media item is of a higher priority than operation 470 which relates to retrieving a high resolution version of the media item. Both of these operations may also run in secondary threads, while operations 400, 420 and 422 may run in the main thread. This ensures that while higher resolution media items are being retrieved for better quality display, media items are also being loaded into the next media frame(s) for fast access.
Referring to
Referring to
In one embodiment, in order to ensure a smooth transition between pages of the media arrangement as the visible area changes and to reduce the chances of encountering a blank media frame, it may be advantageous to pre-fetch media items.
In accordance with one embodiment, operation 500 for pre-fetching media items begins by reviving a triggering (block 502). The triggering event could be, for example the launching of the application for creating optimal media arrangements, the user selecting a particular media arrangement for viewing, or the start or stopping of scrolling. During continuous scrolling, a triggering event may be the recognition that the visible area is continuously changing. Once such a triggering event is received, operation 500 may first determine if the visible area is changing (i.e. scrolling is in progress) (block 504). If the visible area is changing (the “YES” prong of block 504), then operation 500 may identify media items in the upcoming visible areas that need to be pre-fetched (block 506).
In one embodiment, the operation may identify media items that correspond to an area of the media arrangement that will be made visible next. To do this, in one embodiment, the operation may select media items in the page adjacent to the current visible area in the direction of scrolling. The number of media items that may be identified for pre-fetching may vary. In one embodiment, the operation may identify media items corresponding to the next two visible areas in the direction of scrolling. Alternatively, the operation may identify media items for just the next visible area. In another embodiment, the operation may identify media items corresponding to the next three visible areas. In yet another embodiment, the operation may identify media items for up to ‘n’ visible areas in the direction of scrolling.
Once all of the media items that need to be pre-fetched have been identified, the operation 500 may schedule pre-fetching operations separately for each of those media items to retrieve low resolution versions of those media items that are not already available in the low resolution cache (block 508). The pre-fetch operations scheduled are similar to the operation 450 in that they include steps for determining if the low resolution version of the media item is available at the low resolution cache and if not retrieving the low resolution version, storing it at the disk cache, decompressing it, and then storing it in the low resolution cache.
In one embodiment, the pre-fetch operations may be scheduled in order of priority. Generally, priority may be given to media items that will be viewable closest to the current visible media frames. Thus, if, for example, media items for the next two visible areas are being pre-fetched, the media items for the next visible area are scheduled to be pre-fetched first, and among those media items the ones closest to the edge of the current visible area are given the highest priority.
After scheduling these pre-fetch operations, the procedure may cancel all other previously pending pre-fetch operations (i.e., not the ones that were just scheduled) (block 510). That is because, as the user goes through the media arrangement (e.g., by scrolling through the media arrangement) prior pre-fetch operations that have not yet been completed may no longer be needed as the user may have already moved passed those media items, or the priority for those operations may need to be changed as the location of those media items with respect to the current visible area may have changed. In order to enable quick cancelation, all operations may have cancelation points between each step.
Once prior pending pre-fetch operations are canceled, the operation 500 may schedule pre-fetch operations for retrieving high resolution versions of media items for any media items that is identified as being part of the final destination visible area (block 512). As discussed above, this can be done by calculating the rate of scrolling and deceleration and determining what area of the media arrangement will be included in the final destination once scrolling stops. Because the user will likely spend more time viewing the stationary part of the media arrangement, quality may be of more importance for the final destination and as such high resolution pre-fetch operations may be scheduled for those media items. The pre-fetch operations scheduled are similar to the operation 470 in that they include steps for determining if the high resolution version of the media item is available at the high resolution cache and if not retrieving the high resolution version, storing it at the disk cache, decompressing it, and then storing it in the high resolution cache. It should be noted that these high resolution pre-fetch operations may be scheduled concurrently with the low resolution pre-fetch operations of block 508. However, high resolution pre-fetch operations may receive a lower priority than low resolution pre-fetch operations. Additionally, in one embodiment, a high resolution pre-fetch operation for each media item may be dependent on the low resolution pre-fetch operation for that media item. As such the high resolution operation may not occur until the low resolution operation has completed.
When at block 504 it is determined that the visible area is not changing (i.e., there is no scrolling and the visible area is stationary), operation 500 may load high resolution media items into the visible media frames (block 514). This may be done to improve the quality of the user experience by providing high resolution media items when the user is likely to spending more time viewing the media frames due to the stationary status of the media arrangement. Steps involved in the loading of high resolution media items may include determining if the high resolution media item is available at the high resolution cache and if not going to operation 470 to retrieve the high resolution version, before loading it.
Once high resolution media items have been loaded for the visible media frames, operation 500 may identify media items corresponding to areas of the media arrangement that are adjacent to the currently visible area. Such media items may be identified for performing pre-fetching operations (block 516). The number of media items that may be identified for pre-fetching may vary. In one embodiment, the operation may identify media items that correspond to one area to the left and one area to the right of the currently visible area, each identified area corresponding to the size of the visible area. In other words, the operation may identify media items that may be made visible next in either direction of scrolling. In one embodiment, the operation may identify media items corresponding to ‘n’ areas to the left (e.g., four) and ‘m’ areas to the right (e.g., four) of the current visible area for a total of n m (e.g., 8) visible areas. Assuming that each visible area includes an average of about 16 media frames, this means up to 128 media items may be pre-fetched. Considering an average size of about 64 KB per low resolution media item, this may require about 8 MB to 10 MB of memory space, when all media items are decompressed, which will most likely not exceed the memory storage constraints of the device.
After identifying the media items for which pre-fetch operations should be performed, operation 500 may schedule low resolution pre-fetch operations for retrieving the low resolution version of the identified media items (block 516). The pre-fetch operations scheduled may be similar to the operation 450 in that they may include steps for determining if the low resolution version of the media item is available at the low resolution cache and if not retrieving the low resolution version, storing it at the disk cache, decompressing it, and then storing it in the low resolution cache.
In one embodiment, similar to the scheduling of block 508, the pre-fetch operations may be scheduled in order of priority by giving higher priority to media items that will be viewable closest to the current visible area. Thus, those media items that are closest to the edge of the current visible area may be given the highest priority. After scheduling these pre-fetch operations, the procedure may cancel all other previously pending pre-fetch operations (not the ones that were just scheduled) (block 520). That is because, as there's no scrolling occurring at the moment, all other previously scheduled pre-fetch operations may have the wrong priority or they may no longer be needed.
Once previously pending pre-fetch operations are canceled, operation 500 may move to schedule high resolution pre-fetch operations for certain identified media items. Generally, these may be scheduled for media items that correspond to areas adjacent to the current visible area and are a subset of the media items identified in block 516. For example, if block 516 identifies media items corresponding to four areas to the left and four areas to the right of the current visible area, block 522 may schedule high resolution pre-fetch operations for media items corresponding to one area to the left and one area to the right of the current visible page. That is because high resolution pre-fetch operations may take longer to complete and may require more memory. As such they may only be performed for areas immediately next to the currently visible area. Similar to the high resolution pre-fetch operations of block 512, the pre-fetch operations of block 522 may also be scheduled at the same time as the low resolution pre-fetch operations of block 518, but the high resolution operations may receive lower priority and they may be dependent on the low resolution operation being completed first.
As discussed above, in order to efficiently pre-fetch and load media items into media frames of a media arrangement, multiple operations may be scheduled and/or occurring at the same time in different threads and/or different queues. In order for all of the operations to work effectively together and to reach a fast frame-per-second rate for the media arrangement (e.g., 60 frames-per-second), thread and scheduling priorities may need to be made. In one embodiment, loading of low resolution media items into a media frame (operations 400 and 420) may be granted the highest scheduling and thread priority. Such operations may occur at the main thread and may be given the designation of Very High Priority. Operations involving retrieving of low resolution media times (operation 450) may also be designated as Very High Priority operations when they are performed in order to load a media frame as part of operation 400. However, these low resolution retrieval operations may occur at a secondary thread. The next priority level, Normal Priority, may be given to loading high resolution media items such as operation 422 which may also occur at the main thread. Retrieval of high resolution media items (operation 470) may also receive a Normal Priority designation when such an operation is performed as part of loading a media frame. However, the retrieval operation may be performed at a secondary thread. Pre-fetch operations for retrieving low resolution media items receive the next designation which is Low Priority, while pre-fetch operations for retrieving high resolution media items receive the lowest priority level, Very Low Priority.
In the operations queue 614 having a depth of four, the operations shown in box 610 may be scheduled such that all four of the 602 operations are performed first. Operations 604 will be next in the queue after operations 602-3 and 602-4 are completed. Next, operations 606 will be performed in order, and then finally operations 608 will be done at the end of the queue. Thus, by using thread priority and operations scheduling priority, embodiments described herein provide an efficient progressive method of loading and retrieving media items for a media arrangement that reduces the number blank media frames presented to the user and provides high resolution media items when possible and/or needed.
Additionally, in one embodiment, in an independent background thread running at a very low priority, all media items from services not cached on cache disk 300 may be pre-loaded in order to reduce latencies. This may involve operations such as updating media items that have not been to a local metadata cache and/or retrieving low resolution media items in a low priority thread designated for such tasks. Such operations may be performed as long as no other operations are being run. When the visible area starts changing, this background routine may be paused so it does not interfere with the other operations. This operation may be stopped in one embodiment, if the visible area begins changing.
Referring to
Processor 705 may execute instructions necessary to carry out or control the operation of many functions performed by device 700. Processor 705 may, for instance, drive display 710 and receive user input from user interface 715. User interface 715 can take a variety of forms, such as a button, keypad, dial, a click wheel, keyboard, display screen and/or a touch screen. Processor 705 may also, for example, be a system-on-chip such as those found in mobile devices and include a dedicated graphics processing unit (GPU). Processor 705 may be based on reduced instruction-set computer (RISC) or complex instruction-set computer (CISC) architectures or any other suitable architecture and may include one or more processing cores. Graphics hardware 720 may be special purpose computational hardware for processing graphics and/or assisting processor 705 to process graphics information. In one embodiment, graphics hardware 720 may include a programmable graphics processing unit (GPU).
Sensor and camera circuitry 750 may capture still and video images that may be processed, at least in part, in accordance with the disclosed techniques by video codec(s) 755 and/or processor 705 and/or graphics hardware 720, and/or a dedicated image processing unit incorporated within circuitry 750. Images so captured may be stored in memory 760 and/or storage 765. Memory 760 may include one or more different types of media used by processor 705 and graphics hardware 720 to perform device functions. For example, memory 760 may include memory cache 305, read-only memory (ROM), and/or random access memory (RAM). Storage 765 may store media (e.g., audio, image and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. Storage 765 may include one or more non-transitory storage mediums including, for example, magnetic disks (fixed, floppy, and removable) and tape, optical media such as CD-ROMs and digital video disks (DVDs), and semiconductor memory devices such as Electrically Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). Persistent disk cache 300 may be maintained within at least a portion of storage 765. Memory 760 and storage 765 may also be used to tangibly retain computer program instructions or code organized into one or more modules and written in any desired computer programming language. When executed by, for example, processor 705 such computer program code may implement one or more of the operations described herein.
It is to be understood that the above description is intended to be illustrative, and not restrictive. The material has been presented to enable any person skilled in the art to make and use the inventive concepts described herein, and is provided in the context of particular embodiments, variations of which will be readily apparent to those skilled in the art. For example, some of the disclosed embodiments may be used in combination with each other. As another example, an implemented cache system may use more than three levels. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of the invention therefore should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.”
