The present invention relates to a method and server for establishing the coordinated consumption of a streamed media object by multiple devices.
For members of a party visiting a space with which media objects are associated, the shared experience can be enhanced by the sharing of such objects by the group members for presentation on mobile devices carried by the group members. Conventionally, media objects are shared by reference, for example by passing an appropriate URI. Where the media object is a streamed object, a recipient using a shared reference to the media object will typically experience the streamed media object from its beginning, whilst the person who passed on the reference will already be some way through the streamed object. However, the person who passed on the reference may wish the recipient of the reference to experience the media object in a synchronized manner, i.e. to ensure that they both experience the same parts of the media stream in the same order and at the same time. Colloquially, one person wishes to invite the second person to “listen to this” (or “look at this” etc). Multicast streaming protocols are known which enable a single media stream to be sent to multiple devices over the Internet with synchronization of multiple media channels within a composite, structured media stream (e.g. SMIL). However such protocols are not widely deployed in the internet.
It is an object of the present invention to provide a way of establishing coordinated presentation of a streamed media object on multiple devices without the use of multicasting protocols.
According to one aspect of the present invention, there is provided a method of establishing coordinated consumption of a streamed media object by first and second devices, comprising the steps of:
According to another aspect of the present invention, there is provided a server for use in establishing coordinated consumption of a streamed media object by first and second devices, the server comprising:
According to another aspect of the present invention, there is provided a method of coordinated consumption of a streamed media object by first and second devices, the media object being accessible for streaming from a server, the method comprising the steps of:
Embodiments of the invention will now be described, by way of non-limiting example, with reference to the accompanying diagrammatic drawings, in which:
In considering such an arrangement, it is convenient, though not essential, to introduce the abstraction of a virtual feature which is the subject of each zone. Each such virtual feature is given a number of properties such as a unique identifier, a location in the real-world environment, the real-world extent of the zone associated with the feature, a subject description indicating what the feature concerns, and a set of one or more media-object identifiers identifying the media objects (or “feature items”) associated with the feature. The zone associated with a virtual feature is referred to hereinafter as the ‘active zone’ of the feature.
For a feature that is intended to correspond to a particular real-world item (and typically having an active zone that maps to an area about a real-world object), this can be indicated in the subject description of the feature. Using the feature abstraction makes it easier to associate feature items that all relate to the same zone and also facilitates adding/removing these features items since data about the real-world extent of the related zone is kept with the feature and not each feature item.
Each feature is represented by a feature record held in a data-handling system, the feature records together defining the aforesaid virtual world that maps to the real-world environment. Each feature can be thought of as existing in this virtual world with some of these virtual features mapping to real-world objects.
As already noted, when a user is detected as within an active zone of a feature, one or more feature items are delivered to the mobile device of the user for presentation to the user. A feature item can be presented automatically to the user upon delivery or the item can be cached and only presented upon the user having expressed an interest in the feature in some way such as by dwelling in the active zone of the feature more than a minimum time or by explicitly requesting presentation of the feature item. Indeed, the delivery of the feature item to the mobile device can also be deferred until the user is detected as having expressed an interest in the feature; however, since this approach can introduce a delay before the item is available for presentation, the embodiments described below deliver feature items to the mobile device of the user without awaiting a specific expression of interest in each feature (though, of course, a general filtering may be applied as to what items are delivered according what types of features are of interest to the user). Preferably, each feature or feature item is given a property indicating whether feature item delivery is to be effected automatically upon delivery or only after a user has expressed an interest in the feature; this enables important items (such as warning messages concerning features associated with potentially hazardous real-world items) to be pushed to the user whilst other items are subject to an expression of interest by the user. Advantageously, a user may elect to have feature items automatically presented even when the corresponding feature/item property does not require this. Furthermore, since as will be described hereinafter, pre-emptive caching of feature items in the user's mobile device may be implemented, automatic presentation is qualified so as only to apply where the user is in the active zone of the feature with which the feature item is associated.
Considering the
Virtual features are also defined in correspondence to the majority of openings 23 between rooms, the active zones 25 associated with the features again been indicated by dashed lines. Typically, the feature items associated with these features are incidental information concerning the room about to be entered and are automatically presented. It will be seen from
On entering the exhibition hall 10, a user 30 collects a mobile device 31 from the reception desk 18 (or the user may have their own device). This device 31 cooperates with location-related infrastructure to permit the location of the user in the hall 10 to be determined. A number of techniques exist for enabling the location of the user to be determined with reasonable accuracy and any such technique can be used; in the present example, the technique used is based on an array of ultrasonic emitters 33 (represented in
The exhibition hall is equipped with a wireless LAN infrastructure 36 comprising a distribution system and access points 37. The wireless LAN has a coverage encompassing substantially all of the hall 10, the boundary of the coverage being indicated by chain-dashed line 38 in
It will be appreciated that communication between the device 31 and service system 35 can be effected by any suitable means and is not limited to being a wireless LAN.
Much of the foregoing functionality will typically be provided by a program-controlled general purpose processor though other implementations are, of course, possible.
The visit data held by memory 44 will typically include a user/device profile data (for example, indicating the subjects of interest to the user, the intended visit duration, and the media types that can be handled by the device), an electronic map of the hall 10, the user's current location as determined by the subsystem 40, and the identity of the feature (if any) currently being visited together with the IDs of its related feature items. The visit data also includes a feature history for the visit, which is either:
If a visited-feature history list is kept, a history of accessed features can be embedded in it by providing each feature in the history with an associated flag to indicate whether or not the feature was accessed whilst current. Although keeping a visited-feature history provides more information about the visit, it will inevitably use more memory resources than an accessed-feature history and in many cases it will only be desired to track features which the user has found sufficiently of interest to access an associated feature item. Where the purpose of the feature history is simply to keep a list of features (and related feature items) that were of interest to the user, it may be desirable to exclude from the list features for which items were automatically presented but are not associated with exhibits (real or virtual)—that is, exclude features concerned with incidental information about the hall.
The feature history preferably covers the whole of the visit though it may alternatively only cover the most recently visited/accessed features. In either case, the most recent several entries in the history list form what is hereinafter referred to as the “feature tail” of the user and provides useful information about the path being taken by the user.
The visit data held in memory 43 may further include details of a planned route being followed by the user, and a history of the locations visited by the user (this may be a full history or just the locations most recently visited—hereinafter termed the “location tail” of the user).
The service system 35 comprises the following main functional elements:
The functional elements of the service system 35 can be configured as a set of servers all connected to the LAN 51 or be arranged in any other suitable manner as will be apparent to persons skilled.
The mobile device 31 and service system 35 provide a number of useful capabilities that will each be described in detail below after an overview of the general operation of the mobile device and service system during a visit. It is to be understood that the split of functionality between the mobile device 31 and service subsystem 35 can be varied substantially form that indicated for the
In general terms, a user starting a visit can request a route to follow using the user interface 48 of the mobile device 31 to indicate parameters to be satisfied by the route. This route request is sent by the visit manager to route planner 50 and results in the download to the mobile device 31 of a planned route. The path guide 49 then provides the user (typically, though not necessarily, only when asked) with guide indications to assist the user in following the planned route. Where the interface 48 includes a visual display, this can conveniently be done by displaying a map showing the user's current location and the planned route; in contrast, where only an audio interface is available, this can be done by audio cues to indicate the direction to follow. A user need not request a planned route and in this case will receive no guide indications. A user may request a route plan at any stage of a visit (for example a route to an exhibit of interest).
As the user moves through the hall, the location determination subsystem 40 sends periodic location reports 62 (see
When a location report 62 is received by the manager 54, it passes on the user's current location in the report to the pheromone trail subsystem 55 to enable the latter to build up trail data from all devices; additionally, the user and/or type identifier may be passed on to subsystem 55 if provided in the location report. The user's current location is also passed to the item-data response subsystem 56 together with any profile data and prediction-assist data in the location report 62. The item-data response subsystem 56 then constructs and sends a response 65 (see
More particularly, the location-item to feature translation unit 57 of subsystem 56 uses the data passed to subsystem to determine the feature, if any, currentlybeing visitedbythe user and thus what feature items are relevant to the user in their current location. In doing this, the unit 57 may also use the supplied profile data to disregard both features that do not relate to a subject of interest to the user and feature items of a media type that cannot be handled by the mobile device 31. The unit 57 may also use elements of the prediction-assist data (for example, the location or feature last encountered before the current one) to enable it to determine the direction of progression of the user and thus to select between feature items of a feature in dependence on the direction of approach of the user. This is done, for example, for the features associated with openings 25 in order to select a feature item appropriate to entering a room. The IDs of feature items identified by the unit 57 together with the identity of the corresponding feature and the status of the automatic presentation flag of the feature, form a first part 66 of the response 65 to be sent back to the mobile device 31. Where the current location does not correspond to the active zone of any feature, the first response part 66 simply indicates this.
A second part 67 of the item-data response 65 is produced by the prediction unit 58 and comprises a list of the feature items most likely to be needed in the near future by the mobile device 31; for each such feature item, the second response part 67 includes the feature ID, its type, size and probability of usage (discussed in detail hereinafter). Like the unit 57, the unit 58 uses supplied profile data to disregard feature items of features not of interest to the user or of a media type that cannot not be handled by the mobile device 31. The number of feature items identified in response part 67 is preferably limited (for example, to ten such items). The item-data response subsystem 56 then sends the response 65 back to the mobile device 31 of the user by using a return address supplied with the original location report 62 and passed to subsystem 56 by the manager 54.
Rather than having the prediction unit 58 provide a prediction each and every time the mobile device 31 sends a location report, it is possible to arrange for the prediction unit 58 only to operate when required by the mobile device 31 with the latter only requiring a prediction, for example, every nth location report or only after the user has moved a certain distance since the last prediction made by unit 58. Conveniently, the location report field used to carry the prediction-assist data is also used to indicate when a prediction is required by, for example, setting the field to a predetermined value when prediction is not required.
The item-data response received back at the mobile device 31 is processed by the visit manager 47. If the first part 66 of the response identifies a feature (thereby indicating that the current location of the user corresponds to the active zone of feature), the manager 47 updates the ‘current feature’ data in memory 45 to the feature identifier and item IDs in the first response part. These item IDs are also passed to the cache manager 45 and are used by the latter to request immediate delivery of these items from the server 53 of the service system to cache 44, if not already present in the cache. If the feature history data held by memory 43 relates to visited, rather than accessed, features, and ifthe feature identifier and item IDs in the first response part 66 differ from the most recent entry in the feature history list, the latter is updated with the feature identifier and item IDs from the first response part 66.
In the case that no feature is identified in the first part of the response 65, the ‘current feature’ data in memory 43 is set to null.
The manager 47 also determines whether the (first) feature item (if any) identified in the first response part 66 is to be immediately presented to the user, this determination taking account of the setting of the automatic presentation flag in the first part of the response, any user indication (stored, for example in the profile data) that all items are to be automatically presented, and any monitored indications of the user's interest in the currently-visited feature. Where a feature item identified in the first response part is to be immediately presented to the user, the manager 47 requests the item from the cache manager 45 (there may be a delay in the delivery of the item if it has not yet been received from the server 53). At the same time, if the feature history concerns accessed features the manager 47 updates the feature history with an entry corresponding to the feature identifier and item IDs forming the ‘current feature’ data; where the feature history although recording all visited features, provides for indicating whether a feature has been accessed, the manager updates the feature history accordingly.
With respect to the data contained in the second part 67 of the response 65, the visit manager simply passes this data to the cache manager 45 which determines whether or not to request from server 53 any of the items identified that are not already in the cache 44. The cache manger 47 in making this determination takes account of the probability that an item will be needed in the near future and the available cache space. The cache manager 45 may decide to create additional cache space by flushing one or more items from the cache and/or by reducing the space they occupy, as will be more fully described hereinafter.
In this manner, the cache manager 45 seeks to ensure that the next item requested by the visit manager 47 as the user progresses to the next feature will already be in the cache 44.
Following the processing of an item-data response by the visit manager, whenever a feature item is accessed (presented) either as a result of the visit manager 47 determining that the current feature is of interest to the user or as result of the user specifically requesting the item (for example, after inspecting the list of items associated with the current feature), then where the feature history data records accessed feature information, the visit manager 47 checks if the feature associated with the accessed item is the current feature and, if so, updates the feature history to record the feature as an accessed one if not already so indicated.
The visit manager can also be arranged to keep a list in memory 43 of the individual feature items accessed.
Having described the general operation of the mobile device 31 and service system 35, a more detailed description will now be given of some of the functionality embodied in the arrangement of
Coordinated Consumption
Often a user visiting the exhibition hall 10 will be doing so as a member of a party, be it a family group, a tour party or some other group. Where at least some members of the party have respective mobile devices, the situation is likely to arise that one member with a mobile device accesses a feature item that they then wish to share with the other party members with mobile devices; indeed, this may the case not only for feature items but for any data item available from the service system 35 or other source accessible via the communications infrastructure exemplified in the embodiment of
However, where the item concerned is a streamed media object (in particular, audio and/or video streams), simply having each mobile device independently access the object will result in uncoordinated consumption of the object at each device.
Three arrangements for providing for coordinated consumption of the streamed media object are described below with respect to
Coordinated consumption of a streamed media object on different devices implies coordinated presentation of the object on each device (in this context, “coordinated” is not intended to be restricted to precisely synchronized presentations and is intended to cover presentations that closely match each other). Since streamed media may be sent at a rate greater than its rate of consumption with the receiving device buffering the received but not yet consumed portions of the stream, there may be an offset at a receiving device between the current presentation position within the media object and the current receiving position within the same object (that is, the current position reached within the object as regards the most recently received media-object data at the device—typically, this will be closely similar to the current sending position reached by the media-object server in streaming the media object to the device). This offset between the receiving and presentation positions at the receiving device is referred to below as the “R-P offset”. The R-P offset will typically change (increase) during the course of the streaming of a media object though its size may be limited by the size of the cache provided at the device for buffering streamed objects.
Where the rate of sending of the streamed media object is, at least on average, matched to the rate of consumption of the object, the R-P offset will be non-existent or small.
A user using a device for the presentation of a media object which the user then decides should be shared with others by presentation on their individual devices, has a choice (at least in theory) regarding whether to continue with the user's own consumption of the media object or to pause this consumption whilst the other devices establish respective streams for the media object concerned. If the initiator of the coordinated consumption pauses media object consumption, the coordination task is simplified; however, if the initiator continues consumption of the media object, there will be a presentation run on between the presentation position reached when the initiator's device requests the other devices to effect coordinated consumption of the media object, and the presentation position reached by the time the other devices are ready to present the media object to their users. This presentation run on is referred to below as the “PRO advance.”
Considering first the arrangement of
More particularly, in
It will first be assumed that the presentation of the media object 200 at the device 31A is not paused at the time the message 202 is sent.
The mobile device 31B on receiving the message 202 estimates an appropriate PRO advance value and then, with or without specific consent from the user 30B, sends a message 204 to the server 53 (arrow 205). The message 204, in addition to containing contact data for the device 31B, also contains the ID of the media object 200 and a predicted start position within the media object from where the server should start streaming the object to the device 31B for the latter to present the object to the user 30B substantially in coordination with the ongoing presentation of the object to the user 30A by the device 31A. The predicted start position is the current presentation position indicated by the position indicator in message 202 plus the estimated PRO advance. The PRO advance can comprise one or more of the following components:
The PRO advance value can be omitted or can be estimated by the server 53 (in which case the position indicator contained in message 202 is forwarded to the server in message 204 instead of the predicted start position).
The message 204 is received by the interface unit 70 of the server 53 and triggers the instantiation of a stream delivery entity 72 (typically a software process) for streaming the media object 200 to the device 31B. The sending start position within the media object 200 is the predicted start position either contained in the message 204 or derived by the unit 70 on the basis of the information contained in the message 204. Once established, the entity 72 initiates streaming of the media object 200 to the device 31B starting at the predicted start position; the device 31B presents the received media-object stream (arrow 206) to the user 30B without delay.
Because the position indicator contained in message 202 relates to the current presentation position reached by the device 31A in presenting the streamed media object, and further because the start position used for the media stream 206 is based on that position indicator, whether or not the device 31A caches the media stream 201 is irrelevant to the operation of the
At the time of sending the message 202 the user 30A may decide to pause the presentation of the media object 200 at the device 31A, this pause being to enable the user 30B to access the media object 200 at the earliest possible coordinated position for going forward from the first user's current position. In this case, the pausing of consumption at device 31A is indicated in the message 202 with the result that the PRO advance is set to zero so that the media-object stream 206 begins from the position indicated by the position indicator in message 202, that is, the paused presentation position of stream 201 at device 31A. As soon as the device 31B starts to receive the stream 206, it starts to present the media object 200 and simultaneously sends a restart message (not depicted in
In fact, rather than the second device 31B starting presentation based on when it sends the restart message to the first device 31A, the second device 31B can be arranged to wait for a “go” message back from the first device 31A before starting. This enables the first device 31A to ensure that all devices which are to participate in coordinated consumption, are ready to do so before consumption begins (in other words, the device 31A waits to receive restart messages from all expected participating devices indicating they are ready to start presentation before it sends out the “go” signal).
It may be noted that provided the first device 31A has a cache for storing the stream 201, pausing the presentation of the stream does not require the sending of the stream to be paused though this can be done, for example, by sending a pause message 208 from the device 31A to the server 53 as indicated by dashed arrow 209 (in which case the sending of the stream 201 must be restarted in due course, for example by a message, not depicted in
A number of variants are possible to the
In another variant, in additional to the current presentation position reached in the media object 200 by device 31A being included in message 202, a more up-to-date version of this information can be passed to the device 31B in a message sent in response to device 31B indicating that it is starting to receive the media object stream 206 from server 53. This enables the device 31B to make adjustments regarding where in the stream 206 it should start presentation of the media object 200.
Considering next the arrangement illustrated in
More particularly, as in the
The interface unit 70 instantiates a stream delivery entity 72 for streaming the object 200 to the device 31B. The entity 72 uses the ID of device 31A and the ID of the object 200 to determine from the stream delivery entity 71 (see arrow 210) the current sending position reached in object 200 for the stream 201. The entity 72 then starts to stream the media object 200 in stream 206 to the device 31B from a position in object 200 corresponding to the current sending position for stream 201 less the R-P offset specified in message 204; in the situation where the presentation of the object 200 to the user 30A at device 31A has not been paused, the P-R offset can be decreased to take account of run on of the presentation during the period from when the message 202 was generated and sent until the time the stream 206 starts to be received at the device 31B. Assuming the presentation at device 31A has not been paused, as soon as the device 31B starts to receive the stream 206, it presents it to the user 31B; this presentation will be substantially coordinated with the ongoing presentation of the media object 200 at device 31A.
If presentation of the media object at device 31A was paused when the message 202 was generated and sent, then the same approaches as described above with respect to the
Where presentation at device 31A is paused at the time of sending the message 202, the
Similar variants to those discussed above for the
The
Considering the
The interface unit 70 instantiates a stream delivery entity 72 for streaming the object 200 to the device 311B. The entity 72 copies the stream 201 to form stream 206 which it sends to the device 31B. Where the presentation of the stream 201 at device 31A has not been paused at the time the message 202 was sent, then in the case where the effective rate of sending of the stream 201 is equal to its rate of presentation (for example, in the situation where the device 31A does not cache the stream 201), the device 31B starts presentation of the stream 206 to the user 30B immediately the device 31B receives the stream 206 with the result that the users 30A and 30B are presented with the object 200 in synchronism. However, if the device has a cache and receives the stream 201 at a greater rate than its presentation rate, were the device 311B to start to present the stream 206 immediately upon receipt, this presentation of media object 200 would be in advance of the presentation of the object 200 at device 31A by an amount corresponding to the R-P offset for the device 31A. In order to overcome this problem, the sending position of the stream 201 is “jumped-back” to the presentation position for stream 201 at device 31A at the time the message 202 is sent, and the cache for stream 201 in device 31A is flushed. Jump-back of the stream sending position can be effected by including the presentation position for device 31A in message 202, this position being passed in message 204 to the interface unit 70 of server 53 which uses it to cause the stream sending entity 71 to “jump-back” its sending position for stream 201 (preferably, in order to avoid the user 30A experiencing a jump-back in what is being presented to the user, allowance should be made for the presentation run on in the period between when the message 202 was generated and when the jump-back effected at the server). Jump-back can alternatively be effected by the device 31A sending a jump-back message 208 (see arrow 209) to the server 53 although in this case the entity 71 should only send the jumped-back stream 201 at the presentation rate of the media object until the entity 72 and the stream 206 have been established (since otherwise an R-P offset could develop in device 31A before the device 311B is ready to start presenting the object 200).
If presentation of the media object at device 31A is paused when the message 202 is generated and sent, then the above-described jump-back approach is preferably used with the jump back being effected at the time the entity 72 is ready to start streaming and a cache-flush command being simultaneously sent to the device 31A. In this case, it does not matter whether the sending of the stream 201 is or is not paused whilst the presentation at device 31A is paused since either way both devices 31A and 311B will receive the same jumped back streams and be ready to present them from the paused presentation position of stream 201 when required. As regards the actual (re)starting of presentation at the devices 31A and 31B, the same approaches as described above with respect to the
Similar variants to those discussed above for the
It will be appreciated that the arrangements of
Of course, the arrangements of
For any of the arrangements of
In addition, for any of the arrangements, one or both of the devices 31A and 31B can be arranged to send further coordination signals at any time during the course of the coordinated consumption of the media object 200 by the devices, in order to compensate for drift in consumption rates. Thus, for example, the device 31A as the instigating device, can be arranged to send periodic coordination signals to the device 31B which indicate the current position reached by the device 31A; the device 31B uses this information either to jump backwards/forwards in the stream it is receiving or to make appropriate adjustments to its rate of consumption of the media stream to return to synchronization with consumption by device 31A (for example, by the time the next coordination signal is expected to be received).
As well as the sending of periodic coordination signals, one or both of the devices 31A, 31B can be arranged to send change signals in correspondence to its own changes in position in the media stream (other than resulting from normal progression therethrough) and/or changes in its own progression mode through the media stream (such as, without limitation, rewind and fast forward), these change signals including an indication of the new position/progression mode to be adopted by the receiving device.
Whilst coordinated consumption could be effected in the above-described manner for static devices separate by large distances, it is expected that the above-described methods are most suitable where at least one of the devices is a mobile one and the devices have been brought into close proximity.
It will be appreciated that many other variants are possible to the above described arrangements. For example, and as already noted, the distribution of functionality between mobile devices and the service system is not limited to the distributions described above since the availability of communication resources makes it possible to place functionality where most suitable from technical and commercial considerations. Furthermore, in the foregoing reference to a mobile device is not to be construed as requiring functionality housed in a single unit and the functionality associated with a mobile device can be provided by a local aggregation of units.
It will be appreciated that the above described methods and arrangements for coordinating the consumption of streamed media objects are not limited to situations where the media objects are associated with a currently visited space.
Number | Date | Country | Kind |
---|---|---|---|
0218188.1 | Aug 2002 | GB | national |
0223135.5 | Oct 2002 | GB | national |