This application is related to Ser. No. 10/549,912 filed Sep. 20, 2005, naming Messrs. Alvarez-Arevalo, Turnbull and Walker as co-inventors.
This application is the US national phase of international application PCT/GB2005/001011 filed 16 Mar. 2005 which designated the U.S. and claims benefit of GB 0406901.9, dated 26 Mar. 2004, the entire content of which is hereby incorporated by reference.
The present invention is concerned with methods and apparatus for transmitting recorded material, such as video, audio of other material to be played in real time, over a network.
According to one aspect of the present invention, there is provided a method of transmitting a recording comprising:
In another aspect, the invention provides a method of transmitting a recording comprising:
Further aspects of the invention are set out in the claims
Some embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:
In
By “bit-rate” here is meant the bit-rate generated by the original encoder and consumed by the ultimate decoder; in general, this is not the same as the rate at which the streamer actually transmits, which will be referred to as the transmitting bit-rate. It should also be noted that these files are generated at a variable bit-rate (VBR)—that is, the number of bits generated for any particular frame of the video depends on the picture content. Consequently, references to bit-rate refer to the average bit-rate.
The server has a transmitter 12 which serves to output data via a network 2 to a terminal 3. The transmitter is conventional, perhaps operating with a well known protocol such as TCP/IP. A control unit 13 serves in conventional manner to receive requests from the terminal for delivery of a particular sequence, and to read packets of data from the store 11 for sending to the transmitter 12 as and when the transmitter is able to receive them. Here it is assumed that the data are read out as discrete packets, often one packet per frame of video, though the possibility of generating more than one packet for a single frame is not excluded. (Whilst is in principle possible for a single packet to contain data for more than one frame, this is not usually of much interest in practice).
Note that these packets are not necessarily related to any packet structure used on the network 2.
The terminal 3 has a receiver 31, a buffer 32, and a decoder 33.
Some networks (including TCP/IP networks) have the characteristic that the available transmitting data rate fluctuates according to the degree of loading on the network.
Some theoretical discussion is in order at this point.
As shown in
In a simple case, each packet corresponds to one frame, and the time-stamps ti increase monotonically, that is, ti+1>ti for all i. If however a frame can give rise to two or more packets (each with the same ti) then ti+1≧ti. If frames can run out of capture-and-display sequence (as in MPEG) then the ti do not increase monotonically. Also, in practice, some frames may be dropped, so that there will be no frame for a particular value of ti.
These times are relative. Suppose the receiver has received packet 0 and starts decoding packet 0 at time tref+t0. At “time now” of tref+tg the receiver has received packet tg (and possibly more packets too) and has just started to decode packet g.
Packets g to h−1 are in the buffer. Note that (in the simple case) if h=g+1 then the buffer contains packet g only. At time tref+tj the decoder is required to start decoding packet j. Therefore, at that time tref+tj the decoder will need to have received all packets up to and including packet j.
The time available from now up to
tref+tj is (tref+tj)−(tref+tg)=tj−tg. (1)
The data to be sent in that time are that for packets h to j, viz.
which at a transmitting rate R will require a transmission duration
This is possible only if this transmission duration is less than or equal to the time available, i.e. when the currently available transmitting rate R satisfies the inequality
Note that this is the condition for satisfactory reception and decoding of packet j: satisfactory transmission of the whole of the remaining sequence requires that this condition be satisfied for all j=h . . . N−1.
For reasons that will become apparent, we rewrite Equation (4) as:
Note that
Also, we define Δεi=(bi/R)−Δti
Note that th−1−tg is the difference between the time-stamp of the most recently received packet in the buffer and the time stamp of the least recently received packet in the buffer—i.e. the one that we have just started to decode.
Then the condition is
For a successful transmission up to the last packet N−1, this condition must be satisfied for any possible j, viz.
The left-hand side of Equation (7) represents the maximum timing error that may occur from the transmission of packet h up to the end of the sequence, and the condition states, in effect that this error must not exceed the ability of the receiver buffer to accommodate it, given its current contents. For convenience, we will label the left-hand side of Equation (7) as Th—i.e.
So that Equation (7) may be written as
Th≦th−1−tg (9)
Consider the situation at time tg=t0, that is, when the decoder is to commence decoding of the first packet. In the general case, the above condition will not be satisfied when there is only one packet in the buffer (h=1). The receiver waits for the buffer contents to reach a satisfactory level before it commences decoding. Using the above condition, it becomes apparent that the receiver should wait at least until the buffer contains packet H−1 where H is the smallest value of h for which the condition
Th≦th−1−t0 (10)
is satisfied.
In this embodiment of the invention, one of the functions of the control unit 13 is that, each time it sends a packet to the transmitter 12, it evaluates the test embodied in Equation 10.
If this test is not passed, the packet counter is incremented at 106 and control returns to step 102 where, as soon as the transmitter is ready to accept it, a further packet is read out and transmitted. If the test is passed, then it is known that the receiver is safe to begin decoding as soon as it has received this packet. Therefore at step 105 the control unit sends to the transmitter a “start” message to be sent to the receiver. When the receiver receives this start message, it begins decoding. If there is any possibility of messages being received in a different order from that in which they were sent, then the start message should contain the packet index n so that the receiver may check that packet n has actually been received before it commences decoding. Alternatively, the transmitter could send values of Tn+1 to the receiver, and the receiver itself performs the test.
Following the sending of the “start” message, the packet counter is incremented at 107 and another frame transmitted at 108: these steps are repeated until the end of the file is reached, this being recognised at 109 and the process terminates at 110.
The preceding description assumes that the control unit performs this calculation each time it sends a packet to the transmitter, which is computationally quite intensive. An alternative is to perform the calculation less often, perhaps once every five packets, which reduces the amount of computation but may result in the buffering of more frames than is necessary.
Another alternative is to complete the computation as soon as it is able to do so (i.e. without waiting for the next packet) and then send a start message (with starting packet number) to the receiver. A yet further alternative is to perform the computation before transmitting any packets at all. Once the value of h is determined, we then transmit packets 0 to h−1 in reverse order (packet h−1, packet h−2, . . . packet 0). In this case it ceases to be necessary to transmit an explicit “start” command. Standard receivers that support UDP transport protocol are able to reorder packets, and will automatically wait until packet 0 has arrived before commencing decoding. In fact, it is sufficient that packet 0 is withheld until after packets 1 to h−1 have been sent (whose order is immaterial).
This however precludes the possibility of taking into account changes in the transmitting data rate R during the waiting period, and is therefore satisfactory only if such changes are not expected.
Observe (by inspection of Equation (3)) that the significance of the rate R is in calculating the time taken to send packets h to j. Therefore the actual rate used to transmit packets 0 to h−1 is of no consequence as it does not affect the result.
Another attractive option is to perform as much as possible of the computation in advance. If a system in which only one value of R is possible, or permitted, then the computation of Tn+1 at step 103 and the test of step 104 can be performed in advance for each frame up to the point where the test is passed, and the result recorded in the file, for example by recording the corresponding value of n in a separate field at the start of the file, or by attaching a special flag to frame n itself. Thus in
R1=0.5RA
R2=0.7RA
R3=RA
R4=1.3RA
R5=2RA
So each packet h has these five precalculated values of Th stored in it. If required (for the purposes to be discussed below) one may also store the relative time position at which the maximum in Equation (8)) occurs, that is,
Δth max=tj max−th where tj max is the value of j in Equation 8 for which Th is obtained.
In this case the flowchart proceeds as follows following transmission of frame n:
112: interrogate the transmitter 12 to determine the available transmitting rate R;
103A:EITHER—in the event that R corresponds to one of the rates for which Th has been precalculated—read this value from the store;
OR—in the event that R does not so correspond, read from the store the value of Th (and, if required, th max) that correspond to the highest one (R−) of the rates R1 . . . R5 that is less than the actual value of R, and estimate Th from it;
104A: Apply the test Tn+1+Δ≦tn−t0, where Δ is a fixed safety margin;
Continue as before.
The estimate of Th could be performed simply by using the value Th− associated with R−; this would work, but since it would overestimate Th it would result, at times, in the receiver waiting longer than necessary. Another option would be by linear (or other) interpolation between the values of Th stored for the two values of R1 . . . R5 each side of the actual value R. However, our preferred approach is to calculate an estimate according to:
Where R− is the highest one of the rates R1 . . . R5 that is less than the actual value of R, Ti− is the precalculated Th for this rate, Δti max− is the time from ti at which Ti− is obtained (i.e. is the accompanying value of Δth max−). In the event that this method returns a negative value, we set it to zero.
Note that this is only an estimate, as Th is a nonlinear function of rate. However with this method Ti′ is always higher than the true value and automatically provides a safety margin (so that the margin Δ shown above may be omitted).
Note that these equations are valid for the situation where the encoding process generates two or more packets (with equal ti) for one frame, and for the situation encountered in MPEG with bidirectional prediction where the frames are transmitted in the order in which they need to be decoded, rather than in order of ascending ti.
We will now describe an alternative embodiment in which the mathematics is converted into an equivalent form which however, rather than performing the calculations for each packet individually, makes use of calculations already made for a preceding packet. Recalling Equation (8):
which may be rewritten
Provided that Th+1≧0, which will be true at the beginning of the file, this becomes
Th=Th+1+Δεh (14)
Or generally
Consider the test
Th≦th−1−t0
which may be written
if a=0, this becomes
Noting that t−1 is a meaningless quantity (appearing on both sides on the inequality) so that it can be given any value, it is convenient to define t−1 as equal to t0, whence we obtain
Thus the test of Equation (10)
Th≦th−1−t0
could instead be written
Then the first test (h=1) is Test 1:
Or, if we define
the first test is Z1≦0?
The second test is Z2≦0
The xth test is Zx≦0
But
So each test can update the previous value of Z, as shown in the flowchart of
The step 201 of calculating T0 could be done in advance and the values stored. This procedure could of course be adapted, in a similar manner to that previously described, to accommodate different values of R.
It is not essential that this process begin with T0. One could start with T1 (in which case the first test is T1≦0?) or, if one chooses always to buffer at least two (or more) packets one could start with T2, etc.
Although the example given is for encoded video, the same method can be applied to encoded audio or indeed any other material that is to be played in real time.
If desired, in multiple-rate systems, these methods may be used in combination with the rate-switching method described in our international patent application WO04/086721 (corresponding to U.S. application Ser. No. 10/549,912).
Number | Date | Country | Kind |
---|---|---|---|
0406901.9 | Mar 2004 | GB | national |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/GB2005/001011 | 3/16/2005 | WO | 00 | 9/21/2006 |
Publishing Document | Publishing Date | Country | Kind |
---|---|---|---|
WO2005/093995 | 10/6/2005 | WO | A |
Number | Name | Date | Kind |
---|---|---|---|
4419699 | Christopher et al. | Dec 1983 | A |
5025458 | Casper et al. | Jun 1991 | A |
5430485 | Lankford et al. | Jul 1995 | A |
5534937 | Zhu et al. | Jul 1996 | A |
5598352 | Rosenau et al. | Jan 1997 | A |
5768527 | Zhu et al. | Jun 1998 | A |
5874997 | Haigh | Feb 1999 | A |
5923655 | Veschi et al. | Jul 1999 | A |
5949410 | Fung | Sep 1999 | A |
5953350 | Higgins | Sep 1999 | A |
6014694 | Aharoni et al. | Jan 2000 | A |
6016307 | Kaplan et al. | Jan 2000 | A |
6085221 | Graf | Jul 2000 | A |
6101221 | Varanasi et al. | Aug 2000 | A |
6130987 | Tanaka | Oct 2000 | A |
6148135 | Suzuki | Nov 2000 | A |
6169843 | Lenihan et al. | Jan 2001 | B1 |
6195368 | Gratacap | Feb 2001 | B1 |
6223211 | Hamilton et al. | Apr 2001 | B1 |
6332157 | Mighdoll et al. | Dec 2001 | B1 |
6366614 | Pian et al. | Apr 2002 | B1 |
6397251 | Graf | May 2002 | B1 |
6438317 | Imahashi et al. | Aug 2002 | B1 |
6452922 | Ho | Sep 2002 | B1 |
6453112 | Imahashi et al. | Sep 2002 | B2 |
6502125 | Kenner et al. | Dec 2002 | B1 |
6560334 | Mullaney et al. | May 2003 | B1 |
6678332 | Gardere et al. | Jan 2004 | B1 |
6704288 | Dziekan et al. | Mar 2004 | B1 |
6771703 | Oguz et al. | Aug 2004 | B1 |
6792047 | Bixby et al. | Sep 2004 | B1 |
6931071 | Haddad et al. | Aug 2005 | B2 |
6976208 | Kim et al. | Dec 2005 | B1 |
7016970 | Harumoto et al. | Mar 2006 | B2 |
7096481 | Forecast et al. | Aug 2006 | B1 |
7111061 | Leighton et al. | Sep 2006 | B2 |
7333721 | Maehashi et al. | Feb 2008 | B2 |
7340505 | Lisiecki et al. | Mar 2008 | B2 |
7471874 | Kanemaru et al. | Dec 2008 | B2 |
7558869 | Leon et al. | Jul 2009 | B2 |
7620137 | Lottis et al. | Nov 2009 | B2 |
20020031120 | Rakib | Mar 2002 | A1 |
20020100052 | Daniels | Jul 2002 | A1 |
20020102978 | Yahagi | Aug 2002 | A1 |
20020136205 | Sasaki | Sep 2002 | A1 |
20030002482 | Kubler et al. | Jan 2003 | A1 |
20030053416 | Ribas-Corbera et al. | Mar 2003 | A1 |
20030145007 | Kenner et al. | Jul 2003 | A1 |
20030169777 | Fuse | Sep 2003 | A1 |
20030233666 | Onomatsu et al. | Dec 2003 | A1 |
20040141731 | Ishioka et al. | Jul 2004 | A1 |
20050071876 | van Beek | Mar 2005 | A1 |
20050117891 | Suzuki | Jun 2005 | A1 |
Number | Date | Country |
---|---|---|
1426235 | Jun 2003 | CN |
10125017 | Dec 2002 | DE |
0817488 | Jan 1998 | EP |
0868084 | Sep 1998 | EP |
0966175 | Dec 1999 | EP |
WO 9522233 | Aug 1995 | WO |
WO 9965026 | Dec 1999 | WO |
WO 0189142 | Nov 2001 | WO |
0249343 | Jun 2002 | WO |
WO 0249343 | Jun 2002 | WO |
WO 02095637 | Nov 2002 | WO |
Number | Date | Country | |
---|---|---|---|
20080025340 A1 | Jan 2008 | US |