Patent Application
20080074542
The foregoing summary, as well as the following detailed description of preferred embodiments of the invention, will be better understood when read in conjunction with the accompanying drawings. For the purpose of illustrating the invention, there is shown in the drawings embodiments which are presently preferred. It should be understood, however, that the invention is not limited to the precise arrangement shown. In the drawings:
While this invention is susceptible of embodiments in many different forms, this specification and the accompanying drawings disclose only some forms as examples of the use of the invention. The invention is not intended to be limited to the embodiments so described, and the scope of the invention will be pointed out in the appended claims.
The preferred embodiment of the embedded multimedia system according to the present invention is described below with a specific application to a variable length coding scheme. The multimedia system may employ a media play back application. The audio decoder may be compatible with windows media audio, an advanced audio coding (AAC) decoder, an AAC Plus decoder, an enhanced AAC plus decoder (eAAC+), MP3, and Real Audio bitstream formats. However, it will be appreciated by those of ordinary skill in the art that the present invention is also well adapted for other types of audio decoders including for Game Audio applications.
Referring now to the drawings in detail, wherein like numerals are used to indicate like elements throughout, there is shown in FIGS. 1 and 2A-2B an embodiment of an embedded multimedia system, generally designated at 10, according to the present invention.
In general, the embedded multimedia system 10 includes a plurality of media outputs such as audio through speaker 44 and video through video display 46. The system 10 further outputs a playback time display 48 (shown in phantom) which may be integrated in or separate from the video display 46. While the embedded multimedia system 10 illustrated includes both audio and video, any one user-selected program may include only audio or both audio and video.
In the embedded multimedia system 10, the host microprocessor 12 includes program instructions which assemble a packed audio bitstream in a bitstream assembling module 14 and sends the packed audio bitstream to the audio decoder 32 in the DSP 30, as best seen in
In the exemplary embodiment, the predetermined audio playback times for the frames FR1, FR2 . . . FRX of a program are denoted as 24A, 24B, 24X. Furthermore, the LUT 22 has a predetermined byte count 26A, 26B, . . . , 26X associated with each frame's predetermined audio playback time 24A, 24B, . . . 24X.
The host microprocessor 12 further includes program instructions operable to function as an error-robust playback time stamp report cross referencing module 16 which cross references the stored predetermined audio playback time 24A, 24B . . . 24X for a particular frame in the LUT 22. Moreover for resynchronization operations, the host microprocessor 12 further includes program instruction operable as a resynchronization evaluation module 18 and a resynchronization module 20.
The audio decoder 32 includes program instructions operable to function as an error-robust audio playback time stamp report generator 40. The error-robust audio playback time stamp report generator 40 includes a report interval comparer 34, sample counter 36, and a byte counter 38, as best seen in
The report interval comparer 34 compares a call back interval 54 with time so that the error-robust audio playback time stamp report generator 40 can send the report. The report interval comparer 34 may receive and extract one or more of a call back intervals 54 stored in shared memory 50, the call back interval 54 being written by the host microprocessor 12. The audio decoder 32 can use the call back interval for use throughout the program.
The error-robust audio playback time stamp report generator 40 creates an error-robust audio playback time stamp report which is sent to the host microprocessor 12. The error-robust audio playback time stamp report includes a sample count from the sample counter 36 and the byte count from the byte counter 38.
In operation, the host microprocessor 12 will set and/or send a call back interval 54 in terms of a number (N) of audio samples played out to instruct the audio decoder 32 as to how frequently the error-robust audio playback time stamp report should be sent back to the host microprocessor 12. The call back interval 54 can be used to assist the host microprocessor 12 in constructing the LUT 22 by the LUT constructing module 15. The audio decoder 32 extracts the call back interval 54 from the shared memory 50. Alternately, the call back interval 54 may be sent in a host command to the DSP 30. Furthermore, the audio decoder 32 keeps track of how many bytes of the bitstream are consumed in byte counter 38, and how many samples are decoded/sent out to the speaker 44 via the sample counter 36. The error-robust audio playback time stamp report generator 40 reports back both the sample count of the sample counter 36 and the byte count of byte counter 38 at the requested call back interval 54 or integer multiples of the call back interval 54.
The host microprocessor 12 maintains a look up table (LUT) 22 that maps the number of bitstream bytes (byte count) 26A, 26B, . . . , 26X with the predetermine audio playback time 24A, 24B, . . . , 24X from the start of the playback. When the host microprocessor 12 receives the error-robust audio playback time stamp report, the host microprocessor 12 recognizes that the number of samples may not be an accurate indication of timing due to the possibility of at least one bitstream error. Therefore, the host microprocessor 12 uses the number of bytes consumed (byte count) from the report to index into the LUT 22 to find the predetermined audio playback time 24A, 24B, . . . , 24X. Preferably, the “index into the LUT 22” should use a closest match search to find the entry in the LUT 22 that is the closest to the byte count sent in the report, since any errors in the packed audio bitstream may result in the byte count being temporarily not aligned with a frame boundary.
With the byte count and sample count known, the host microprocessor 12 can determine where the audio is in playout and syncs up at least one of the audio, the video to associated with the audio, and playback time display.
When a command for the media (audio or video) playback to rewind, fast forward or stop is received, the host microprocessor 12 instructs the audio decoder 32 to do so by sending a corresponding command. The audio decoder 32 will reset the sample and byte counters 36 and 38. Furthermore, the host microprocessor 12 will re-construct the LUT 22 via the LUT constructing module 15. Sample and byte commands may be used to reset the sample and byte counters 36 and 38 to zero. Alternately, the sample and byte commands may simply advance forward or backward, the sample and byte counters 36 and 38.
The host microprocessor 12 and/or DSP 30 know when to reset the byte/sample counter based on the determinations made at steps S172A, 172B, S172C of
Various bitstream error detecting and handling schemes are available in audio decoders. The audio decoder 32 could either silence out the output or conceal the output pulse code modulation (PCM) when it detects or hits an error bitstream. Since audio decoding has been well established no further explanation of decoding and error handling are set forth.
The error robust audio playback time stamp reporting method 100 and multimedia system 10 take advantage of the fact that the bit stream length remains the same even under hostile channel conditions. The host microprocessor 12 uses the audio byte count to cross reference the predetermined audio playback time through the packed bitstream position at an audio bitstream frame boundary and therefore can determine accurate “actual” audio playback time, regardless of any errors in the bitstream such as for resynchronization of system 10.
Referring now to
Returning again to step S106, if the determination at step S106 is “YES” (meaning an error has been detected), then step S106 is followed by step S108 where the error is handled using known error correction and handling techniques suitable for the error detected. Step S108 is also followed by step S109 where the samples which are played out at step S110 (step S110 follows step S109) are counted.
As can be appreciated, steps S102, S106, and S108 are part of the decoding process where the audio decoder 32 decodes and corrects or handles bit errors in the received bitstream before being playout (step S110) through the speaker 44.
Returning again to step S109, step S109 is followed by step S112 to determine if the call back interval 54 or an integer multiple of the call back interval 54 has been reached. The call back interval 54 is a function of the sample count. If the determination is “NO,” step S112 returns to step S109. If the determination at step S112 is “YES,” step S112 is followed by step S114 where the error-robust audio playback time stamp report with the byte and sample counts are sent to the host microprocessor 12. Steps S114 is followed by step S116 where a determination is made whether there is any more programming. If the determination at step S120 is “YES,” then step S120 returns to step S102. Otherwise, if the determination is “NO” at step S120, then the method 100 ends.
Referring now to
Step S154 is followed by S156 where the actual audio playback time is calculated using the byte count and the cross referenced predetermined audio playback time. Step S156 is followed by step S158 where a determination is made whether resynchronization is needed based on the result of the calculation in step S156. If the determination is “NO,” the method 150 ends. Steps S156 and S158 are performed by the resynchronization evaluation module 18.
On the other hand, if the determination at step S158 is “YES,” step S158 is followed by steps S160 and S162. At step S162, the resynchronization module 20 will synchronize the playback time display. At step S160, a determination is made whether the program also includes video. If the determination is “NO,” step S160 is followed by step S164 where the audio-audio sync is synchronized. On the other hand, if the determination is “YES” at step S160, the program includes both video and audio. Therefore, the audio-video sync needs to be synchronized at step S166. Steps S162, S164 and S166 end the resynchronization method 150. If resynchronization is done with insertion or delaying of a video frame or other manipulation of the video, the LUT 22 does not need to be modified. Thus, the resynchronization at any point in time of the video would realign the LUT 22 so that the next reporting of the sample count and the byte count would be aligned with the LUT 22 (assuming no error occurred). On the other hand, if resynchronization required an audio frame to be dropped or otherwise manipulated, the LUT 22 may require reconstruction.
The embedded multimedia system 10 may require the synchronization information for other media applications not mentioned above since to mention each specific application would be prohibitive.
Referring now to
Steps S172A, 172B, and/or 172C are followed by step S174 where the host microprocessor 12 constructs or re-constructs the LUT 22 with a new byte count S174 and the predetermined audio playback time 24A, 24B, . . . , 24X associated therewith. Step S174 is followed by step S176 (shown in phantom) where the sample counter 36 is reset. Step S176 (shown in phantom) is followed by step S178 (shown in phantom) where the byte counter 38 is reset. The LUT 22 is constructed using a new or reset byte count. In one embodiment, the sample counter 36 and byte counter 38 may be reset to zero (0) or some other number determined by the host microprocessor 12 and/or DSP 30. Step S178 (shown in phantom) is followed by optional step S180 (shown in phantom) where a call back interval 54 or counter for the call back interval 54 is set or reset. The call back interval 54 can be sent as deemed appropriate. For example, the call back interval 54 at step S180 may only be set when a new program is started for playback and not at other times.
The steps S176 and S178 are shown in phantom to indicate, that as the result of the host microprocessor 12 receiving a user input for a new program selection, fast-forward or rewind, the interface between the host microprocessor 12 and the DSP 30 would cause the sample counter 36, the byte counter 38 to be set or reset. Likewise, the call back interval 54 may be set or reset. Moreover, the resetting of the sample counter 36 and the byte counter 38 may also be done automatically by the DSP 30.
In the embodiment of
Referring now to
In view of the above, the present invention provides an error robust reporting method and embedded multimedia system 10 which achieves accurate time stamp reporting back to the host microprocessor for system synchronization. The method and system 10 also compensate for the cumulative nature of error in an audio bitstream to synchronize the system.
It will be appreciated by those of ordinary skill in the art that by the embedded multimedia system, method and program instructions disclosed herein, the error robust reporting achieves accurate time stamp reporting and compensates for the cumulative nature of errors in an audio bitstream. The error robust reporting also compensates for the cumulative nature of errors when the protocol does not require re-transmission. For example, the protocol may be the UDP Lite or other “best-effort” (un-reliable) protocols. The error robust reporting also allows improved synchronization.
The foregoing description of the embodiments of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention. The embodiments were chosen and described in order to explain the principles of the invention and its practical application to enable one skilled in the art to utilize the invention in various embodiments and with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims appended hereto, and their equivalents.
