Backfill of video stream

Abstract

A data streaming system includes one or more processors, memory, and a circular buffer stored in the memory. The system includes a media loader process that retrieves frames from a media source and writes the retrieved frames in chronological order to the circular buffer. The system includes a forward dispatch process that transmits frames from the circular buffer to a destination server in chronological order in real-time. The system also includes a backfill dispatch process that executes in response to receiving a backfill request from the destination server, transmitting frames from the circular buffer to the destination server in reverse chronological order.

Description

BACKGROUND

In recent years, network-based video streaming devices such as Wi-Fi-enabled security monitor cameras have been developed which streams recorded data in real time to servers via network connections. Network connections, in particular wireless connections (e.g., Wi-Fi), can be easily interrupted. When the network connection between a streaming device and the server is disrupted, there can be loss of data. Solutions are needed to recover the lost data.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.

FIG. 1 is a functional diagram illustrating a system for streaming media data in accordance with some embodiments.

FIG. 2 is a block diagram illustrating an embodiment of a network-enabled media capturing and streaming platform,

FIG. 3A is a diagram illustrating an example representation of the available video data.

FIG. 3B illustrates an example where backfill is taking place. In this case, data is resent in the opposite direction of the time axis along arrow 300.

FIG. 3C is a diagram illustrating the effect of a process that sends data in the same direction as the time axis.

FIG. 4 is a flowchart illustrating an embodiment of a process for back filling streaming media data.

FIG. 5 is a flowchart illustrating an embodiment of a process for detecting that a portion of the first media data stream needs to be resent.

FIG. 6 is a diagram illustrating an example implementation of an embodiment of a media data streaming system.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.

Data streaming is described. In some embodiments, a first media data stream is sent to a destination. In the event that it is detected that a portion of the first media data stream needs to be resent (e.g., the portion is not received by the destination due to network failure or the like), buffered data is sent in a second media data stream, where the buffered data. corresponds to at least some of the missing portion. The buffered data is backfilled. In other words, in the second media data stream, data that is obtained (e.g., recorded) more recently is sent prior to media data that is obtained less recently. Although video streaming is described in the following examples for purposes of illustration, the technique is also applicable to other types of streaming media such as audio.

FIG. 1 is a functional diagram illustrating a system for streaming media data in accordance with some embodiments. An example of system 102 is the Dropcam® Wi-Fi Internet Video Camera System 100, which includes various subsystems as described below, includes at least one microprocessor subsystem (also referred to as a processor or a central processing unit (CPU)) 102. For example, processor 102 can be implemented by a single-chip processor or by multiple processors. In some embodiments, processor 102 is a general purpose digital processor that controls the operation of system 100. Using instructions retrieved from memory 110, the processor 102 controls the reception and manipulation of input data (e.g., recorded media data), and the output of data via an I/O interface (e.g., network interface 116)

Processor 102 is coupled bi-directionally with memory 110, which can include primary storages such as a random access memory (RAM) and/or a read-only memory (ROM). As is well known in the art, primary storage can be used as a general storage area and as scratch-pad memory, and can also be used to store input data and processed data. Primary storage can also store programming instructions and data, in the form of data objects and text objects, in addition to other data and instructions for processes operating on processor 102. Also as is well known in the art, primary storage typically includes basic operating instructions, program code, data and objects used by the processor 102 to perform its functions (e.g., programmed instructions). For example, memory 110 can include any suitable computer-readable storage media, described below, depending on whether, for example, data access needs to be bi-directional or uni-directional. For example, processor 102 can also directly and very rapidly retrieve and store frequently needed data in a cache memory (not shown).

An optional removable mass storage device 112 provides additional data storage capacity for the system 100, and is coupled either bi-directionally (read/write) or uni-directionally (read only) to processor 102. An optional fixed mass storage 120 can also, for example, provide additional data storage capacity. Examples of mass storage 120 include a hard disk drive, a flash drive, a solid state drive, or the like. It will be appreciated that the information retained within mass storage 112 and 120 can be incorporated, if needed, in standard fashion as part of memory 110 (e.g., RAM) as virtual memory.

In addition to providing processor 102 access to storage subsystems, bus 114 can also be used to provide access to other subsystems and devices. As shown, these can include a network interface (also referred to as the communication interface) 116, a microphone 106, a camera 104, as well as any optional auxiliary input/output interface devices such as a display 118, a keyboard and a pointing device, a sound card, speakers, (not shown) and other subsystems as needed.

The network interface 116 allows processor 102 to be coupled to another computer, computer network, or telecommunications network using a network connection (e.g., an Ethernet connection, a Wi-Fi connection, etc.). For example, through the network interface 116, the processor 102 can send streaming media data (including audio and/or video) to another device such as a media server. Information, often represented as a sequence of instructions to be executed on a processor, can be received from and outputted to another network. An interface card or similar device and appropriate software implemented by (e.g., executed/performed on) processor 102 can be used to connect the computer system 100 to an external network and transfer data according to standard protocols. For example, various process embodiments disclosed herein can be executed on processor 102, or can be performed over a direct connection such as a Universal Serial Bus (USB) interface or over a network such as the Internet, intranet networks, or local area networks, in conjunction with a remote processor that shares a portion of the processing. Additional mass storage devices (not shown) can also be connected to processor 102 through network interface 116.

In addition, various embodiments disclosed herein further relate to computer storage products with a computer readable medium that includes program code for performing various computer-implemented operations. The computer-readable medium is any data storage device that can store data which can thereafter be read by a computer system. Examples of computer-readable media include, but are not limited to, all the media mentioned above: magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks; magneto-optical media such as optical disks; and specially configured hardware devices such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and ROM and RAM devices. Examples of program code include both machine code, as produced, for example, by a compiler, or files containing higher level code (e.g., script) that can be executed using an interpreter.

The computer system shown in FIG. 1 is but an example of a computer system suitable for use with the various embodiments disclosed herein. Other computer systems suitable for such use can include additional or fewer subsystems. In addition, bus 114 is illustrative of any interconnection scheme serving to link the subsystems. Other computer architectures having different configurations of subsystems can also be utilized.

FIG. 2 is a block diagram illustrating an embodiment of a network-enabled media capturing and streaming platform. Device 202 can be implemented by a system such as 100 or by any other appropriate recording and streaming device. An example of device 202 is the Dropcam®, Wi-Fi Internet Video Camera. Device 202 includes one or more interfaces configured to capture media data (e.g., camera and/or microphone interfaces configured to capture video and/or audio), and record the captured media information to a local storage. The device includes one or more network interfaces that allow the device to be connected to a server 204 via a network 206 such as the Internet. Alternatively, device 202 can also connect to the server via an external host. For example, device 202 can be connected to a host via a USB, Bluetooth, or other local connections, and use the host as a proxy for communicating with the server over the network.

In some embodiments, device 202 is configured to continuously capture video and stream the video to server 204 via the network, using a protocol such as Real Time Streaming Protocol (RTSP), DropTalk, or any other appropriate streaming protocol. Server 204 (also referred to as a media server) buffers the received video stream, saves it to storage, and makes it available for viewing by a user. For example, the server can make the video available to a client device by implementing a client-server communications protocol such as Hypertext Transfer Protocol (HTTP), where the video is accessible at a given Universal Resource Locator (URL). user accesses the video by visiting the URL using a web browser or a standalone application running on a client device 208 (e.g., a personal computer, a smartphone, a tablet device, or the like). The user can view the video live, i.e., in real time as the video is captured and streamed to the server by the device, or view previously recorded video. A user interface with time indication (e.g., a scrubber bar) is provided for the user via the web browser or the standalone application to control the portion of the video to view.

In some embodiments, while the video is being captured, it is also buffered in a local storage space (e.g., a disk, a flash drive, or other memory device) that is a part of the device. The amount of video that is buffered depends on the quality of the video being stored and the amount of storage space available. In some embodiments, a few minutes or a few hours' worth of video is buffered. In some embodiments, the storage space is implemented as a circular buffer in which the oldest video content is overwritten by the newest video content when the storage space is filled up and additional space is needed.

If there is disruption in the device's video stream to the server, some of the video data will not be correctly received by the server. For example, if the network connection is disrupted temporarily, the device will not be able to access the server via the Internet, causing disruption to the video stream. FIG. 3A is a diagram illustrating an example representation of the available video data. Information conveyed in the example shown may be implemented as data structures or as a user interface displayed to the user. The x-axis represents time. The shaded portions represent available video data, and in particular available video data on the server. In some embodiments, the video data is stored as a Motion Picture Experts Group (MPEG) stream comprising a plurality of video frames with timestamps. Between time t1 and t2, there is a gap where the video stream is disrupted (possibly due to a network outage or the like). Without any backfill, the user will simply not be able to view the video captured between t1 and t2. To overcome the problem, in some embodiments, the video that is buffered on the camera itself is used to backfill the gap once the network connection is restored.

In this example, the video stream includes timestamps so that the data on the device and the data on the server can be synchronized. After the disruption has occurred and the connection is restored, the server will request the device to reseed the missing portion of video in an attempt to backfill the video. FIG. 3B illustrates an example where backfill is taking place. In this case, data is resent in the opposite direction of the time axis along arrow 300. In other words, data that was recorded more recently is resent before data that was recorded earlier. For example, video frame 302 (with a timestarnp of 00:00:30) is resent before video frame 304 (with a timestamp of 00:00:29), which in turn is resent before video frame 306 (with a timestamp of 00:00:28).

One reason to backfill in the opposite direction of the time axis is that the camera is still recording live video and may be overwriting old data in the buffer. A backfill process that goes in the opposite direction of the time axis means that there may be a gap of lost data at the beginning of the disruption period (e.g., portion 308) that has already been overwritten by new data, but the portion of missing data that is resent would be continuous. If the process were to resend data in the same direction as the time axis, there can be portions of the old data that are not yet uploaded on to the server but is overwritten by new data and becomes irretrievably lost. Depending on the speed of the overwrite and the speed of the upload, there can be multiple such missing data portions, resulting in a backfilled stream that is choppy and is unpleasant to the viewer.

FIG. 3C is a diagram illustrating the effect of a process that sends data in the same direction as the time axis. As shown, recorded data is written continuously to a circular buffer, and old data can be overwritten by new data. To resend the portion of data between t1-t2, data is resent in the same direction as the time axis. The portions shown in white correspond to portions of the buffered data that have been overwritten by newer data and therefore cannot be resent. Hashed portions such as 312 are resent before the buffer is overwritten by new data. The overall effect of the resent stream is choppy because of the discontinuities.

FIG. 4 is a flowchart illustrating an embodiment of a process for back filling streaming media data. Process 400 can be performed on a device such as 100. At 402, a first media data stream is sent to a destination such as a video server. In some embodiments, the first media data stream includes a recorded video stream. In some embodiments, the recorded video stream is captured in real-time by the device.

The first media data stream is written to a local storage (e.g., a mass storage component such as 112 or 120 of FIG. I) continuously as the media data is captured. As will be described in greater detail below, in some embodiments, the local storage is implemented using a circular buffer.

At 404, it is detected that a portion of the first media data stream needs to be resent. Details of an example process for how to detect the missing portion is described below in connection with FIG. 5.

At 406, a second media data stream that comprises media data corresponding to at least some of the missing portion is sent to the destination. In some embodiments, once the processor on the device detects that a portion of the media stream needs to be resent, the processor notifies the communications interface and causes appropriate data to be transferred from a storage buffer to the communications interface, which sends the missing portion to the destination. Media data that is obtained more recently is sent prior to media data that is obtained less recently. Referring to FIG. 3B, data frame 302, which is obtained more recently than data frame 304, is sent before data frame 304.

In some embodiments, the second media data stream is generated by reading the recorded media data from the local storage. The device continues to send media data that corresponds to the missing portion until either the entire missing portion has been sent, or newer data that is recorded after the end of the disrupted period (e.g., data recorded after time t2) is encountered.

Upon receiving the resent media data, the server stores the media data. Since the media data is sequenced according to the frame identifiers, the missing portion can now be played back properly.

FIG. 5 is a flowchart illustrating an embodiment of a process for detecting that a portion of the first media data stream needs to be resent. Process 500 can be used to implement 404 of process 400.

In some embodiments, the destination sends acknowledgments to the device upon successfully receiving data in the first media data stream from the device. Depending on the communication protocol used between the device and the destination, the acknowledgments can have different formats and the interval at which the acknowledgements are sent can differ. For example, in some embodiments, the device sends data in blocks. Each block comprises a set of media data frames and some identification information e.g., a block identifier, the range of frame identifiers for the set of media data frames, the range of timestamps associated with the set of media data frames, or any other appropriate information that identifies the block or its contents). Accordingly, the acknowledgement also includes the identification information to indicate that the block or the set of media data frames that is successfully received. At 502, one or more acknowledgements are received from the server.

When the communication channel between the device and the server is interrupted, the server will not receive certain data and therefore will not send acknowledgements with respect to the data. In some embodiments, lower layer (e.g., the TCP layer) software handles the process of reestablishing the connection. Upon being notified that the communication channel has been reestablished, the server sends a backfill request to the device, instructing the device to check for any data that has not been successfully acknowledged and therefore should be resent. At 504, the backfill request from the server is received by the device.

At 506, the device checks its records of media data that has been sent and corresponding acknowledgements to determine which data blocks or data frames have not been acknowledged. For example, if the device has sent media data frames with identifiers ranging from 1-200 to the server but has only received acknowledgements for media data frames 1-150 when the backfill request is received, then the portion that needs to be resent includes frames 151-200, which will be sent in reverse order according to the frame identifiers.

FIG. 6 is a diagram illustrating an example implementation of an embodiment of a media data streaming system. System 600 implements its local storage buffer using a circular buffer 602. In the example shown, circular buffer 602 is a fixed-sized circular buffer storing non-uniformly sized data chunks (referred to as CircularChunkBuffer). Uniformly sized data chunks can be used in other embodiments. Each data chunk includes a set of frames. As shown, a media loader task 612 (implemented as a thread or a process executing on the processor of the device) retrieves frames from a media source (e.g., the camera interface) and places the frames into circular buffer 602. The direction in which the frames are written is referred to as the forward direction, and the opposite direction is referred to as the backward direction. Since a circular buffer does not have a beginning or an end, as the media loader task continuously writes frames to the circular buffer, old frames can be overwritten by new frames.

Data in circular buffer 602 is dispatched (i.e., transmitted) to the destination by a forward dispatch task 604 and a backfill dispatch task 606. Each task transmits a corresponding media data stream. The forward dispatch task is started as soon as the device is connected to a server, taking media data stored in circular buffer 602 in the forward direction according to time, and transmitting the data to the server. A forward cursor 608 that advances in the forward direction is used to track the position of the media data being transmitted and allow the forward dispatch task to transmit the appropriate media data frame. In some embodiments, once a frame is transmitted by the forward dispatch task, the forward cursor advances to the next frame in the forward direction.

Backfill dispatch task 606 is started when it is detected that a portion of the media data in the first media data stream sent by the forward dispatch task needs to be resent, Backfill dispatch task 606 transmits media data in circular buffer 602 in the backward direction, starting with the last frame that needs to be resent. A backfill cursor 610 that advances in the backward direction is used to track the position of the media data that needs to be resent. In some embodiments, once a frame is transmitted by the backfill dispatch task, the backward cursor advances to the next frame in the backward direction. In the example where frames 151-200 need to be resent, the backfill cursor points to frame 200 initially since frame 200 should be resent first. Once frame 200 is sent, the backfill cursor advances to the next frame 199, then 198, and so on until frame 151 is resent. If, however, certain old frames have been overwritten, then the backfill process stops when the last available frame that needs to be resent is resent. For example, if frames 151 and 152 have been overwritten by new frames 806 and 807, then the backfill cursor stops advancing when frame 153 has been sent and the backfill process is completed.

In some situations, the media data stream comprising backfilled data and the media data stream comprising live data can be transmitted contemporaneously. In some embodiments, the live video stream is given higher priority than the backfilled data stream over the transmission channel. For example, the live video stream will occupy as much bandwidth as needed if the available bandwidth exceeds the bandwidth of the live video stream or all the available bandwidth if the available bandwidth is less than the bandwidth of the live video stream; and buffered data is sent to the server using any remaining bandwidth. Alternatively, the live video stream and the buffered data can be configured to occupy some predefined proportion of the available bandwidth (e.g., the live stream occupies 80% of the bandwidth and the buffered data occupies 20%). In some embodiments, each stream is sent over a different TCP connection, and the TCP connection associated with the live data stream is assigned a higher priority than the TCP connection associated with the backfilled data stream. For example, the live data stream can be configured to “very high” priority and the buffered data can be configured to “medium” priority when their respective TCP connections are opened.

Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.

Claims

1. A camera system, comprising: one or more processors;memory having a limited capacity;a camera element configured to continuously capture video data in real-time and store the captured video in the memory;a microphone configured to continuously capture audio data in real-time and store the captured audio in the memory; anda communication interface;wherein the one or more processors are configured to: continuously capture video and audio data in real-time via the camera element and the microphone, respectively;while capturing the video and audio data in real-time, continuously store the captured video and audio data in the memory having the limited capacity;continuously send the continuously captured video and audio data in real-time to a remote server system via the communication interface for storage by the remote server system, wherein the video and audio data stored at the remote server system is subsequently accessible by a client access device; andwhile sending the continuously captured video and audio data in real time to the remote server system: determine that a portion of the video and audio data sent to the remote server system was not received by the remote server system, thereby creating a gap in the video and audio data stored at the remote server system and available for access by the client access device; andwhen at least some of the portion is still available in the memory having the limited capacity, resend the at least some of the portion from the memory to the remote server system via the communication interface to fill at least some of the gap.

2. The camera system of claim 1, wherein the client access device is configured to display video and audio data sent from the camera system to the remote server system and stored at the remote server system.

3. The camera system of claim 1, wherein the client access device is configured to display video and audio data sent from the camera system to the client access device in real time via the remote server system.

4. The camera system of claim 1, wherein resending the portion occurs concurrently while continuing to capture video and audio data at the camera element.

5. The camera system of claim 1, wherein resending the portion occurs concurrently while continuing to continuously send the video and audio data in real-time to the remote server system.

6. The camera system of claim 1, wherein segments of the captured audio and video are timestamped, and determining that a portion of the video and audio data sent to the remote server system has not been received by the remote server system includes determining which timestamped segments of the captured audio and video have not been acknowledged by the remote server system.

7. The camera system of claim 1, wherein determining that a portion of the video and audio data sent to the remote server system has not been received by the remote server system includes receiving a backfill request from the remote server system.

8. The camera system of claim 1, wherein data that was stored more recently is resent before data that was stored earlier.

9. The camera system of claim 1, wherein data that was stored earlier is resent before data that was stored more recently.

10. The camera system of claim 1, wherein the camera system comprises the remote server system.

11. A method, comprising: at a camera system having one or more processors, memory having a limited capacity, a camera element configured to continuously capture video data in real-time and store the captured video in the memory, a microphone configured to continuously capture audio data in real-time and store the captured audio in the memory, and a communication interface: continuously capturing video and audio data in real-time via the camera element and the microphone, respectively;while capturing the video and audio data in real-time, continuously storing the captured video and audio data in the memory having the limited capacity;continuously sending the continuously captured video and audio data in real-time to a remote server system via the communication interface for storage by the remote server system, wherein the video and audio data stored at the remote server system is subsequently accessible by a client access device; andwhile sending the continuously captured video and audio data in real time to the remote server system: determining that a portion of the video and audio data sent to the remote server system was not received by the remote server system, thereby creating a gap in the video and audio data stored at the remote server system and available for access by the client access device; andwhen at least some of the portion is still available in the memory having the limited capacity, resending the at least some of the portion from the memory to the remote server system via the communication interface to fill at least some of the gap.

12. The method of claim 11, wherein determining that a portion of the video and audio data sent to the remote server system has not been received by the remote server system includes receiving a backfill request from the remote server system.

13. The method of claim 11, wherein data that was stored more recently is resent before data that was stored earlier.

14. The method of claim 11, further comprising at the remote server system: receiving a stream of audio and video from the camera element;storing the received stream of audio and video data in memory at the remote server system; andproviding access to the received audio and video data to the client access device.

15. The method of claim 11, wherein the camera system comprises the remote server system, further comprising: receiving by the remote server system a stream of audio and video from the camera element;storing the received stream of audio and video data in memory at the remote server system; andproviding by the remote server system access to the received audio and video data to the client access device.

16. The method of claim 15, further comprising, at the remote server system: identifying the gap in the received audio and video; andin response to identifying the gap, sending a backfill request to the camera element.

17. A non-transitory computer readable storage medium storing one or more programs configured for execution by a camera system having one or more processors, memory having a limited capacity, a camera element configured to continuously capture video data in real-time and store the captured video in the memory, a microphone configured to continuously capture audio data in real-time and store the captured audio in the memory, and a communication interface, wherein the one or more programs comprise instructions for: continuously capturing video and audio data in real-time via the camera element and the microphone, respectively;while capturing the video and audio data in real-time, continuously storing the captured video and audio data in the memory having the limited capacity;continuously sending the continuously captured video and audio data in real-time to a remote server system via the communication interface for storage by the remote server system, wherein the video and audio data stored at the remote server system is subsequently accessible by a client access device; andwhile sending the continuously captured video and audio data in real time to the remote server system: determining that a portion of the video and audio data sent to the remote server system was not received by the remote server system, thereby creating a gap in the video and audio data stored at the remote server system and available for access by the client access device; andwhen at least some of the portion is still available in the memory having the limited capacity, resending the at least some of the portion from the memory to the remote server system via the communication interface to fill at least some of the gap.

18. The computer readable storage medium of claim 17, wherein resending the portion occurs concurrently while continuing to capture video and audio data at the camera element.

19. The computer readable storage medium of claim 17, wherein resending the portion occurs concurrently while continuing to send the video and audio data from the memory to the remote server system.

20. The computer readable storage medium of claim 17, wherein segments of the captured audio and video are timestamped, and determining that a portion of the video and audio data sent to the remote server system was not received by the remote server system includes determining which timestamped segments of the captured audio and video have not been acknowledged by the remote server system.