Patent Application
20190182304
Many people use electronic devices such as smartphones, tablets, personal computers, etc. to consume streaming data (e.g., multimedia data including audio, video, etc.). User devices may utilize various protocols to receive streamed data wirelessly. Such protocols may not include support for such streaming and may have payload sizes that are no efficient for such usage.
Streaming resources typically encode data for improved compression and the encoded packet size may vary depending on the particular streaming arrangement being used. The size of such encoded packets may not match the payload size of a particular communication protocol and thus may require fragmentation and assembly.
Such fragmentation and assembly typically requires a well-defined message structure that includes a message type and a sequence number. Such identifiers reduce the payload size that is available for data transfer.
Thus there exists a need for an efficient way to fragment and reassembly streaming data while minimizing payload size for identifying information.
Some embodiments may provide ways to optimize streaming content based on attributes of available communication channels. Such attributes, include message payload size, streaming payload size, etc. may be analyzed to generate a set of message structures that efficiently utilize the entire message payload size.
A maximum number of message types may be determined based on the number of bytes in a message payload. The number of message types may be used to determine the size needed for a message identification. Such a message identification may be associated with a message structure and a message sequence.
A roster of available message structures may be generated based on the size of the message payload, streaming payload, and message ID. Each message structure may include a message ID, one or more full streaming payloads, a secondary payload, and a tertiary payload (if necessary).
During streaming, incoming streaming data may be fragmented and linked according to the sizes specified by each message structure. The data may then be sent to a destination using the available message structures.
The destination may parse the incoming messages to determine a message ID, and thus, an associated structure. The destination may then fragment the incoming data based on the message structure and assemble complete streaming packets (or identify complete received packets).
The complete streaming packets may then be provided to a player or other appropriate resource for presentation to a user.
The preceding Summary is intended to serve as a brief introduction to various features of some exemplary embodiments. Other embodiments may be implemented in other specific forms without departing from the scope of the disclosure.
The exemplary features of the disclosure are set forth in the appended claims. However, for purpose of explanation, several embodiments are illustrated in the following drawings.
The following detailed description describes currently contemplated modes of carrying out exemplary embodiments. The description is not to be taken in a limiting sense, but is made merely for the purpose of illustrating the general principles of some embodiments, as the scope of the disclosure is best defined by the appended claims.
Various features are described below that can each be used independently of one another or in combination with other features. Broadly, some embodiments generally provide a universal messaging protocol for limited payload size.
A first exemplary embodiment may provide a streaming media system comprising: a source device that provides streaming content; and a destination device that receives streaming content from the source device across a communication channel, wherein: the streaming content is sent from the source device to the destination device using a message structure comprising packets having a first payload size, each packet includes an identifier, and each packet includes at least one streaming payload having a second payload size.
A second exemplary embodiment an automated method of generating a message structure comprising a sequence of messages, the method comprising: determining a first payload size associated with a communication channel between a source device and a destination device; determining a second payload size associated with an encoding algorithm used to provide streaming content from the source device to the destination device; determining a maximum number of message types based on the first payload size; and determining a size of a unique identifier associated with each message in the sequence of messages.
A third exemplary embodiment provides an automated method of providing streaming content, the method comprising: receiving streaming content at a source; identifying a message structure comprising a unique identifier, a location of a streaming payload, and a location of at least one partial payload; fragmenting the received data based on the message structure; generating an optimized packet using the fragmented data; and sending the optimized pack to a destination.
Several more detailed embodiments are described in the sections below. Section I provides a description of a system architecture of some embodiments. Section II then describes methods of operation used by some embodiments. Lastly, Section III describes a computer system which implements some of the embodiments.
The source 110 may be an electronic device such as a smartphone, tablet, personal computer, server, storage device, etc. The source may be able to access and/or otherwise communicate with local and/or remote devices such as servers, storages, etc. The source may provide streaming content to the destination device 120.
The destination 120 may be a smartphone, tablet, headphones, personal computer, etc. that may be able to provide streaming content such as audio, video, multimedia, graphics, games, voice over IP (VoIP), etc.
The communication pathway 130 may be a wired (e.g., Ethernet, USB, etc.) and/or wireless channel (e.g., Bluetooth, BLE, Wi-Fi, etc.). The channel may allow for two-way communication between the source 110 and the destination 120. In some embodiments, the channel 130 may allow communication among multiple sources 110 and/or multiple destinations 120.
In this example, the total message 205-250 size is twenty bytes, the streaming payload 270 size is ten bytes, and the message ID 260 size is one byte. The secondary payload 280 (if applicable), and tertiary payload 290 (if applicable) utilize the remaining nine bytes of total message size.
Further, in this example, the messages 205-250 may be sent in a sequence from top to bottom in the diagram 200. In this way, the nine byte tertiary payload of message 205 may be combined with the one byte secondary payload of message 210 in order to form a complete ten byte streaming payload (in addition to the two complete streaming payloads in message 205 and message 210). Likewise, the eight byte tertiary payload of message 210 may be combined with the two byte secondary payload of message 215 and so on until the tertiary byte of message 245 is combined with the secondary byte of message 250, such that a total of nineteen streaming payloads may be sent in this ten message example.
Each identifier 260 may represent both the message type (i.e., distribution of streaming payload, secondary payload, and tertiary payload bytes) and the message sequence (i.e., message one 205 to message ten 250 in this example). The ID 260 may be a fixed number of bytes (one in this example) depending on the number of messages in a sequence.
Each streaming payload 270 may be a fixed size payload (ten bytes in this example) that is associated with a particular streaming protocol. In some embodiments, the total message payload may be large enough to include multiple streaming payloads (e.g., a thirty byte message in this example would include two full streaming payloads in each message, as well as secondary and tertiary payloads, as shown). The streaming payloads are indicated by the clear fill pattern and labels.
Each secondary payload 280 and each tertiary payload 290 may combine to utilize the remaining message bytes (nine in this example). Each secondary payload 280 is indicated by a first fill pattern, while each tertiary payload 290 is indicated by a second fill pattern. In this example, for consistency, the first message 205 is represented as having no secondary payload 280 (or a secondary payload having size zero). Similarly, the last message 250 is represented as having no tertiary payload 290 (or a tertiary payload having size zero). One of ordinary skill in the art will recognize that the secondary and tertiary payloads may alternatively be referred to as “partial” payloads, with the understanding that sequential partial payloads are able to be combined into full streaming payloads.
In this example, one secondary payload 280 is combined with one tertiary payload 290 to generate a full streaming payload. Other embodiments may reduce the size of the streaming payload in some cases (e.g., a fifteen byte message length in this example may not include a full ten byte streaming payload in some messages). As shown, the message sequence may define the relative positions of the streaming payloads 270, secondary payloads 280, and tertiary payloads 290 within each message. In this example, the ID 260 is always the first byte. Different embodiments may arrange the ID and payload in various different ways.
Although the examples above and below may discuss data grouped into “bytes”, different embodiments may be implemented using different such discrete groupings (e.g., using bits, kilobytes, megabytes, etc.). In addition, different embodiments may include different numbers of devices (e.g., sources, destinations, etc.) and/or different interfaces or intermediary devices. For instance, in some embodiments, a “source” may be (and/or may be associated with) an intermediary device that receives streaming content from a server and delivers the content to the destination.
As shown, the process may then retrieve (at 310) a payload size associated with the current streaming session. Such payload sizes (and/or other relevant parameters) may include, for instance, total message payload size, encoding scheme, etc. The information may be retrieved from an appropriate resource (e.g., a look-up table, a remote server, the source, etc.).
Next, the process may determine (at 320) the maximum number of message types based on the payload sizes. Such a determination may be made using an equation such as equation 1 below.
M=n*(n+1)/2 (1)
Where M is the maximum number of message types and n is the total message payload size.
The process may then determine (at 330) the number of bytes needed for the ID and sequencing portion of the message (e.g., ID and sequence 260 described above) may be calculated using equation 2 below.
B=FLOOR[(⅛)log2(n*(n+1)/2)] (2)
Where B is the number of bytes needed for the ID and sequencing portion, and n is the total message payload size. Thus, for example, if n equals twenty, there are two hundred ten possible message types and one byte is used to identify them. As another example, if n equals twenty-five, there are three hundred twenty-five potential message types and two bytes are used to represent them. In these examples, bytes are used (thus, the one-eighth factor), but different embodiments may use different units.
Using such an algorithm, various types of encoding schemes may be used (with encoded packet size varying from one to n−1). In addition, representing the sequence number and message structure in a single ID reduces the processing time needed to parse a packet, fragment a packet, re-assemble a packet, and/or handle packet loss.
Next, the process may retrieve (at 340) an encoding packet size. The encoded packet size may be a discrete value between one and n−1, as described above. Various appropriate encoding schemes may be used by different embodiments (e.g., G.729)
The process may then define (at 350) the message structures and associate them with a unique ID and then may end. In the example of
The message structures may be defined similar to the structures shown in diagram 200. Thus, for the first message, the ID and one or more complete streaming payloads may be joined. The remaining message payload, if any, may be utilized for a tertiary (or secondary) payload in the first message.
The size of the secondary payload of the second message may then be calculated by determining a difference between the streaming payload size and the size of the tertiary payload in the first message. Again, one or more complete streaming payloads may be added to the ID and secondary payload. The second message may further include a tertiary payload that is calculated by determining the remaining message payload after complete payloads have been added.
From the third message onward, the size of the secondary payload for each message may be calculated by determining a difference between the tertiary payload (if any) size of the previous message and the streaming payload size. This may be repeated until a duplicate message structure is identified and discarded.
More generically, if the tertiary payload size of the previous message is not zero, the size of the secondary payload may be calculated using equation 3 below, while the size of the tertiary payload may be calculated using equation 4 below.
Size(SPN)=Size(STRP)−Size(TPN-1) (3)
Size(TPN)=n−B−Size(STRP)−Size(SPN) (4)
Where SPN is the secondary payload of the Nth message, STRP is the streaming payload, TPN is the tertiary payload of the Nth message, and TPN-1 is the tertiary payload of the previous message (i.e., the message before the Nth message in the sequence). As above, n is the total message payload size and B is the number of bytes needed for the ID and sequencing portion as calculated above using equations (1) and (2).
If the tertiary payload size of the previous message is zero, the size of the secondary payload may be set to zero and the size of the tertiary payload may be calculated using equation 4 above (where size(SPN) is set to zero).
As described above in reference to data structure diagram 200, the messages may be sent in a defined sequence (where position within the sequence is indicated by the message ID). As such, the streaming payloads, secondary payloads, and/or tertiary payloads may be provided in a specified order such that the payloads (and partial payloads) may be provided in the same order as the original streaming content. Thus, for example, the tertiary payload of message 205 may be joined with the secondary payload of message 210 with the information in the secondary payload coming after the information in the tertiary payload when the secondary and tertiary payloads are combined to form a full streaming payload.
As shown, process 400 may receive (at 410) streaming data. Such data may be received from various appropriate resources (e.g., local storage, content servers, etc.). The data may be received over various appropriate connections (e.g., local, wired, wireless, networks, etc.).
Next, the process may identify (at 420) the message structure. The message structure may be identified using a look-up table or other appropriate resource using the results of a process such as process 300. The process 400 may identify a next message to be sent and determine the associated structure (i.e., ID size and position, size and position of streaming payload(s), size and position of secondary and tertiary payloads if needed, etc.).
The process may then fragment (at 430) the received data. Such fragmenting may involve dividing the incoming data to match the sizes specified by the current message structure (whether streaming payload size, secondary payload size, or tertiary payload size). In some embodiments, fragmentation may not be required for all incoming packets (e.g., if an incoming packet size matches a streaming payload size, at least some of the incoming packets may be pass through without fragmentation).
Next, the process may generate (at 440) an optimized packet using the identified message structure. Such generation may include placing the current ID and payloads in the order specified by the message structure. Such payloads may have been previously fragmented (at 430) as necessary, and thus may be linked to form a message of the appropriate structure (and total size).
The process may then send (at 450) the optimized packet to the destination. Such sending may include various intermediary operations (e.g., sending a complete packet to a local storage for queuing). In addition, the source may interact with the destination in various ways before sending any optimized packet. For instance, the source may wait for a request from the destination before sending each packet. As another example, the source may wait for a confirmation that a previous packet was properly received before sending the next packet.
Process 400 may then determine (at 460) whether the streaming session has ended. Such a determination may be made in various appropriate ways (e.g., termination request from the destination, lack of destination response for a specified time, end of streaming content reached, etc.).
If the process determines (at 460) that streaming has not been terminated, the process may repeat operations 410-460 until the process determines (at 460) that streaming has been terminated and then may end.
As shown, process 500 may receive (at 510) attributes for the current streaming session. In some cases, the attributes may be the various payload sizes described above (e.g., total message size, streaming size, etc.). The destination may then perform a process similar to process 300 above in order to identify the various message structures. Alternatively, the received attributes may include a look-up table or other resource that represents the message structures defined by the source.
The process may then receive (at 520) a data packet. Such a data packet may be an optimized packet sent (at 450) above.
Next, process 500 may determine (at 530) the message attributes for the received data packet. Such determination may include, for instance, determining a size and location of a message ID. The message ID may then be used to identify a message structure (including size and location of streaming payload(s), secondary payload, and tertiary payload, if any) using a look-up table or other appropriate resource. Throughout this disclosure, the term “size” may refer to a value that defines a size in bytes (or other appropriate unit) of any particular element. The “location” may refer to a value that defines an absolute position within a message payload. For instance, a message payload having a discrete number of bytes (or other units) may define each location by specifying a starting location for the element (where the starting location may include integer values ranging from a first byte such as the ID byte in the example of chart 200 to a last byte). Thus, in the example of
Process 500 may then fragment (at 540) the received packet based on the payload locations and sizes. Next, the process may re-assemble (at 550) any partial streaming packet(s) included in the message with previously-received partial packets, if available. Any remaining incomplete streaming packets may be set aside until the next message is received.
The process may then provide (at 560) the complete packets to the destination player. Next, the process may determine (at 570) whether the streaming has been terminated. Such a determination may be made in various appropriate ways (e.g., termination request received at the destination, lack of source communication for a specified time, end of streaming content message received, etc.).
If the process determines (at 570) that streaming has not been terminated, the process may repeat operations 520-570 until the process determines (at 570) that streaming has been terminated and then may end.
One of ordinary skill in the art will recognize that the processes described above may be implemented in various different ways without departing from the scope of the disclosure. For instance, the operations may be performed in different orders than shown. As another example, some operations may only be performed based on satisfaction of some specified criteria. As still another example, some operations and/or sets of operations may be performed iteratively. In addition, some embodiments may include additional operations and/or omit various listed operations.
Many of the processes and modules described above may be implemented as software processes that are specified as one or more sets of instructions recorded on a non-transitory storage medium. When these instructions are executed by one or more computational element(s) (e.g., microprocessors, microcontrollers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc.) the instructions cause the computational element(s) to perform actions specified in the instructions.
In some embodiments, various processes and modules described above may be implemented completely using electronic circuitry that may include various sets of devices or elements (e.g., sensors, logic gates, analog to digital converters, digital to analog converters, comparators, etc.). Such circuitry may be able to perform functions and/or features that may be associated with various software elements described throughout.
Computer system 600 may be implemented using various appropriate devices. For instance, the computer system may be implemented using one or more personal computers (PCs), servers, mobile devices (e.g., a smartphone), tablet devices, and/or any other appropriate devices. The various devices may work alone (e.g., the computer system may be implemented as a single PC) or in conjunction (e.g., some components of the computer system may be provided by a mobile device while other components are provided by a tablet device).
As shown, computer system 600 may include at least one communication bus 605, one or more processors 610, a system memory 615, a read-only memory (ROM) 620, permanent storage devices 625, input devices 630, output devices 635, audio processors 640, video processors 645, various other components 650, and one or more network interfaces 655.
Bus 605 represents all communication pathways among the elements of computer system 600. Such pathways may include wired, wireless, optical, and/or other appropriate communication pathways. For example, input devices 630 and/or output devices 635 may be coupled to the system 600 using a wireless connection protocol or system.
The processor 610 may, in order to execute the processes of some embodiments, retrieve instructions to execute and/or data to process from components such as system memory 615, ROM 620, and permanent storage device 625. Such instructions and data may be passed over bus 605.
System memory 615 may be a volatile read-and-write memory, such as a random access memory (RAM). The system memory may store some of the instructions and data that the processor uses at runtime. The sets of instructions and/or data used to implement some embodiments may be stored in the system memory 615, the permanent storage device 625, and/or the read-only memory 620. ROM 620 may store static data and instructions that may be used by processor 610 and/or other elements of the computer system.
Permanent storage device 625 may be a read-and-write memory device. The permanent storage device may be a non-volatile memory unit that stores instructions and data even when computer system 600 is off or unpowered. Computer system 600 may use a removable storage device and/or a remote storage device as the permanent storage device.
Input devices 630 may enable a user to communicate information to the computer system and/or manipulate various operations of the system. The input devices may include keyboards, cursor control devices, audio input devices and/or video input devices. Output devices 635 may include printers, displays, audio devices, etc. Some or all of the input and/or output devices may be wirelessly or optically connected to the computer system 600.
Audio processor 640 may process and/or generate audio data and/or instructions. The audio processor may be able to receive audio data from an input device 630 such as a microphone. The audio processor 640 may be able to provide audio data to output devices 640 such as a set of speakers. The audio data may include digital information and/or analog signals. The audio processor 640 may be able to analyze and/or otherwise evaluate audio data (e.g., by determining qualities such as signal to noise ratio, dynamic range, etc.). In addition, the audio processor may perform various audio processing functions (e.g., equalization, compression, etc.).
The video processor 645 (or graphics processing unit) may process and/or generate video data and/or instructions. The video processor may be able to receive video data from an input device 630 such as a camera. The video processor 645 may be able to provide video data to an output device 640 such as a display. The video data may include digital information and/or analog signals. The video processor 645 may be able to analyze and/or otherwise evaluate video data (e.g., by determining qualities such as resolution, frame rate, etc.). In addition, the video processor may perform various video processing functions (e.g., contrast adjustment or normalization, color adjustment, etc.). Furthermore, the video processor may be able to render graphic elements and/or video.
Other components 650 may perform various other functions including providing storage, interfacing with external systems or components, etc.
Finally, as shown in
As used in this specification and any claims of this application, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic devices. These terms exclude people or groups of people. As used in this specification and any claims of this application, the term “non-transitory storage medium” is entirely restricted to tangible, physical objects that store information in a form that is readable by electronic devices. These terms exclude any wireless or other ephemeral signals.
It should be recognized by one of ordinary skill in the art that any or all of the components of computer system 600 may be used in conjunction with some embodiments. Moreover, one of ordinary skill in the art will appreciate that many other system configurations may also be used in conjunction with some embodiments or components of some embodiments.
In addition, while the examples shown may illustrate many individual modules as separate elements, one of ordinary skill in the art would recognize that these modules may be combined into a single functional block or element. One of ordinary skill in the art would also recognize that a single module may be divided into multiple modules.
The foregoing relates to illustrative details of exemplary embodiments and modifications may be made without departing from the scope of the disclosure as defined by the following claims.
