Patent Application
20190132640
This disclosure relates to playback, control, and redistribution of internet streaming content.
Devices such as smartphones, tablets, computers, and other devices are sometimes used to access internet streaming media content at sites such as YouTube, Netflix, and Pandora, as only three of innumerable examples. The content can be played on the device, or redistributed to a different playback device, such as a television, a portable speaker, a smartphone, or a tablet, to name several non-limiting examples. For audio/video media, sometimes the video is played on the device, and the audio is sent to a wireless audio playback device, or another audio device with better sound quality and greater volume. In such cases, though, the audio and video may not be synchronized due to delays from wireless transmission and processing. Playback of audio/video media can be simplified if the wireless audio playback device is enabled to directly access and play the media.
All examples and features mentioned below can be combined in any technically possible way.
In one aspect, a device for automated playback and redistribution of internet streaming content includes a processor, and a storage device having stored executable instructions that are operable, when executed by the processor, to cause the processor to receive a command to select a web media service, and in response to the command, automatically retrieve user credentials for the selected web media service, use the command to select a web browser instance, retrieve the sign-in page from the selected web media service, insert the user credentials into the sign-in page, log into the web media service, select a stream on the web media service, and initiate playback of the selected stream. The device can also be configured to further control the selected stream in response to received content control commands. This further control can include further control of playback of the selected stream (e.g., play, stop, pause, resume, forward, reverse, skip, and the like). The further control may be of device transport controls.
Embodiments may include one of the above or below features, or any combination thereof. Playback of the selected stream may be accomplished using the device. Playback of the selected stream may be accomplished using another connected device. The instructions may be further operable to send the selected stream to a final destination, where the selected stream is rendered.
Embodiments may include one of the above or below features, or any combination thereof. The stored executable instructions may comprise an application programming interface (API) that starts and controls a web browser instance. The stored executable instructions may comprise a plurality of APIs comprising one API for each of a plurality of separate web media services. Using the command to select a web browser instance may comprise determining from the command a selection of the API for a particular web media service. Using the command to select a web browser instance may further comprise running the selected API, to automatically connect to the particular web media service. The stored executable instructions may further comprise a web browser that is configured to access information from the world wide web. The plurality of APIs may reside within the web browser.
Embodiments may include one of the above or below features, or any combination thereof. The stored executable instructions may comprise an audio sink program that is configured to send the selected stream to a separate connected audio playback device, and/or a video sink program that is configured to send the selected stream to a separate connected video playback device, and/or a buffer that is configured to be coupled to a separate pull-mode video device.
In another aspect, a device for automated playback and redistribution of internet streaming content includes a processor and a storage device having stored executable instructions that are operable, when executed by the processor, to cause the processor to receive a command to select a web media service and in response to the receipt of the command, automatically retrieve user credentials for the selected web media service, use the command to select a web browser instance, retrieve the sign-in page from the selected web media service, insert the retrieved user credentials into the retrieved sign-in page, log into the web media service, select a stream on the web media service, and initiate playback of the selected stream using the device. The stored executable instructions may comprise a plurality of application programming interfaces (APIs) that each start and control a web browser instance and comprise one API for each of a plurality of separate web media services. Using the command to select a web browser instance may comprise determining from the command a selection of the API for a particular web media service, and running the selected API to automatically connect to the particular web media service. The stored executable instructions may further comprise a web browser that is configured to access information from the world wide web. The plurality of APIs may reside within the web browser.
Automated playback and redistribution of internet streaming content can be accomplished by a device such as a wireless audio playback device that is configured to receive a command to select a web media service, retrieve user credentials for the selected web media service, use the command to select a correct web browser instance, retrieve the sign-in page from the selected web media service, insert the user credentials, log into the service, select the stream on the web media service, and initiate playback of the stream. The automated playback and redistribution of internet streaming content thus allows a user to access and play web media simply by commanding the device to do so. The device can also be configured to further control internet streaming content in response to received commands. This further control can include further control of playback (e.g., pause, resume, forward, reverse, skip, and the like).
Elements of figures are shown and described as discrete elements in a block diagram. These may be implemented as one or more of analog circuitry or digital circuitry. Alternatively, or additionally, they may be implemented with one or more microprocessors executing software instructions. The software instructions can include digital signal processing instructions. Operations may be performed by analog circuitry or by a microprocessor executing software that performs the equivalent of the analog operation. Signal lines may be implemented as discrete analog or digital signal lines, as a discrete digital signal line with appropriate signal processing that is able to process separate signals, and/or as elements of a wireless communication system.
When processes are represented or implied in the block diagram, the steps may be performed by one element or a plurality of elements. The steps may be performed together or at different times. The elements that perform the activities may be physically the same or proximate one another, or may be physically separate. One element may perform the actions of more than one block. Audio signals may be encoded or not, and may be transmitted in either digital or analog form. Conventional audio signal processing equipment and operations are in some cases omitted from the drawing.
System 10,
One process for the automated playback and redistribution of internet streaming content is shown in
The automated playback and redistribution of internet streaming content can be accomplished using a device that is configured to play audio or audio/video streamed content. A non-limiting example detailed herein is a wireless audio playback device, shown in
Software stack 50 includes components that already exist in a wireless audio playback device, including web browser 54, network services 52, GStreamer 56, audio path 60 and Advanced Linux Sound Architecture (ALSA) 62 that are involved in audio output, Digital Living Network Alliance (DLNA) media server 64 and Miracast protocol driver 66 that are involved in video output, and virtual personal assistant (VPA) client 58 (in this case, for “Alexa” from Amazon). Note that these components, and the manners in which they operate to accomplish the functionalities of the wireless audio playback device, are known in the field and so are not further described herein. Components added to stack 50 to enable the automated playback and redistribution of internet streaming content include a web app 74 and a Selenium web driver 72 that both reside within web browser 54, GStreamer audio sink 78 that is involved in streaming audio to another destination, GStreamer video sink 76 that is involved in streaming video to another destination, and HTTP Live Streaming (HLS) server 80 that acts as a buffer for pull-mode video devices. Note that the Selenium web driver is one non-limiting example of a web driver that can be used herein. Also note that GStreamer is one non-limiting example of an audio/video pipeline that can be used herein. Further, note that depending on the use of the system, the Gstreamer audio and video sinks, and the HLS server, may not be necessary. The Web Driver is configured to accept commands and send them to a browser. The software stack includes one Web Driver for each supported internet media service, as is further described below.
The software stack preferably comprises an application programming interface (API) that starts and controls a web browser instance. The API may be but need not be the Selenium Web Driver. There are preferably a plurality of APIs comprising one such API for each of a plurality of separate web media services. The selection of the correct web browser instance in this case may comprise determining from the command a selection of the API for a particular web media service. The selected API can then be run so as to automatically connect to the particular web media service. Preferably, the APIs reside within a web browser that is part of the software stack.
As discussed above, the software stack preferably but not necessarily also includes an audio sink program that is configured to send the selected stream to a separate connected audio playback device and/or a video sink program that is configured to send the selected stream to a separate connected video playback device, and/or a buffer that is configured to be coupled to a separate pull-mode video device.
A state machine 82 that illustrates one example of an automated playback and redistribution of internet streaming content is shown in
The software stack is run in several different processes. The first is a daemon, which is a standard daemon/service waiting from an input/utterance/command to load the appropriate Selenium driver. In the state machine description, the command is coming from the VPA/Alexa/Voice Personal Assistant. Another process is the Selenium Web driver and web application which receive a playback command from the daemon. Both the Selenium driver and the web application reside in the web browser space. The web application is the generic engine to fetch the audio/video content and play it back. The Selenium driver mimics the actions of a real end-user who would be using the media service to do the playback using a mouse etc. in the appropriate web site. Note also that GStreamer is running inside the web browser as a library. The GStreamer library can be extended with different plug-ins and sinks. GStreamer will drive the content to Miracast and/or DLNA play-to directly when required. Another process is the HLS web server, which waits from the playlist and the video content request from an external pull mode application (e.g., a smartphone video player). Push mode video devices are handled differently as set forth in the State Machine and in Table 1.
The intent indicates an actionable command that is understood by a software entity. A voice (speech) processing system can be described as a series of stages: Voice Pick-up—typically comprising of a microphone (or collection of mics) on the product to convert speech input into a digital audio signal. Wake word detection—a pre-programmed word such as “Alexa” on the product is recognized by this detection sub-system. Speech Transmission—the digital audio signal for an entire phrase (such as “Alexa, what's the weather?”) is transferred from the product to an external voice processing software (typically a Cloud Service—such as Amazon's Alexa Voice Service). Speech-To-Text (STT) conversion—the digital audio signal is converted to a text phrase (such as “Alexa, what's the weather?”). Intent Mapping—the phrase “Alexa, what's the weather” is converted to a command that can be acted upon by a software entity (such as a REST API command “GET {Attribute: Weather}”. Intent Transmission—the command is sent to the product or another Cloud Service to perform an operation (such as requesting weather.com to report the weather). Intent Processing—the product or cloud service executes the command and reports the result to the requesting software entity. Command Response—once the command has executed, a response is generated in a format that software entity understands (such as “{Weather:Cloudy}”). Text-To-Speech (TTS) conversion—the command response is converted to speech as a digital audio signal, typically by a Cloud Service (such as “It's Cloudy today”). Voice Response Transmission—the response is sent as a digital audio signal to the product. Voice Response Output—the product's speaker (or collection of speakers) converts the digital audio signal to analog audio to be heard.
The Selenium Web Driver for each media service can be developed as follows. In general, the DOM element names for the login credentials and the playback control are extracted from the media content provider web site. These DOM element names are fed to a web driver template to generate a specific media content provider web driver. More specifically, this can be accomplished by opening the media content provider web page and extracting the DOM element names for the login credential input (user name and password input element), logging into the media content provider and finding the media playback control DOM element name, and then editing the generic web driver to replace the ‘media_content_url’ variable with the new media content provider url, replace the ‘user_name_dom_element_name’ and ‘playback_control_dom_element_name’ with the DOM credential element names found as above, and replace the ‘playback_control_dom_element_name’ with the DOM playback control element name found as above. Note that the action of writing a Selenium Web Driver for a media content provider can be automated. Also, the Selenium Web Drivers can reside in the cloud and be accessed by the device. This way, updates to all devices (such as the addition of a new supported media service, or changes to reflect changes made to the media service web site) can be accomplished by making changes to the appropriate Selenium Web Driver in the cloud server.
An exemplary wireless audio playback device 110 will now be described in greater detail with reference to
The assigned entities can be associated with different ones of the digital audio sources such that a single audio playback device 110 can provide for single press access to various different digital audio sources. In one example, the assigned entities include at least (i) user-defined playlists of digital music and (ii) an Internet radio station. In another example, the assigned entities include individual radio stations provided by an Internet radio site.
Notably, the preset indicators 418 operate in the same manner, at least from a user's perspective, regardless of which entities are assigned and which of the digital audio sources provide the assigned entities. That is, each preset indicator 418 can provide for single press access to its assigned entity whether that entity is a user-defined playlist of digital music provided by an NAS device or an Internet radio station provided by an Internet music service, for example.
With reference to
The network interface 420 provides for communication between the audio playback device 110, a remote server, the audio sources, an audio playback device controller such as a smartphone or tablet, and other audio playback devices 110 via one or more communications protocols. The network interface 420 may provide either or both of a wireless interface 430 and a wired interface 432. The wireless interface 430 allows the audio playback device 110 to communicate wirelessly with other devices in accordance with a communication protocol such as such as IEEE 802.11 b/g. The wired interface 432 provides network interface functions via a wired (e.g., Ethernet) connection.
In some cases, the network interface 420 may also include a network interface processor 434. If the audio playback device supports Apple AirPlay® (a proprietary protocol stack/suite developed by Apple Inc., with headquarters in Cupertino, Calif., that allows wireless streaming of audio, video, and photos, together with related metadata between devices), then processor 434 may include network media processor functionality. When network media processor functionality is present, if a user connects an AirPlay® enabled device, such as an iPhone or iPad device, to a LAN, the user can then stream music to the network connected audio playback devices 110 via Apple AirPlay®. The network interface processor 434 provides network access (i.e., the Wi-Fi network and/or Ethernet connection can be provided through the network interface processor 434) and can support AirPlay® audio. AirPlay® audio signals are passed to the processor 422, using the I2S protocol (an electrical serial bus interface standard used for connecting digital audio devices), for downstream processing and playback. The audio playback device 110 in this case can support audio-streaming via multiple protocols such as AirPlay® and/or DLNA's UPnP protocol, all integrated within one device.
All other digital audio coming from network packets comes straight from the network interface processor 434 through connection (e.g., USB bridge) 436 to the processor 422 and runs into the decoders, DSP, and eventually is played back (rendered) via the electro-acoustic transducer(s) 415.
The network interface 420 can also include a Bluetooth (BT) system-on-chip (SoC) 438 for Bluetooth applications (e.g., for wireless communication with a Bluetooth enabled controller (not shown)). Suitable BT SoC devices are known in the art and so are not further described herein.
Streamed data pass from the network interface 420 to the processor 422. The processor 422 can execute instructions within the audio playback device (e.g., for performing, among other things, the automated playback and redistribution of internet streaming content, digital signal processing, decoding, and equalization functions), including instructions stored in the memory 428. The processor 422 may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor 422 may provide, for example, for coordination of other components of the audio playback device 110, such as control of user interfaces, and applications run by the audio playback device 110.
The processor 422 provides a processed digital audio signal to the audio hardware 424 which includes one or more digital-to-analog (D/A) converters for converting the digital audio signal to an analog audio signal. The audio hardware 424 also includes one or more amplifiers which provide amplified analog audio signals to the electroacoustic transducer(s) 415 for playback. In addition, the audio hardware 424 may include circuitry for processing analog input signals to provide digital audio signals for sharing with other devices.
The memory 428 may include, for example, flash memory and/or non-volatile random-access memory (NVRAM). In some implementations, instructions (e.g., software) are stored in memory 428. The instructions, when executed by one or more processing devices (e.g., the processor 422), perform one or more processes, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory 428, or memory on the processor). The instructions may include instructions for performing decoding (i.e., the software modules include the audio codecs for decoding the digital audio streams), as well as digital signal processing and equalization.
Embodiments of the systems and methods described above comprise computer components and computer-implemented steps that will be apparent to those skilled in the art. For example, it should be understood by one of skill in the art that the computer-implemented steps may be stored as computer-executable instructions on a computer-readable medium such as, for example, floppy disks, hard disks, optical disks, Flash ROMS, nonvolatile ROM, and RAM. Furthermore, it should be understood by one of skill in the art that the computer-executable instructions may be executed on a variety of processors such as, for example, microprocessors, digital signal processors, gate arrays, etc. For ease of exposition, not every step or element of the systems and methods described above is described herein as part of a computer system, but those skilled in the art will recognize that each step or element may have a corresponding computer system or software component. Such computer system and/or software components are therefore enabled by describing their corresponding steps or elements (that is, their functionality), and are within the scope of the disclosure.
A number of implementations have been described. Nevertheless, it will be understood that additional modifications may be made without departing from the scope of the inventive concepts described herein, and, accordingly, other embodiments are within the scope of the following claims.
This application claims priority of Provisional Patent Application 62/579,297, filed on Oct. 31, 2017.
| Number | Date | Country | |
|---|---|---|---|
| 62579297 | Oct 2017 | US |
