Patent Application
20180115844
The current application is directed to audio systems and in particular to audio systems incorporating location based services.
Homes or other locations may include speakers in multiple different locations. A central audio receiver connected to the speakers in the different locations may allow audio to be played in the different locations. The central audio receiver may allow different audio to be played at the different locations.
An additional, alternative and/or improved audio system capable of playing audio to different locations is desirable.
In accordance with the present disclosure there is provided an audio system comprising: a plurality of speaker devices each comprising: a network interface; a processing unit for executing instructions; a memory unit storing instructions, which when executed by the processing unit configure the respective speaker device to provide: a synchronization client for synchronizing an internal clock with a common clock; location based service (LBS) functionality for transmitting an LBS signal for use in determining a location of an external device; and media player client functionality for controlling playback of media based on one or more received playback commands; and a portable device comprising: a network interface; a processing unit for executing instructions; and a memory unit storing instructions, which when executed by the processing unit configure the portable device to provide: location based service (LBS) functionality for receiving the LBS signal transmitted by one or more of the plurality of speaker devices; and media player control functionality for transmitting playback commands to one or more media player clients.
In a further embodiment, the system further comprises an access point including NTP server functionality and player server functionality.
In a further embodiment of the system, one or more of the plurality of speaker devices and the portable device comprise NTP server functionality and player server functionality.
In a further embodiment of the system, the memory of the portable device further stores instructions for configuring the portable device to provide functionality for: determining a distance to one or more of the plurality of speaker devices based on the LBS signals received from the one or more of the plurality of speaker devices; and determining one or more of the plurality of speaker to playback media on.
In a further embodiment of the system, the memory of the portable device further stores instructions for configuring the portable device to provide functionality for: transitioning media playback from one or more of the plurality of speaker devices to one or more different speaker devices.
In a further embodiment of the system, transitioning media playback comprises: streaming media to the one or more different speaker devices; beginning playback of the media at a specified time of the synchronized internal clock; adjusting playback speed based on an amount of buffered media.
In a further embodiment of the system, adjusting playback speed comprises adjusting the playback speed according to playback speed factor α:
where: bufferthreshold is a buffer level required to sustain normal playback of the media; and buffercurrent is the current amount of media data in the buffer.
In a further embodiment of the system, the memory of the portable device further stores instructions for configuring the portable device to provide functionality for: adjusting a volume of one or more of the speaker devices playing back media based on a distance of the portable device from the respective speaker device.
In a further embodiment of the system, adjusting the volume comprises adjusting the volume according to:
where: L0 is a desired volume level, in dB, at a selected one of the plurality of speaker devices; d0 is a distance from the portable device to the selected speaker device; Ln is the volume of the nth speaker device in dB; and dn is a distance from the portable device to the nth speaker device, where the plurality of speaker devices, other than the selected speaker device, are enumerated from n=1 to N−1 with N being the number of speaker devices playing back media.
In a further embodiment of the system, rein the selected speaker device is the closest speaker device to the portable device.
In accordance with the present disclosure, there is further provided a method for controlling media playback at one or more speaker devices, the method comprising: playing back audio at a first speaker device of a plurality of speaker devices, the first speaker device associated with a first location;
detecting a location change of a mobile device from a first location to a second location based on one or more location based services (LBS) signals transmitted by a plurality of speaker devices; and transitioning media playback from the first speaker device to a second speaker device of the plurality of speaker devices associated with the second location.
In a further embodiment of the method, transitioning playback comprises stopping playback at the first speaker device and starting playback at the second speaker device.
In a further embodiment of the method, each of the speaker devices transmits LBS signals including a unique identifier associated with a location of the respective speaker device.
In a further embodiment of the method, each of speaker devices comprises a respective clock that is each synchronized to a common clock.
In a further embodiment of the method, transitioning media playback comprises: streaming media to the one or more different speaker devices; beginning playback of the media at a specified time of the synchronized internal clock; adjusting playback speed based on an amount of buffered media.
In a further embodiment of the method, adjusting playback speed comprises adjusting the playback speed according to playback speed factor α:
where: bufferthreshold is a buffer level required to sustain normal playback of the media; and buffercurrent is the current amount of media data in the buffer.
In a further embodiment, the method further comprises: adjusting a volume of one or more of the speaker devices playing back media based on a distance of the portable device from the respective speaker device.
In a further embodiment of the method, adjusting the volume comprises adjusting the volume according to:
where: L0 is a desired volume level, in dB, at a selected one of the plurality of speaker devices; d0 is a distance from the portable device to the selected speaker device; Ln is the volume of the nth speaker device in dB; and dn is a distance from the portable device to the nth speaker device, where the plurality of speaker devices, other than the selected speaker device, are enumerated from n=1 to N−1 with N being the number of speaker devices playing back media.
In accordance with the present disclosure, there is further provided a speaker device comprising: a network interface; a processing unit for executing instructions; a memory unit storing instructions, which when executed by the processing unit configure the respective speaker device to provide: a synchronization client for synchronizing an internal clock with a common clock; location based service (LBS) functionality for transmitting an LBS signal for use in determining a location of an external device; and media player client functionality for controlling playback of media based on one or more received playback commands
In accordance with the present disclosure, there is further provided a portable device comprising: a network interface; a processing unit for executing instructions; and a memory unit storing instructions, which when executed by the processing unit configure the portable device to provide: location based service (LBS) functionality for receiving the LBS signal transmitted by one or more of the plurality of speaker devices; and media player control functionality for transmitting playback commands to one or more media player clients.
An audio system may include a number of speakers in different locations. For example, in a home, speakers may be located in a living room, an office, an outdoor patio, etc. Audio can be played back over the different speakers from a central control device. For example, an individual may control the audio played back over the speakers in the different locations from a smart phone or other portable device. The user may select what audio is playing at the different locations. The speakers may be controlled to playback the same audio at one or more of the locations or may playback different audio to each location. Further, the audio playback may be controlled to provide automatic location switching so that the audio is played over speakers at different locations based on the location of the individual.
As described in further detail below, the audio system provides speaker devices that include a clock that can be synchronized with a central clock. All of the clocks of the speaker devices within a single audio system may have a time difference within, for example, 5 milliseconds. The synchronized speaker devices may provide seamless transitioning between different speaker locations so that music being played over the system can be switched to different speaker locations without interrupting the playback. The speaker devices may allow their locations, or the proximity to their locations to be determined. The location awareness may allow automatic transitioning from one speaker location to another based on a determined proximity of the user to the different speaker locations. For example, if the user is within a certain distance from a speaker location, the speaker will start playing automatically and when the user moves from a first location to a second location, the speaker that is in the first location will stop playing and the speaker in the second location will start playing. Although described as transitioning audio playback from speakers at a first location to a second location, it is possible for the playback to occur on the playback device or smart phone itself and transition to or from a speaker location. The playback device may be synchronized with the speaker locations, and any other devices. As audio playback is transitioned from one playback location to another, the volumes may be gradually reduced/increased to provide a smooth transition between playback locations.
An application may be provided, for example on the a user's smart phone, that allows a music play profile to be defined. The music play profile may specify playback control options. For example, the profile may specify that during a particular time, say 9:00 am to 5:00 pm the audio system should automatically play music based on the location of the user; while outside of the time, the playback locations should be specified by the user. Further, the application may allow a user to specify particular songs, artists, albums genres, playlists etc to play at a particular speaker location at specific times. For example, the application may allow a user to schedule a ‘wakeup’ playlist be played in a ‘bedroom’ location starting at 6:45 am. Additional schedules of playback locations, times and audio may be specified. The application may allow remote control of the audio system. The communication between components may be done over a network and as such communication can be achieved over any network, assuming that any intervening network components, such as firewall and/or NAT devices are configured to allow the appropriate traffic.
The separate speaker device 102b may comprise a processing unit 104 for executing instructions. The processing unit may be provided by a microcontroller or microprocessor. For example, the processing unit may be provided by 700 MHz single-core ARM processor, although other processors may be used. A memory unit 106 may store both data and instructions. The memory unit 106 may comprise a hierarchical memory structure, including registers, random access memory (RAM), read-only memory (ROM), flash-based memory devices, solid state drives (SSDs) and hard disk drives (HDDs). The separate speaker device 102b may further comprise input/output (I/O) interfaces 108 which may provide an interface to other components. For example the I/O connectors may provide an interface for connecting to the speakers 102a. The I/O interfaces may also provide wireless interfaces such as Bluetooth communication radios, including Bluetooth 4.0 chips, or other types of wireless communication including for example Zigbee, WiFi etc. The separate speaker device 102b may further comprise a network interface 110 for connecting to a network. The network connection may be provided by a wired or wireless network connection. Although the network connection may be considered as an I/O interface, it is described separately for clarity.
As described above, the separate speaker device 102b includes a processing unit 104 for executing instructions and a memory unit 106 for storing instructions. The instructions stored in the memory unit 106, when executed by the processing unit 104, configure the separate speaker device to provide various functionality 112, including player client functionality 114, a network time protocol (NTP) client functionality 116 and location based services (LBS) sender functionality 118.
The NTP client functionality 116 is responsible for synchronizing a clock of the separate speaker device 102b with a NTP Server. The LBS sender functionality 118 sends an LBS signal via a particular protocol, such as iBeacon protocol or Zigbee protocol, etc. that is used by a mobile device to determine the device's proximity to the separate speaker device transmitting the LBS signal. The player client functionality 114 is responsible for receiving streamed audio and control commands, such as play, pause, etc and carries the commands. The player client functionality 114 may receive commands, and streamed audio, from corresponding player server functionality.
The speaker device 120 is substantially the same as the speaker device 102 and as such the individual components are not described further. Although the components of speaker device 120 are substantially similar to those of speaker device 102, all of the components are provided within a single enclosure. That is the speaker device 120 combines the speaker 102a and the separate speaker device 102b as a single device.
The system 100 further comprises an access point player 130. The access point player 130 may be provided by a wireless access point or router that provides additional functionality. The wireless access point player 130 may comprise a processing unit 132 for executing instructions. The processing unit 132 may be provided by a microcontroller or microprocessor. A memory unit 134 may store both data and instructions. The memory unit 134 may comprise a hierarchical memory structure, including registers, random access memory (RAM), read-only memory (ROM), flash-based memory devices, solid state drives (SSDs) and hard disk drives (HDDs). The memory unit may provide sufficient storage for storing audio files for playback along with related metadata, such as art work etc. The access point player 130 may further comprise input/output (I/O) interfaces 136 which may provide an interface to other components. For example the I/O interface may provide an interface for connecting to external storage devices such as a USB drive. The access point player 130 may further comprise a network interface 138 for providing network connectivity to a plurality of devices.
As described above, the access point player 130 includes a processing unit 132 for executing instructions and a memory unit 134 for storing instructions. The instructions stored in the memory unit 134, when executed by the processing unit 132, configure the access point player 130 to provide various functionalities 140 in addition to the networking functionality typically provided by access points or routers. The functionalities 140 may include player server functionality 142 and NTP server functionality 144. The player server functionality 142 may receive player control command message over the network, which may be sent from, for example, an application running on a mobile devices. The player control server may process the message or may forward the message to one or more connected devices. For example, a “pause” command may cause the player server functionality to stop streaming audio to connected speaker devices. The player server functionality 142 may also be responsible for streaming audio music from the access point player 130 to connected devices. The access point player 130 may further comprise NTP server functionality 144 that allows the synchronizing of clocks of devices with NTP client functionality, such as speakers.
The audio system 100 further comprises a portable device such as a smart phone. The portable device 150 comprises a processing unit 152. The processing unit 152 may be provided by a microcontroller or microprocessor. For example, the processing unit may be provided by Snapdragon processor, and A8 processor or other processors. A memory unit 154 may store both data and instructions. The memory unit 154 may comprise a hierarchical memory structure, including registers, random access memory (RAM), read-only memory (ROM), flash-based memory devices, solid state drives (SSDs) and hard disk drives (HDDs). The portable device 150 may further comprise input/output (I/O) interface 156 which may provide an interface to other components. The I/O interface 156 may provide wireless interfaces such as Bluetooth communication radios, including Bluetooth 4.0 chips, or other types of wireless communication including for example Zigbee, WiFi etc. The portable device 150 may further comprise a network interface 158 for connecting to a network. The network connection may be provided by wired or wireless network connection. Although the network connection may be considered as an I/O interface, it is described separately for clarity. The network interface may be provided by one or more networks including WiFi networks, WiMax networks, cellular networks or other network types.
As described above, the mobile device includes a processing unit 152 for executing instructions and a memory unit 154 for storing instructions. The instructions stored in the memory unit 154, when executed by the processing unit 152, configure the portable device 150 to provide various functionality 160 including player control client functionality 162 and LBS receiver functionality 164. The player control client functionality 162 accepts user input of control commands such as play, pause, stop, song selection, location switching, etc, as well as providing a user interface. The player control client functionality 162 communicates with player server functionality 142. The LBS receiver functionality 164 receives LBS signals transmitted from LBS senders and determines the distance to the device sending the LBS signal. Example protocols that may support LBS Receiver and LBS Sender functionality are iBeacon protocol, Zigbee protocol, etc.
As depicted, each of the speaker devices 406, 408 broadcast location based service signals 410, 412 that include an identifier associated with the locations of the speaker devices 406, 408. The player control client 402 detects one or more of the LBS signals 414 and determines a relevant location based on the received signal. In
When transitioning playback between devices, audio will need to be streamed over the network to the next playback device to start playing. However, if the network is congested the audio data may not arrive at the speaker by the time it needs to play the music. If the required audio data is not available when required to be played, the audio may stall or skip the missing data. Skipping or stalling playback may be avoided by ensuring the required audio data is available at the playback device by the time playback is required. In order to provide the audio data on time, streaming of the audio data to the speaker may begin before the user actually switches to the speaker. For example, when a user begins the process to select a playback location, streaming to one or more speakers, such as the nearest speaker or all speakers, may begin. When the user finally selects the speaker to transition playback to, the mobile device may still play the music for some time threshold, such as 1 second for example. This overlap time may be used to stream as much audio data to the speaker as possible. On the speaker or other synchronized playback device, if it does not receive enough audio data for playback at the appropriate time, that is the time the audio transition is to occur, playback may begin, however the audio may be slowed down in an attempt to allow more time to buffer the required amount of audio data. For example, the playback speaker may slow down the audio playback by a factor α, which may be dynamically tuned according to the buffer level until the buffer level reaches the buffer threshold to playback at regular speed. Further, α may be slowly varied as playback returns to normal speed in order to mask the slowed playback from the user. If α is set to be 1, the playback speed is normal, that is 1 second audio segment is played back in 1 second. If α is set to 1.1 then the playback is slowed so that 1 second audio segment is played back in 1.1 seconds. α may be set according to:
Where:
bufferthreshold is a buffer level required to sustain normal playback of the audio and may be based on a bit rate of the audio as well network conditions; and
buffercurrent is the current amount of audio data in the buffer.
At some point, the user may move 424 to the location of the second speaker device 408. The mobile device, or more particularly the LBS receiver functionality, receives the LBS signal 426 and determines the location. Upon detecting the change in location, a transition command 428 may be sent from the player control client functionality 402 to the player server functionality 404. The transition command may include the new location identifier. The player server functionality 404 receives the transition command and sends commands to the speaker devices in order to carry out the transition. As depicted, the player server functionality 404 sends a stop command 430 to the speaker currently playing the audio and a play command 432 to the speaker at the new location. The stop and play commands may include an indication of when to carry out the command. As depicted, the play command may indicate that it should be carried out in 200 milliseconds from the current time and the stop command may indicate that it should be carried out in 400 milliseconds from the current time. By having the stop command carried out after the play command, the transition between speaker devices 406, 408 may be less noticeable. The transition between speaker locations may also be based on the proximity of the mobile device to a speaker location. For example, as the user moves closer to a location, the volume of the new location may be increased as the playback volume of the previous location is decreased.
As described, all the devices that are running NTP client functionality synchronize their clocks with the NTP Server functionality. The application running on the mobile device may send out a message indicating that it will start playing a song at time X. Because the clocks on all speaker devices are synchronized, all the speaker devices will receive the music and start playing the song at the same point in time. Due to network congestion or packet loss, the music data may be received after the time when it is assumed to be played, in that case, the packet will be dropped and only the music data that should be played at the specific time point will actually be played. Music data that is received prior to the time it is supposed to be played may be stored temporarily on the speaker devices.
The speaker devices may be registered with the access point player 130 so that the access point player can send data or commands to specific speaker devices. Alternatively, the speaker devices may not register with the access point player, but instead the access point player may broadcast commands and data to all devices connected to a particular unit. The access point and the speaker devices, and mobile device may support WiFi Protected Setup (WPS) to simply the setup of the system. A user may press a button on the speaker device, which causes the speaker device to scan for possible WiFi access points. The user may click a button in the player application, which causes the application to send a message to the access point enabling the WPS on the access point to allow the speaker devices to connect to the network.
The above has described audio playback as occurring on one playback device at a time, such as at a speaker location or at a the mobile device. However, the synchronization provided by the NTP functionality between devices may allow the audio playback on multiple devices simultaneously. Further, with the LBS functionality able to determine or at least estimate distances from the user, or the mobile device carried by the user, to the different speaker locations, it is possible to adjust the volume level of playback speakers so that the volume the user hears from each speaker is approximately equal. The mobile device may determine the distance to each speaker location and the mobile device. One speaker location may be selected as providing the desired volume level. The volume of the remaining speakers may be set according to:
Where:
L0 is the desired volume level, in dB, at the selected speaker location;
d0 is the distance from the mobile device to the selected speaker location;
Ln is the volume at the nth speaker location in dB; and
dn is the distance from the mobile device to the nth speaker, where the speakers other than the selected speaker are enumerated from n=1 to N−1 with N being the number of speaker locations.
The speaker volumes may be periodically updated, for example once a second, based on movement of the user or mobile device.
As depicted in
As depicted in
The mobile device uses the received LBS signals to determine the new location and a transition command may be sent from the mobile device to the player server, causing the player server to transmit a playback ‘Stop’ command for the current location (1106) as well as a playback ‘Play’ command for the new location (1108). The timing of the ‘Stop’ and ‘Play’ command may be offset so that the ‘Stop’ command occurs after the ‘Play’ command so that the audio is played momentarily on speakers at both locations.
As described above, a mobile device may be used in determining a users location relative to one or more speakers, which in turn may be used to determine one or more playback locations for audio. The mobile device is described as being provided by a smart phone, however, it may be provided by various devices, such as for example a wearable device like a smart watch, fitness tracking band. Additionally, the functionality described above as being provided by the mobile device may be provided by a combination of interacting devices. For example, the functionality describe above may be provided by a smart phone device and a connected smart watch. The smart watch or other wearable device may be used to determine a distance of the user to the speakers. The wearable device may be more likely to be carried by the user and as such provide a more accurate distance measurement from the user to the speakers.
It should be noted that the algorithms illustrated and discussed herein as having various modules which perform particular functions and interact with one another. It should be understood that these modules are merely segregated based on their function for the sake of description and represent computer hardware and/or executable software code which is stored on a computer-readable medium for execution on appropriate computing hardware. The various functions of the different modules and units can be combined or segregated as hardware and/or software stored on a non-transitory computer-readable medium as above as modules in any manner, and can be used separately or in combination.
While particular implementations and applications of the present disclosure have been illustrated and described, it is to be understood that the present disclosure is not limited to the precise construction and compositions disclosed herein and that various modifications, changes, and variations can be apparent from the foregoing descriptions without departing from the spirit and scope of an invention as defined in the appended claims.
This application claims priority to U.S. Provisional Patent Application No. 62/132,285 filed Mar. 12, 2015, titled “Location Based Services Audio System”, which is incorporated herein by reference in its entirety.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US16/21844 | 3/10/2016 | WO | 00 |
| Number | Date | Country | |
|---|---|---|---|
| 62132285 | Mar 2015 | US |
