Patent Application
20240414408
Audio, such as music, may be output through speakers. The audio can be playback of previously-recorded audio or a live performance.
In some implementations, a system includes one or more speakers, one or more cameras, and a control device. The control device may be configured to obtain a sequence of multiple images of a physical area captured by the one or more cameras during playback of audio, via the one or more speakers, at the physical area. The control device may be configured to extract one or more images from the sequence of multiple images, where the one or more images depict one or more people present at the physical area. The control device may be configured to provide the one or more images to a machine learning model, where the machine learning model is trained to determine a change of a volume of the audio, a change of a tempo of the audio, a change of a genre of the audio, or a change of an audio track of the audio based on an input of the one or more images. The control device may be configured to transmit a signal for the one or more speakers to cause the one or more speakers to output an adjustment to the playback of the audio that is based on an output of the machine learning model.
In some implementations, a method may include obtaining, by a device, a sequence of images of a physical area, where at least one image of the sequence of images is captured during playback of audio at the physical area and depicts one or more people present at the physical area. The method may include generating, by the device, a signal in accordance with an output of computer vision processing of the at least one image. The method may include providing, by the device, the signal to audio output hardware to cause an adjustment to the playback of the audio.
In some implementations, a device may include one or more memories and one or more processors, coupled to the one or more memories. The one or more processors may be configured to obtain one or more images of a physical area captured during audio output through a speaker at the physical area, where the one or more images depict one or more people present at the physical area. The one or more processors may be configured to cause, based on the one or more images, an adjustment to at least one of the audio output or a lighting at the physical area.
The following detailed description of example implementations refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements.
An audio system may be used to output audio, such as music, through speakers or other audio output hardware. For example, an audio system may be used at a commercial setting, such as at a bar, a nightclub, or a restaurant, or at a personal setting, such as at a home, to output audio for an audience. Generally, there is a volume range of the audio that will be suitable for all or most members of a given audience. However, this volume range may vary from location to location as well as with respect to different audience compositions. Moreover, inefficiency is associated with playing audio too loud or too soft for a given audience.
For example, an audio system may consume excessive power by playing audio louder than needed or desired by a given audience. As another example, playing audio too softly also results in wasted power consumption, as the audio system is still consuming power, but the audio is not reaching the intended audience. Furthermore, playing audio that an audience finds unappealing results in wasted power consumption, as the audio system is still consuming power, but the audio content is unwanted by the audience. Some implementations described herein provide an audio system that can efficiently and dynamically manage audio output to optimize power consumption of the audio system.
In some examples, a lighting system may be used in connection with the audio system. For example, an optical output of the lighting system may be synchronized with, or otherwise accompany, an audio output of the audio system. Similarly as described above, using lighting that an audience finds unappealing results in wasted power consumption, as the lighting system is still consuming power, but the lighting is unwanted by the audience. Some implementations described herein provide a lighting system that can efficiently and dynamically manage lighting (e.g., optical output) to optimize power consumption of the lighting system.
The control device may be used to adjust audio and/or lighting at a physical area. The physical area may be a bar, a night club, a restaurant, a dance floor, a movie theater, or a house, among other examples. In some implementations, the control device may be communicatively coupled (e.g., wirelessly or using wires) with one or more cameras present at a physical area or otherwise directed at the physical area (e.g., to enable the control device to control and/or receive data from the one or more cameras). In some implementations, the control device may share a housing with a camera. In some implementations, the control device may be communicatively coupled (e.g., wirelessly or using wires) to one or more lighting devices (e.g., single light units and/or light arrays) present at the physical area or otherwise directed at the physical area (e.g., to enable the control device to control the one or more lighting components). In some implementations, the control device may share a housing with a lighting device. In some implementations, the control device may be communicatively coupled (e.g., wirelessly or using wires) to one or more speakers present at the physical area or otherwise directed at the physical area (e.g., to enable the control device to control or transmit signals to the one or more speakers). In some implementations, the control device may share a housing with a speaker. In some implementations, a system may include one or more speakers, one or more cameras, one or more lighting devices, and/or the control device.
In some implementations, the control device may store a plurality of audio tracks (e.g., audio files) and/or the control device may include an interface to enable the control device to load or otherwise access audio tracks stored on another device. The audio tracks may be associated with metadata indicating, for each audio track, a title of the audio track, a genre of the audio track, a playback time of the audio track, a tempo (e.g., a predominant tempo) of the audio track, a musical key (e.g., a predominant musical key) of the audio track, and/or a lighting configuration for the audio track. In some implementations, the control device may generate the metadata (e.g., indicating playback times, tempos, and/or musical keys) by scanning the audio tracks. For example, the control device may scan an audio file using an audio analysis technique that includes analyzing a waveform associated with the audio file. The audio analysis technique may include analyzing the waveform of the audio file to identify time points where beats occur to thereby identify tempo (which may be referred to as “beat detection”). Additionally, or alternatively, the audio analysis technique may include analyzing a harmonic content of the audio file to identify a predominant musical key (which may be referred to as “key detection”).
As shown in
As shown in
As shown in
The machine learning model may perform computer vision processing of the one or more images. The computer vision processing may include processing of the one or more images in connection with object detection, motion analysis, facial recognition, facial emotion detection, and/or object tracking, among other examples. In some implementations, the machine learning model may be trained to identify, in the one or more images, a crowd density of the one or more people depicted in the one or more images. For example, the crowd density may indicate a closeness of the people to one another. Additionally, or alternatively, the machine learning model may be trained to identify, in the one or more images, positions of the one or more people relative to one or more speakers of the physical area. For example, a person standing further from a speaker may indicate that a volume of the audio is too loud, whereas a person standing closer to a speaker may indicate that a volume of the audio is too soft. Additionally, or alternatively, the machine learning model may be trained to identify, in the one or more images, movement intensity levels of the one or more people. For example, a movement intensity level of a person may indicate a speed at which the person is moving, a speed at which arms (e.g., hands) and/or legs (e.g., feet) of the person are moving, a distance that the person travels over a particular time period, and/or a distance that arm movements and/or leg movements of the person travel over a particular time period. Additionally, or alternatively, the machine learning model may be trained to identify, in the one or more images, interaction proximities between the one or more people. For example, an interaction proximity may indicate a distance between two or more people that are interacting with each other (e.g., speaking with each other, dancing with each other, or the like). Additionally, or alternatively, the machine learning model may be trained to identify, in the one or more images, facial expression-based sentiments of the one or more people. For example, a facial expression-based sentiment of a person may indicate whether a person appears happy (e.g., because the person is smiling), whether a person appears bored (e.g., because the person is yawning), or the like.
Additionally, or alternatively, the machine learning model may be trained to identify, in the one or more images, physical characteristics of the one or more people (e.g., ages of the one or more people, genders of the one or more people, or the like). For example, the physical characteristics of the one or more people may indicate music genre preferences of the one or more people. Additionally, or alternatively, the machine learning model may be trained to identify, in the one or more images, clothing styles of the one or more people. For example, the clothing styles of the one or more people may indicate music genre preferences of the one or more people. Additionally, or alternatively, the machine learning model may be trained to identify, in the one or more images, dancing styles of the one or more people. For example, the dancing styles of the one or more people may indicate music genre preferences of the one or more people. Additionally, or alternatively, the machine learning model may be trained to identify, in the one or more images, characteristics of the physical area (e.g., a style of furniture, a decoration style, a luxuriousness, or the like). For example, the characteristics of the physical area may indicate a music genre preference of customers of the physical area or an appropriate volume for the physical area. Additionally, or alternatively, the machine learning model may be trained to identify noises made by the one or more people (e.g., laughing, talking, singing, or the like) and/or noise-based sentiment of the one or more people (e.g., singing along with the audio may indicate that the one or more people are satisfied with the audio).
The aforementioned variables that the machine learning model may identify in the images may be a feature set used by the machine learning model. Accordingly, an output of the machine learning model may be based on the feature set.
The machine learning model may use machine learning algorithms to process the one or more images and extract relevant features from the one or more images. The machine learning model may be trained using a set of training data. The training data may relate to the physical area and/or may relate to multiple different physical areas (e.g., a network of physical areas). The training data may include images and/or video of people at a physical area (e.g., a bar, a nightclub, or the like) during playback of audio. The images and/or video may be manually labeled (e.g., in connection with supervised learning) based on whether the people appear to be enjoying the audio. Moreover, the training data may include metadata associated with the images and/or video, such as data indicating a volume level of the audio playback, one or more characteristics of the audio playback (e.g., a music genre of the audio, the particular audio track being played, a musical key of the audio, a tempo of the audio, or the like), and/or a revenue amount associated with the physical area during playback of the audio (e.g., point-of-sale data), among other examples. The training data may be used to train the machine learning model, which may be based on various algorithms such as convolutional neural networks (CNNs), support vector machines (SVMs), or decision trees. In some implementations, the machine learning model may be trained using an unsupervised learning technique. In one example, the machine learning model may be a CNN.
The trained machine learning model may be used to analyze new images or video of people present at a physical location during playback of audio, such as the one or more images described herein. The machine learning model may detect and track the faces and bodies of the people in the images or video using techniques such as object detection and tracking. The machine learning model may analyze the detected faces and bodies to determine a crowd density (e.g., whether the crowd density is increasing or decreasing) and/or interaction proximities between people (e.g., whether interaction proximities are increasing or decreasing). Furthermore, the machine learning model may analyze facial expressions and/or body movements of each person to determine movement intensity levels (e.g., whether movement intensity levels are increasing or decreasing), sentiment (e.g., whether sentiment is becoming more positive or more negative), and/or dancing styles. Facial expression analysis may use techniques such as facial landmark detection and/or emotion recognition to identify key features such as smiles, frowns, and eyebrow movements. Body movement analysis may involve detecting changes in posture or movement patterns, such as head nodding or foot tapping. Additionally, the machine learning model may analyze the detected faces and bodies to determine physical characteristics of the one or more people and/or clothing styles of the one or more people. The machine learning model may also analyze audio data that accompanies the images/video (or audio data by itself) to identify particular noises (e.g., talking, laughing, or singing), noise-based sentiment (e.g., booing noise may indicate dissatisfaction), and/or a noise volume. The machine learning model may combine these features to make a prediction about whether the people are satisfied or dissatisfied with the audio (e.g., with the genre of the audio, with the particular audio track, with a volume of the audio, with a musical key of the audio, and/or with a tempo of the audio), whether the audio (e.g., a genre of the audio, a volume of the audio, a tempo of the audio, or the like) is appropriate for the physical area, or whether the audio is more likely to increase or decrease revenue of the physical area. The machine learning model may be trained to determine a change of a volume of the audio, a change of a tempo of the audio, a change of a key of the audio, a change of a genre of the audio, and/or a change of an audio track of the audio based on an input of the one or more images. Additionally, or alternatively, the machine learning model may be trained to determine a change to lighting based on the input of the one or more images.
An output of the machine learning model (e.g., of the computer vision processing) may indicate a suitability of the audio for the physical area (e.g., may indicate whether an audience at the physical area is responding favorably or unfavorably to the audio, or may indicate whether a revenue associated with the physical area is more likely to increase or decrease in connection with the audio). For example, the output may include a score, a set of scores, or another metric or metrics indicating the suitability of the audio. In some implementations, an output of the machine learning model (e.g., of the computer vision processing) may indicate a recommendation of an adjustment to the audio (e.g., based on a suitability of the audio for the physical area). For example, the recommendation may include a recommendation of a volume or a volume change of the audio, a recommendation of a tempo or a tempo change of the audio, a recommendation of a musical key or a musical key change of the audio, a recommendation of a musical genre, a recommendation of an audio track, and/or a recommendation of an audio effect. Additionally, or alternatively, an output of the machine learning model (e.g., of the computer vision processing) may indicate a recommendation of an adjustment to a lighting of the physical area (e.g., based on a suitability of the audio for the physical area). For example, the recommendation may include a recommendation of a light intensity or a light intensity change of the lighting, a recommendation of a color or a color change of the lighting, a recommendation of a lighting effect or a lighting effect change of the lighting, a recommendation of a beam width or a beam width change of the lighting, a recommendation to initiate or to stop spotting of the lighting, and/or a recommendation of a movement pattern or a movement pattern change of the lighting.
As shown in
In some implementations, the control device may cause the adjustment to the playback of the audio and/or the lighting in accordance with a recommendation output by the machine learning model (e.g., the control device may select an audio track based on a music genre recommended by the machine learning model). In some implementations, the control device may determine the adjustment to the playback of the audio and/or the lighting based on the output of the machine learning model (e.g., based on a score(s) or a metric(s)). In some implementations, the control device may determine the adjustment as a function of the output (e.g., using an algorithm). For example, based on the machine learning model outputting a first score, the control device may increase the volume of the playback of the audio, and based on the machine learning model outputting a second score, the control device may decrease the volume of the playback of the audio. In some implementations, the control device may determine the adjustment to the playback of the audio using an additional machine learning model. For example, the control device may provide the output of the machine learning model (e.g., a score(s) or a metric(s) indicating a suitability of the audio for the physical area) to the additional machine learning model, and the additional machine learning model may output a recommendation of the adjustment to the playback of the audio and/or to the lighting.
In some implementations, the control device may cause a list of adjustment options to be presented on a display and/or the control device may transmit, to a user device, a message indicating the list of adjustment options. For example, if the output of the machine learning model indicates a recommendation of a music genre, the list of adjustment options may be a list of audio tracks associated with the music genre. In some implementations, the list may also indicate one or more success predictions (e.g., indicating a probability that the option will be suitable for the physical area) for each adjustment option (e.g., an adjustment option may have multiple success predictions for different levels of data, such as a platform-wide success prediction, a regional success prediction, or the like). The control device may receive an indication indicating a selection of an adjustment option from the list of adjustment options. Accordingly, the control device may cause the adjustment to the playback of the audio in accordance with the selection.
To cause adjustment of the audio, the control device may generate and provide (e.g., transmit) a signal (e.g., an electrical signal or a radio signal) to audio output hardware to cause the audio output hardware to output the audio in accordance with the adjustment. For example, the signal may indicate the adjustment or may correspond to the adjusted audio. As an example, the signal may cause the audio output hardware to output an adjustment to the audio that is based on the output of the machine learning model. The audio output hardware may be a speaker, a mixer, an amplifier, or the like. To cause adjustment of the lighting, the control device may generate and provide (e.g., transmit) a signal (e.g., an electrical signal or a radio signal) to a lighting device (e.g., a device that controls a light). For example, the signal may cause the lighting device to adjust the lighting at the physical area based on the output of the machine learning model. As an example, the lighting device may provide Internet of Things (IoT) capability or Bluetooth control of the light.
In some implementations, the control device may cause adjustment to the playback of the audio and/or the lighting based on audience feedback. For example, the audience may be informed of gestures (e.g., hand gestures, head gestures, facial expressions, or the like) that indicate particular feedback regarding the playback of the audio. As an example, a first gesture may indicate feedback to raise a volume, a second gesture may indicate feedback to lower a volume, a third gesture may indicate feedback to change the audio to a particular genre, a fourth gesture may indicate feedback to increase a tempo of the audio, a fifth gesture may indicate feedback to lower a lighting level, and so forth. In some examples, the audio may be requested to vote on whether to make a particular adjustment to the playback of the audio and/or the lighting by making a particular gesture, by standing at a particular area, or by dancing in a particular style, among other examples. As an example, audience members may vote for the audio to be a first genre by standing to a left side of the physical area, and other audience members may vote for the audio to be a second genre by standing to a right side of the physical area.
In a similar manner as described above, the control device may obtain a sequence of images, and the control device may extract one or more images, from the sequence of images, that depict one or more people present at the physical area In a similar manner as described above, the control device may provide the one or more images to a machine learning model (e.g., the same machine learning model described above, or a different machine learning model), and the machine learning model may perform computer vision processing of the one or more images. In some implementations, the machine learning model may be trained to identify, in the one or more images, gestures being made by the one or more people. For example, the machine learning model may identify whether one or more particular gestures are being made or a quantity of people that have made each particular gesture (e.g., concurrently or within a particular time window). Additionally, or alternatively, the machine learning model may identify standing locations of the one or more people. For example, the machine learning model may identify votes of the one or more people based on the standing locations. Additionally, or alternatively, the machine learning model may identify dancing styles of the one or more people. For example, the machine learning model may identify votes of the one or more people based on the dancing styles.
An output of the machine learning model (e.g., of the computer vision processing) may indicate one or more gestures that are identified, a quantity of each gesture that is identified, and/or one or more votes that are identified. In a similar manner as described above, the control device may cause an adjustment to the playback of the audio and/or the lighting. For example, the control device may cause the adjustment to the playback of the audio and/or the lighting based on the output of the machine learning model (e.g., of the computer vision processing). In some implementations, the control device may cause the adjustment to the playback of the audio and/or the lighting based on a particular gesture being identified, based on a threshold quantity of the particular gesture being identified, and/or based on a quantity of votes (e.g., a majority quantity of votes or a threshold quantity of votes) identified.
Techniques described above relating to audio adjustment may be adapted for use for lighting adjustment, and techniques described above relating to lighting adjustment may be adapted for use for audio adjustment. Furthermore, techniques described above relating to adjustment of the playback of audio may be adapted for use in adjustment of the playback of live audio.
As indicated above,
The control device 210 may include one or more devices capable of receiving, generating, storing, processing, and/or providing information associated with audio or lighting adjustment, as described elsewhere herein. The control device 210 may include a communication device and/or a computing device. For example, the control device 210 may include a wireless communication device, a mobile phone, a user equipment, a laptop computer, a tablet computer, a desktop computer, a wearable communication device (e.g., a smart wristwatch, a pair of smart eyeglasses, a head mounted display, or a virtual reality headset), or a similar type of device. Additionally, or alternatively, the control device 210 may include a server, such as an application server, a client server, a web server, a host server, a proxy server, a virtual server (e.g., executing on computing hardware), or a server in a cloud computing system.
The speaker 220 may include one or more speakers capable of outputting sound based on an audio signal. The speaker 220 may include a communication device and/or a computing device. The lighting device 230 may include one or more light emitting devices. For example, the lighting device 230 may include one or more spotlights, one or more strobe lights, one or more moving-head lights, or the like. The lighting device 230 may include a communication device and/or a computing device. The camera 240 may include one or more devices capable of capturing a digital image. The camera 240 may include a communication device and/or a computing device. In some implementations, the control device 210 may include, may be included in, or may be included in a system with, the camera 240.
The remote device 250 may include one or more devices capable of receiving, generating, storing, processing, providing, and/or routing information associated with a machine learning model. The remote device 250 may include a communication device and/or a computing device. For example, the remote device 250 may include a server, such as an application server, a client server, a web server, a database server, a host server, a proxy server, a virtual server (e.g., executing on computing hardware), or a server in a cloud computing system. In some implementations, the remote device 250 may include computing hardware used in a cloud computing environment.
The network 260 may include one or more wired and/or wireless networks. For example, the network 260 may include a wireless wide area network (e.g., a cellular network or a public land mobile network), a local area network (e.g., a wired local area network or a WLAN, such as a Wi-Fi network), a personal area network (e.g., a Bluetooth network), a near-field communication network, a telephone network, a private network, the Internet, and/or a combination of these or other types of networks. The network 260 enables communication among the devices of environment 200.
The number and arrangement of devices and networks shown in
The bus 310 may include one or more components that enable wired and/or wireless communication among the components of the device 300. The bus 310 may couple together two or more components of
The memory 330 may include volatile and/or nonvolatile memory. For example, the memory 330 may include random access memory (RAM), read only memory (ROM), a hard disk drive, and/or another type of memory (e.g., a flash memory, a magnetic memory, and/or an optical memory). The memory 330 may include internal memory (e.g., RAM, ROM, or a hard disk drive) and/or removable memory (e.g., removable via a universal serial bus connection). The memory 330 may be a non-transitory computer-readable medium. The memory 330 may store information, one or more instructions, and/or software (e.g., one or more software applications) related to the operation of the device 300. In some implementations, the memory 330 may include one or more memories that are coupled (e.g., communicatively coupled) to one or more processors (e.g., processor 320), such as via the bus 310. Communicative coupling between a processor 320 and a memory 330 may enable the processor 320 to read and/or process information stored in the memory 330 and/or to store information in the memory 330.
The input component 340 may enable the device 300 to receive input, such as user input and/or sensed input. For example, the input component 340 may include a touch screen, a keyboard, a keypad, a mouse, a button, a microphone, a switch, a sensor, a global positioning system sensor, an accelerometer, a gyroscope, and/or an actuator. The output component 350 may enable the device 300 to provide output, such as via a display, a speaker, and/or a light-emitting diode. The communication component 360 may enable the device 300 to communicate with other devices via a wired connection and/or a wireless connection. For example, the communication component 360 may include a receiver, a transmitter, a transceiver, a modem, a network interface card, and/or an antenna.
The device 300 may perform one or more operations or processes described herein. For example, a non-transitory computer-readable medium (e.g., memory 330) may store a set of instructions (e.g., one or more instructions or code) for execution by the processor 320. The processor 320 may execute the set of instructions to perform one or more operations or processes described herein. In some implementations, execution of the set of instructions, by one or more processors 320, causes the one or more processors 320 and/or the device 300 to perform one or more operations or processes described herein. In some implementations, hardwired circuitry may be used instead of or in combination with the instructions to perform one or more operations or processes described herein. Additionally, or alternatively, the processor 320 may be configured to perform one or more operations or processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.
The number and arrangement of components shown in
As shown in
As shown in
As shown in
Although
The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications may be made in light of the above disclosure or may be acquired from practice of the implementations.
Although particular combinations of features are recited in the claims and/or disclosed in the specification, these combinations are not intended to limit the disclosure of various implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and/or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of various implementations includes each dependent claim in combination with every other claim in the claim set. As used herein, a phrase referring to “at least one of” a list of items refers to any combination and permutation of those items, including single members. As an example, “at least one of: a, b, or c” is intended to cover a, b, c, a-b, a-c, b-c, and a-b-c, as well as any combination with multiple of the same item. As used herein, the term “and/or” used to connect items in a list refers to any combination and any permutation of those items, including single members (e.g., an individual item in the list). As an example, “a, b, and/or c” is intended to cover a, b, c, a-b, a-c, b-c, and a-b-c.
No element, act, or instruction used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items, and may be used interchangeably with “one or more.” Further, as used herein, the article “the” is intended to include one or more items referenced in connection with the article “the” and may be used interchangeably with “the one or more.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, or a combination of related and unrelated items), and may be used interchangeably with “one or more.” Where only one item is intended, the phrase “only one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based, at least in part, on” unless explicitly stated otherwise. Also, as used herein, the term “or” is intended to be inclusive when used in a series and may be used interchangeably with “and/or,” unless explicitly stated otherwise (e.g., if used in combination with “either” or “only one of”).
A set of functions described herein as being performed by a processor or by one or more processors may be performed individually by a single processor or may be performed collectively by multiple processors (e.g., one or more first functions may be performed by one or more first processors and one or more second functions may be performed by one or more second processors). A set of functions described herein as being performed by a machine learning model or by one or more machine learning models may be performed individually by a single machine learning model or may be performed collectively by multiple machine learning models (e.g., one or more first functions may be performed by one or more first machine learning models and one or more second functions may be performed by one or more second machine learning models).
