Consumers of media often experience the media in a manner where it has been recorded live in the form of videos and/or audio files. This media may be pre-recorded or streamed in real time, close to real time, with some amount of latency, downloaded for offline usage, or a combination of the aforementioned methods among other methods. Generally, however, the consumers of such media have little or no control over their sensory experience during the viewing to, listening to, or engaging with the performance or session. While such performances may be recorded with multiple microphones and/or cameras in order to create such effects as stereo or surround sound, the listener is generally given one or limited choices on their experience which is generally dissimilar to the experience of physically being present in the venue where the event takes place.
It is desirable to have systems and methods that allow for a user to experience a sensory environment such as a musical performance, virtually from any location within various types of spaces, for example, a stage. Such systems and methods may allow, for example, a listener to be “virtually” situated in any part of the stage of an orchestra. Perhaps they wish to be able to listen to and feel what a particular violinist, trumpet player or other on-stage performer is experiencing during the performance, or perhaps they wish to experience what the conductor is experiencing. There may even be a desire to experience the performance in a way that would not have been physically possible even if attending in person. Such sensory experiences are not limited to auditory senses, but may extend to other senses such as touch, taste, sight, or smell.
In one implementation, a server may receive data from one or more remote sensing devices (RSDS). The RSDS are also described in co-pending application number 14/629,312, which is hereby incorporated by reference. The RSDS may be distributed throughout a designated space. The server may receive a request for at least some portion of the information received from the RSDS. The server may then send all or at least some portion of the information to the requestor. The content of the information may be based on the request. Further, the information may be comprised of audio information and/or positional information of each RSD. The information may also be in the form of machine and/or human analysis results including musical note duration, pitch, dynamics, and other data measureable, extractable, and inferable from the data captured by the RSDS. Human analysis data may further be provided—data that is difficult to quantify with machines such as emotive descriptions. The information sent to the requestor may be a custom audio mix based on the request. The creation of the custom audio mix may be based on a position selected within the designated space or a position based on any location within the sensing range of one or more of the RSDS. This may include focusing on the brass section or the inverse—“removing” the brass section and focusing on all of the other, non-brass instruments. Any data transmission combination may be possible and any data type may be transmittable, including audio data, data from machine analyses, data from human annotations, visualizations, data extracted from RSDS, a combination of the aforementioned, and other data modalities from libraries and/or from other databases including data from various sources on the Internet.
In another implementation, the server creates a custom audio mix based on the selection of one or multiple “observation” positions within a space or a number of spaces. In another implementation, the sonic delays and dynamic attenuation are calculated to model and fold-in acoustic delay and acoustic energy dissipation based on the distance of each of the RSDS to the position selected within the space. The position or positions may also be based on any location within the sensing range of one or more of the RSDS.
In another implementation, the server may send information back to at least one of the RSDS or to all of the RSDS. The instruction may provide configuration information to the RSDS or result in the generation of a sound impulse by one or more of the RSDS.
In another implementation, the server may send an instruction to each RSD in turn to generate a sound impulse with the remaining RSDS capturing and sending the resulting audio information back to the server. The server may use the resulting audio information gathered to determine the relative position of each of the RSDS to the other RSDS or to determine the approximate relative position of each of the RSDS to the other RSDS. RSDS may also in part or fully contribute to the computations necessary for the sensor network and client interaction, thereby providing and distributed computing design for effective, efficient, and robust signal processing and environmental sensing.
In another implementation, the server creates a custom audio mix based on the selection of a position within a space, where the target position can coincide with the position of one of the coordinates of existing RSDS or any other location by calculating the acoustic delay and acoustic energy dissipation based on the distance of each of the RSDS to the position selected. A simple example: if a target observation position is on a “line” and between two RSDS (RSD_left and RSD_right), 50% of each RSD will contribute to the custom mix with appropriate delay as computed via linear or non-linear distance calculation algorithms from each RSD. The resulting mix can be computed by considering acoustic energy dissipation and delay as a function of distance, temperature, humidity, as well as other spatial information. Utilizing the multiple audio signals and channels, instrument isolation may also be implemented. Using information from surrounding RSDs, any particular RSD's signal may be “soloed” by using source-separation techniques where, for example, the sound of violin A may be isolated by taking into account the RSD A and its neighboring RSDs, creating a “solo” performance of violin A.
In another implementation, one embodiment is a server (or plurality of servers) and a plurality of RSDs distributed in a space. Each RSD may be comprised of a sensor, multiple types of sensors, such as a microphone and have a data connection to the server and/or between RSDs. Each RSD may be capable of sending data—such as audio data—over the data connection to the server and/or between RSDs.
In another implementation, the RSD data, including audio signals may be synchronized using a master timestamp—e.g. generated on a server. Participating RSDs will synchronize to this master timestamp/clock, which is broadcast to RSDs, allowing accurate temporal RSD alignment. Synchronization may be as part of a sensor network setup sequence where, before audio capturing/streaming occurs, network latency between each individual RSD and server, as well as bandwidth, are considered. Synchronization may also be continually adjusted during the recording/streaming phase.
In another implementation, one embodiment is a server (or plurality of servers) and plurality of RSDs distributed in a space. Each RSD may be comprised of two or more microphones forming a microphone array and have a data connection to the server. Each RSD may be capable of sending audio data over the data connection to the server in full duplex format. The microphones may be independently configurable and adjustable to provide custom directionality. The RSDs may further have an attached clamp. The clamp may be configured to attach to a music stand with a locking mechanism. The locking mechanism may be a rotating locking mechanism that will afford a secure attachment to a music stand. The locking mechanism may contain a spring system to allow for flexible adjustment of the clamp for different thickness surfaces, e.g., different music stands. The clamp may be configured to attach magnetically. The clamp may attach via a screw, e.g. a screwed connection to a music stand. The clamping mechanism may be configured to attach to various portions of a music stand. The clamp may also be attached with other non-permanent and permanent adhesives such as hook-and-loop fasteners, VELCRO®.
In another implementation, one embodiment is a server (or plurality of servers) and a plurality of RSDS distributed in a space. Each RSD may be comprised of one or more microphones and has a data connection to the server. Each RSD may be capable of sending audio data over the data connection to the server in full duplex format. Each RSD may contain a microprocessor, a communication module, various I/O, and a power source. The communication module is configured to communicate using the data connection to the server. Each RSD may contain one or more loudspeakers. The data connection between each RSD and the server may be wireless.
The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the following drawings and the detailed description.
The foregoing and other features of the present disclosure will become more fully apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. Understanding that these drawings depict only several embodiments in accordance with the disclosure and are, therefore, not to be considered limiting of its scope, the disclosure will be described with additional specificity and detail through use of the accompanying drawings.
In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented here. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are explicitly contemplated and made part of this disclosure.
In one implementation, a server may receive from one or more remote sensing devices' (RSDS) data and information.
In another implementation, a ubiquitous listening environment is constructed in the representative environment 100 allowing a listener to experience a musical performance—e.g. orchestra, string quartet, or rock band, etc.—from virtually “any” location on the stage. This implementation includes a single or multiple RSDS 120 and one or more servers 130 that receive audio and/or other measurements from the wireless (and/or wired) RSDS 120, which are custom and/or interactively mixed, and finally sent to clients with custom “mixes.” The RSD nodes 120 provide a close-proximity capture of audio signals from the musician behind the music stand, for example, via a single or array of microphones (and/or other sensors). One setup is shown in
An example client interface is shown in
In one implementation, the listening locations may be in the form of an Internet interface of a web-browser, standalone software application, hardware implementation, or a combination of other interaction solutions. For example, in
In another implementation, a user interface for adjusting and monitoring active RSDS is shown in
The gain knob 530 controls the input gain to the microphone sensor, which remotely controls the microphone amplification gain on the RSD side. The RSD checkbox 540, turns on/off an RSD and a row of LEDs 550 indicates online status of a given RSD and its number of microphones with one LED for each microphone (shown here with four microphones per RSD).
In another implementation, the acoustic environment in a given location is simulated by considering all of the active RSD signals, their location (angle and distance), artificial acoustic delay governed by parameters such as distance (inverse square law), speed of sound, temperature, humidity, and architectural details governing given space and selected observation location. That is, the audio output will be a sum of all RSD signals at a given location by considering distance, angle, energy dissipation as well as other parameters including ones outlined above. For example, as temperature may change over the course of a performance, computation of custom mix may also change accordingly.
Returning to
In another implementation, the server 130 may send information back to at least one of the RSDS 120 or to all of the RSDS 120. The instruction may provide configuration information to the RSDS 120 or result in the generation of a sound impulse by one or more of the RSDS. The server 130 may send an instruction to each RSD 120 in turn to generate a sound impulse with the remaining RSDS 120 sending the resulting audio information back to the server 130. The server 139 may use the resulting audio information gathered to determine the relative position of each of the RSDS 120 to the other RSDS 120 or to determine the approximate relative position of each of the RSDS 120 to the other RSDS 120.
In another implementation, the server 130, shown in
As shown in
In one implementation, the system and method described herein are configured for use in an audio-visual setting such as a sporting event. For example, a system could be used at the US Open for example (tennis) or other sports such as baseball, basketball, etc. A further implementation applies the system and methods described herein to remote learning, such as Internet based learning, not just for music but classrooms of every type including dance classes, art classes, traditional classes etc. In tennis, for example, the microphones can be used to determine what kind of spin is being used and also capture the vibe of the stadia. Sports that involve smaller playing surfaces or venues would be readily adaptable to the use of the technology. For example, ping-pong, others include billiard tables, etc. The idea is that any sport that has tables etc. as part of it would be an easy go-to application area.
In another implementation, a dynamic and flexible distributed sensor network is created whereby the RSDS in the sensor network actively participate in not just processing and computing its captured data but also data captured from other RSDS. In one particular implementation, the audio mix that is requested by a client is mixed fully or partially by the RSDS in the sensor network. In this example, one RSD may receive one or more audio data (and other data) from neighboring RSDS to create a subm ix as requested by the client. In this scenario, the RSDS in the sensor network may receive a submix that represent a submix of two or more RSDs which in turn will create a submix that represents a larger submix of RSDs. This allows for significant server bandwidth reduction as only a subset of RSDs (or in extreme cases) one RSD will send the overall mix of sensor network RSDs to the server. The server will then provide the final mix to the requester.
In another implementation, an RSD may be in the form of an off-the-shelf handheld device, such as a smartphone equipped with an internal microphone or high-quality external add-on microphone. In this scenario, the devices enable the creation of a virtual studio where the signals captured by the devices (RSDs) are synchronized, form a sensor network, and stream data to the server (for example, to the “cloud”). The user(s) can then access the individual audio tracks, edit, mix, and manipulate them in the cloud environment bypassing the need for the traditional digital audio workstation (DAW). In this implementation, a virtual studio is created whereby the RSDs provide the technology and means to capture high-quality audio (and any other signal depending on sensor attached), stream data to a server or multiple servers, and allow user access to the data through a standard web-browser. Further, the user would be able to download mixed, individual, and/or processed tracks, metadata, and other data such as control data to a local computer for additional editing.
As shown in
System 1000 may also include a display or output device, an input device such as a key-board, mouse, touch screen or other input device, and may be connected to additional systems via a logical network. Many of the embodiments described herein may be practiced in a networked environment using logical connections to one or more remote computers having processors. Logical connections may include a local area network (LAN) and a wide area network (WAN) that are presented here by way of example and not limitation. Such networking environments are commonplace in office-wide or enterprise-wide computer networks, intranets and the Internet and may use a wide variety of different communication protocols. Those skilled in the art can appreciate that such network computing environments can typically encompass many types of computer system configurations, including personal computers, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. Embodiments of the invention may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination of hardwired or wireless links) through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
Various embodiments are described in the general context of method steps, which may be implemented in one embodiment by a program product including computer-executable instructions, such as program code, executed by computers in networked environments. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.
Software and web implementations of the present invention could be accomplished with standard programming techniques with rule based logic and other logic to accomplish the various database searching steps, correlation steps, comparison steps and decision steps. It should also be noted that the words “component” and “module,” as used herein and in the claims, are intended to encompass implementations using one or more lines of software code, and/or hardware implementations, and/or equipment for receiving manual inputs.
With respect to the use of substantially any plural and/or singular terms herein, those having skill in the art can translate from the plural to the singular and/or from the singular to the plural as is appropriate to the context and/or application. The various singular/plural permutations may be expressly set forth herein for the sake of clarity.
The foregoing description of illustrative embodiments has been presented for purposes of illustration and of description. It is not intended to be exhaustive or limiting with respect to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the disclosed embodiments. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.
This application claims the benefit of U.S. Provisional Application No. 62/209,542, filed on Aug. 25, 2015, which is hereby incorporated by reference in its entirety.
Number | Date | Country | |
---|---|---|---|
62209542 | Aug 2015 | US |