Patent Application
20220368450
The present invention relates to a computer, television, radio, mobile phone etc. of a digital audio in which the secret code/data is overlaid and information based on user interaction and user product experience, after the digital sound in question is played with the mobile phone microphone after being played through the speaker in any music and sound system without being affected by the distorting effects of the analog transmission environment and noise by offering the possibility of valuable information acquisition
Due to the developing technology, data protection becomes more and more important day by day. For this reason, important studies have been carried out in recent years basically aiming to increase data security. When the studies in the literature are examined, it is seen that various algorithms have been developed. It is seen that among these algorithms, steganography-based algorithms give successful results. Steganography is a method of data hiding based on the technique of storing data that is desired to be hidden in a cover data. Data hidden in this method is difficult to decode unless it is a decoder. This approach is also defined as the masking or hiding of data in a medium.
Steganography covers many topics such as invisible ink, hidden places, microdots and computer-based methods.
Today, a wide variety of techniques are used to hide information in any media. For this reason, selection of the method to be used to hide the message in a media should be made after the cover environment is selected. Solutions performed in time space are mostly suitable for file-based data transmission and face intolerable high bit error rates (BER) during the air transfer stage of the voice. The least-valued bit (LSB) encoding, echo hiding, and hiding in silence intervals are common examples. On the other hand, among the solutions in frequency space, which also includes the subject of the invention, it is recommended to use LSB coding, phase coding and amplitude coding approaches in discrete wavelet transforms.
In one of the first studies in the state of the art, data transmission is recommended over the acoustic channel and with real-time audio data hiding. This method, which is inspired by Spread Spectrum (SS) stigmatization, utilizes predictive frequency masking properties of the cover sound, and aims for acoustic positioning and navigation applications for closed areas.
In another data hiding method known for its high capacity feature, the masking feature of the human hearing system was used. In the method in which the audio signal is separated into sub-band signals and some of them are selected to hide data, are based on estimates of the all pass filters finite-length impulse responses. This solution, which is especially resistant to synchronization-disruptive attacks, has been accomplished with non-formal (subjective) hearing tests and has been proposed for sound watermarking applications to prevent unauthorized copying.
In a simulation-based study, which performed data hiding in the sound, resistant to the negativities caused by airborne and speech codecs, sub-band amplitude modulation was used. This method, which could not provide high performance compared to human hearing system perception evaluation and BER results up to 15%, was too far from finding application areas in the sector.
Another watermarking approach developed for digital audio copyright protection is based on the Fast Fourier Transform (FFT). In the method where the sound is divided into non-overlapping frames, data is hidden in the selected peaks of the amplitude spectrum of each frame.
In another study where sound watermarking method has been developed in frequency space, it is aimed to put annotations on acoustic files and to place digital signatures on audio signals (for copyright protection, for example). In the approach where data pre-coding (Dirty Paper Codes) and Low Density Parity Control (LDPC) are used in the log-spectrum space to cancel the interference, Walsh functions are used as digital signatures.
In a study aiming for robustness and imperceptibility optimization, a time diffuse eco-based sound stamping approach is presented. The method using a convex optimization based finite impulse response (FIR) filter design to obtain optimal eco filter coefficients is based on a recommended maximum power spectral margin to ensure that it is not felt by the human hearing system.
The acoustic confidential data transmission method has been developed with another study in the Modulated Complex Lapped Transformation (MCLT) space by means of speaker-microphone arrangement. In the method, the data is hidden by arranging the phases of the sound wave MCLT coefficients. This approach, known to be suitable for short message transmissions at distances of less than 1 meter, can provide limited hearing test performance. This solution was adapted subsequently, and a new approach was obtained that provides better sound quality-data communication performance compromises.
Purpose of the Invention:
The invention subject to registration, while a digital sound in which the secret code/data is overlaid is played from the loudspeaker in any environment and device, the user interactions with the mobile phone microphone using mobile app, it aims to provide information based on product experience.
The invention subject to registration is the participation of users in a campaign via mobile phone, increasing the TV/radio viewing rate (rating) and tracking, copyright tracking and management, participating in sweepstakes or voting, distributing instant coupons, authentication and delivering advertisements to target customer groups, etc. can be used in many areas.
The figures for the robust code/data hiding method against analog transmission (over the air) for digital audio, together with a web system, which is integrated with mobile phone (device) interaction and media players are shown below.
The figures for the sample application areas of the developed robust code/data hiding method against analog transmission (over the air) for digital audio are shown below.
The reference numbers shown on the figures of the robust code/data hiding method against analog transmission (over the air) for digital audio are described below:
The invention is generally about obtaining information based on user (audience/audience), user interaction and user product experience from private or public audio or multimedia broadcasts made via computer, television, radio, mobile phone, music and voice systems. In particular, it concerns the realization of code/confidential data transmission over the air (speaker to microphone) in noisy environments within audio or multimedia signals. As explained in detail in
Invention: Within the scope of the general system architecture, which is shown in
For example; the applications of the invention in
The invention technically consists of two main subcomponents. In
Code/Data Hiding Component: After the cover sound file (100), which is transmitted over the air, is obtained in digital environment, numerical data in the frequency band over the hearing frequency limits are superimposed as narrow band frequency regions. The method used is based on the sequential addition of code/data bits to the frequency values on the hearing region boundaries in the spectrum using Fast Fourier Transform—FFT (103) and InverseFastFourier Transform—IFFT (106) with the sampling frequency of 44100 Hz (101) and above. First, a pilot signal for synchronization is superimposed on the sound segment consisting of 128/256 samples, the secret code/data bits (104) to be added for subsequent sound blocks and then the signals of the control bit are added.
Naturally, the intermittent overlay of frequency components not found in the sound on the spectrum leads to a disruptive effect that can be expressed as “sizzling” and audible by the human ear, even outside the hearing site boundaries. Therefore, filtering 107 is performed to soften these artificial noises, which become audible due to sudden transitions after the overlay. The general block diagram of the computerized code/data hiding process that can be realized with the use of signal processing tools is shown in
Marker signal frequency is 19464 Hz, numerical buried data signal frequencies are determined as f1 logic 1 level signal input (203) to the right of this marker signal frequency and f0 logic 0 level signal input (204) to its left. Each coefficient obtained as a result of the FFT process;
It has a frequency value of 44100/256=172,265 Hz.
Marker signal frequency; (44100/2)−(15×172,265)=19464 Hz,
The signal frequency values for binary (0 or 1) values that compose the code/hidden data are;
(44100/2)−(10×172,265)=18604 Hz and (44100/2)−(20×172,265)=20324 Hz.
The marker frequency, the general audio spectrum, that is, the audio frequency region (500) in the audible band and the working frequency region of the invention, and 19 signal frequency values (501) used in the scope of the invention are presented in
19 different frequency values and working frequency region of the method selected for overlaying within the code/data hiding method are shown in
Each frequency scheme (triple group) can be used for a different application. For each frequency scheme, a different named mobile application can be developed, or a mobile application can be used to decode hidden data by using specified signal frequency scheme.
In the receiver application, a Marker signal other than the code/data bits signals to be hidden is used just before the binary code/data bits for easy and reliable code/data analysis; fM=19464 Hz.
Sine signals are used to the right and left of the Marker signal frequency (±5×Δf) for binary code/data bits to be hidden. In other words;
19464 Hz−(5×171,265 Hz)=18604 Hz for “0” and
19464 Hz+(5×171,265 Hz)=20324 Hz for “1”.
The 256-sample sine signal is shown in
The pseudo code of the method of hiding the code/data bits into the “way” audio file data array presented in
The frame format used in the code/data hiding method is shown in
With this method, 3 codes can be carried in 1 second voice (data can be hidden).
Code/Data Decoding Component: The code/data extraction and decoding method components from over the air covered sound are shown in
Microphone (900) of the mobile device/phone audio input is sampled at 44100 Hz. 512 sample buffers (901) are sampled to start. The first 256 samples of this block are taken and FFT is tested to see if the amplitude in the Marker frequency (19464 KHz) exceeds the predefined threshold value. If the threshold is exceeded, this may be a Marker signal. In this case, the starting point of the reference position of the Marker in the frame must be found. In order to synchronize with the code/data bits in time, FFTs are calculated on the existing 512 block of 64 samples right and left of 128 blocks of the first 256 blocks. Whether the marker frequency amplitude is higher on the right or left, 64 samples are headed in that direction, then FFTs are calculated for 32 right and left of this direction. In which direction the Marker frequency amplitude is higher 32 samples on the right or left, in that direction proceeded. Similarly, progress is continued up to 8, dividing by 2 at each step. As a result, a maximum of 8 samples will have the approximate slip and the location of the Marker (reference point). In this case, the data bits will start 256 samples ahead of the starting point. In addition to existing 512 samples, another 256 samples are collected for each bit and the data bits are detected in similar manner. In other words, some of the code/data bits are in this new blocks.
For the next block of 256 samples from the marker starting sample, FFT is performed and the frequency component amplitudes are examined 5×Δf left and 5×Δf right (18604 Hz and 20324 Hz) of the marker frequency. If amplitude of the component on the left is larger than amplitude of the component on the right and above the default threshold level, the data bit for that block is “0”. If amplitude of the component on the right (20324 Hz) is greater than amplitude of the component on the left (18604 Hz) and is above the default threshold level, the data bit of the block is “1”. If amplitude of the larger component does not exceed the predefined threshold value, the Marker may be detected incorrectly, the data format is incompatible, or data is hidden in a different system (or the mobile device/phone microphone, the hidden sound source is too far from the speaker being played). In this case, Marker detection process is restarted. If the code/data bit is found and having sufficiently high amplitude, 256 new samples are sampled for the analysis of the next bits, and the process continues until 8 bits are completed. In the 256-element block at the end of 8 bits, both the Marker and the data bits should not be present (code silence interval).
If this condition is also met, the data is considered valid as shown in
An example of this is the Goertzel algorithm. In addition, frequency estimation techniques can be used with different approaches (instead of FFT) (e.g., periodogram—spectral density estimation). The pseudo code for the secret code/data extraction component from the overlaid (over-the-air) covered voice is given below (on the next page):
As shown in
Mobile Application: The general scheme of mobile device application, including the hidden code/data extraction component, is presented in
Covered voice analysis and hidden code/data decoding (1201) are performed by taking the covered sound via the mobile telephone microphone (900) and processing it with the covered voice receiving and processing component block (1200). Invention's hidden voice analysis and hidden code/data decoding (1201) subcomponents can be realized by means of mobile phone hardware and operating system features, programming tools with libraries especially containing FFT operations. Using the single tap mobile application icon (1204) on the mobile phone (device) screen (1202), the code/data decoding component is activated instantly (in real time) by starting the covered voice receiving. After detection of valid hidden code in the sound mobile device application (1108) interacts with web server (1106). Mobile device application (1108) can be realized on different mobile application development platforms by considering mobile device operating systems (iOS, Android etc.).
Web Management and Presentation Panel: The general web panel, software and sub-components of the data presentation are shown in
By means of the general system management web panel software, campaign management and data presentation (1305), general system management, ad campaign participation and monitoring of the code/data hiding method, sample application model management and data presentation are realized. Campaign definition, which is hiding the code/data equivalent of the cover sound file, is made in the component (code/data—url matching definition) (1306) and when necessary, campaign editing (code/data—url matching change) can be made from the component (1307). All active campaigns can be tracked in the active campaigns (data representation) section (1308) to visualize the interactions of users and for data analysis.
Three preferred application structures for the operation and systematic use of the invention in relation to the computerized code/data hiding, resistant to air transmission of the voice are shown in
The general system, which is one of the application areas of the invention, and the preferred advertising campaign application processes are presented in
With another preferred embodiment, the exemplary usage of the invention is shown in
In
The invention can be used in a wide range of industries ranging from security to entertainment, from authentication to copyright protection. Automatically generate TV/radio/channel view statistics, promote broadcast views, analyze radio/TV ad effectiveness and cost analysis, copyright and activity tracking, participation in campaigns, participation in surveys, participation in sweepstakes, participation in voting, coupon distributions, multimedia file stamping and general or customized identity can be used in verification fields and customer (user) interactivity.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2019/11637 | Aug 2019 | TR | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/TR2020/050665 | 7/28/2020 | WO |
