Patent Application
20230205926
The embodiments of the present disclosure generally relate to systems and methods for video streaming, and in particular using artificial Intelligence for overcoming privacy threat while video streaming.
The following description of related art is intended to provide background information pertaining to the field of the disclosure. This section may include certain aspects of the art that may be related to various features of the present disclosure. However, it should be appreciated that this section be used only to enhance the understanding of the reader with respect to the present disclosure, and not as admissions of prior art.
For video streaming pipelines, integration of deep-Learning models into a framework for streaming media gives edge over standard methods. For the new age technological capabilities, the streaming media provides inexpensive and widely deployed cameras, increase in edge to cloud networking capacities, increase in computation capacities and options, and advancements in Deep-Learning networks with increase in accuracy. However, with these advancements, the probability of leakage of the crucial user's data also gets increased with the ease of access and use of technology. In case of video analytics, where a lot of biometric data specific to the users gets exposed which may lead to inappropriate usage of this data for attack purposes called as adversarial attacks. Privacy, being one of the very important aspects to be taken care of while designing the systems for ease of clients, needs to be seriously taken into consideration.
A prior art discloses a system that secures data pipeline which is involved in video stream data processing using ML techniques. It used a hybrid execution environment spanning both CPU and GPU. Used data oblivious algorithms to protect against leakage via memory access pattern side channel attacks. Another prior art proposes a new algorithmic technique for learning and a refined analysis of privacy cost within the framework of differential privacy. It demonstrates differential privacy-based technique that can be applied while training deep neural networks with non-convex objectives that too under modest privacy budget and cost in software complexity, training efficiency and model quality.
Another prior art discloses a differential privacy using stochastic gradient descent method. It preserves the sensitive information by limiting privacy loss obtained by log of prediction w.r.t. average score for a particular class. But it does not perform well of complex dataset. Another prior art discloses a improve privacy, this paper has proposed a method to minimize the number of trainable parameters. They had proposed a transfer learning paradigm that finetunes a space subnetwork with differential privacy. One prior art demonstrates denaturing the video to make it safe from a privacy point of view. It is basically blurring the videos while another prior art targets to de-identify a range of hard and soft biometrics which remain recognizable even after applying blurring. In another prior art, they have combined background subtraction based on gaussian mixtures with improved algorithms to find and segment pedestrians and de-identifying is done by altering the appearance of the segmented pedestrians through neural art algorithm which uses responses of DNN to render the pedestrian images in different style.
One prior art discusses systems and techniques to identify and prevent certain fraud attacks that may be used to defeat facial recognition systems. They proposed a concept of likeness scores that are determined for separate regions that biometric data can be segregated into. By tracking individual likeness scores used in access requests their system is able to detect potential fraud attacks. Another prior art proposes stem and method for facial anti-spoofing by determining a focus value. Focus values represents a point at which the image is sharp which in this case corresponds to a face within an image. Identity check are used everywhere to authenticate user and provide a better user experience. Manual identity check mechanisms often result in high cost of operations, inadequate protection of consumer privacy, and significant potential of fraud. This patent has proposed design a digital identity platform that enables relying parties to perform automatic consumer identity checks using data supplied by identity providers (e.g., DMVs, issuers) both in online and in-store settings. The platform uses advanced cryptographic techniques to enable streamlined identity check operations at scale, protects consumer privacy and reduces potential for fraud.
One prior art provides a system and method for video broadcasting applied to the client side in HLS Internet video play systems while another prior art provides a method named PATE (private aggregation of teacher ensembles) that demonstrate a black box fashion method for trained models on similar type datasets. In this, multiple trained models (Teachers) are used to provide an aggregated information on the new instance because of which the inference (student) cannot access parameters from a specific model and thus the sensitive information access is avoided.
These existing payment solutions rely on either personal device or server-based biometric recognition methods. While these methods work well in practice, they typically trade off either privacy, security or convenience for the user. These articles propose privacy-preserving biometric systems that use advanced cryptographic techniques such as homomorphic encryption and secure multi-party computation to enable secure and seamless payment experiences for users.
There is therefore a need in the art to provide a system and a method that can facilitate integrate privacy preserving components and mitigate the problems associated with the prior art.
Some of the objects of the present disclosure, which at least one embodiment herein satisfies are as listed herein below.
It is an object of the present disclosure to provide for a system that prevents user specific sensitive information flowing in a streaming media pipeline.
It is an object of the present disclosure to provide for a system that that restricts adversaries from fetching this sensitive information from the DL/ML models present in a streaming media pipeline.
This section is provided to introduce certain objects and aspects of the present disclosure in a simplified form that are further described below in the detailed description. This summary is not intended to identify the key features or the scope of the claimed subject matter.
In an aspect, the present disclosure provides for a system for preserving privacy for a set of data packets. The system may include a video analytics and privacy preserving module (VA-PP) module that may further include one or more processors operatively coupled to one or more first computing devices associated with one or more users. The one or more processors may execute a set of executable instructions that are stored in a memory upon execution of which, the one or more processor may cause the VA-PP module to receive, a first set of data packets from the one or more first computing device, the first set of data packets pertaining to a video stream, decode, from the first set of data packets, a sequence of frames, extract a set of information from the sequence of frames pertaining to sensitive information associated with a user. Based on the set of information extracted, the system may be configured to obtain, by an inference module, a section of interest from the sequence of frames, identify, from the section of interest one or more features of the user associated with the sensitive information and then replace, from the section of interest, the one or more features of the user with a new set of predefined pixels to obtain a new section of interest that is added to the sequence of frames.
In an embodiment, the section of interest may pertain to a plurality of pixels obtained by calculating gradient of the plurality of pixels using prediction probabilities.
In an embodiment, the new set of predefined pixels may provide false color conversion.
In an embodiment, the system may be configured to blur one or more pixels in the section of interest using a blur filter.
In an embodiment, the system may be further configured to provide color conservation and rescaling of the sequence of frames per a sink pad configuration.
In an embodiment, the system may be configured to post process and encode to convert the sequence of frames into an output video stream.
In an embodiment, the VA-PP module may include one or more processed buffers configured to fine-tune the sequence of frames and integrate privacy preserving techniques to preserve the one or more features associated with the user pertaining to the sensitive information.
In an embodiment, the system may be configured to preserve a set of information associated with any or a combination of the one or more features associated with the user pertaining to the sensitive information, the section of interest, the new set of predefined pixels.
In an embodiment, the system may be configured to provide flexibility to modify a set of preference parameters associated with the new set of predefined pixels as per requirements at any stage.
In an aspect, the present disclosure provides for a user equipment (UE) for preserving privacy for a set of data packets. The UE may include a video analytics and privacy preserving module (VA-PP) module that may include a processor operatively coupled to one or more first computing devices associated with one or more users. The processor may execute a set of executable instructions that are stored in a memory upon execution of which, the processor may cause the VA-PP module to receive, a first set of data packets from the one or more first computing device, the first set of data packets pertaining to a video stream, decode, from the first set of data packets, a sequence of frames, extract a set of information from the sequence of frames pertaining to sensitive information associated with a user. Based on the set of information extracted, the system may be configured to obtain, by an inference module, a section of interest from the sequence of frames, identify, from the section of interest one or more features of the user associated with the sensitive information and then replace, from the section of interest, the one or more features of the user with a new set of predefined pixels to obtain a new section of interest that is added to the sequence of frames.
In an aspect, the present disclosure provides for a method for preserving privacy for a set of data packets. The method may include the step of receiving, by a video analytics and privacy preserving module (VA-PP) module, a first set of data packets from the one or more first computing device (104), the first set of data packets pertaining to a video stream. The video analytics and privacy preserving module (VA-PP) module may include one or more processors operatively coupled to one or more first computing devices associated with one or more users. The one or more processors may execute a set of executable instructions that are stored in a memory. The method may further include the steps of decoding, by the VA-PP module, from the first set of data packets, a sequence of frames, extracting, by the VA-PP module, a set of information from the sequence of frames pertaining to sensitive information associated with a user. Based on the set of information extracted, the method may further include the step of obtaining, by an inference module associated with the VA-PP module, a section of interest from the sequence of frames, identifying, by the VA-PP module, from the section of interest one or more features of the user associated with the sensitive information; and then replacing, by the VA-PP module, from the section of interest, the one or more features of the user with a new set of predefined pixels to obtain a new section of interest that is added to the sequence of frames.
The accompanying drawings, which are incorporated herein, and constitute a part of this invention, illustrate exemplary embodiments of the disclosed methods and systems in which like reference numerals refer to the same parts throughout the different drawings. Components in the drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present invention. Some drawings may indicate the components using block diagrams and may not represent the internal circuitry of each component. It will be appreciated by those skilled in the art that invention of such drawings includes the invention of electrical components, electronic components or circuitry commonly used to implement such components.
The foregoing shall be more apparent from the following more detailed description of the invention.
In the following description, for the purposes of explanation, various specific details are set forth in order to provide a thorough understanding of embodiments of the present disclosure. It will be apparent, however, that embodiments of the present disclosure may be practiced without these specific details. Several features described hereafter can each be used independently of one another or with any combination of other features. An individual feature may not address all of the problems discussed above or might address only some of the problems discussed above. Some of the problems discussed above might not be fully addressed by any of the features described herein.
The ensuing description provides exemplary embodiments only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the ensuing description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing an exemplary embodiment. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the invention as set forth.
Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood by one of ordinary skill in the art that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.
The present invention provides a robust and effective solution to an entity or an organization by enabling the entity to implement a system for that links together a wide variety of media processing systems to complete complex workflows. The system can be configured to read files in one format, process the files, and export the files in another. The system can be a cross-platform and can be easily ported to various operating systems and can be used to integrate privacy preserving components into the system.
Referring to
The system (110) may extract a set of information from the sequence of frames pertaining to sensitive information. The extracted set of information may be preserved using an Up Gradient-False Color Conversion-with Noise and Adversarial Pixel (UG-FCC-NAP). The inferenced and preserved set of information may be provided to further applications and elements of the system that may include deep learning models followed by inference processes and then post processing and encoding to convert the sequence of frames into an output video stream.
In an exemplary embodiment, the system (110) may be configured to provide flexibility to modify the preference parameters as per requirements at any stage of the project.
In an embodiment, the user computing device (104) may communicate with the system (110) via set of executable instructions residing on any operating system. In an embodiment, user computing device (104) may include, but not limited to, any electrical, electronic, electro-mechanical or an equipment or a combination of one or more of the above devices such as mobile phone, smartphone, virtual reality (VR) devices, augmented reality (AR) devices, laptop, a general-purpose computer, desktop, personal digital assistant, tablet computer, mainframe computer, or any other computing device, wherein the computing device may include one or more in-built or externally coupled accessories including, but not limited to, a visual aid device such as camera, audio aid, a microphone, a keyboard, input devices for receiving input from a user such as touch pad, touch enabled screen, electronic pen and the like. It may be appreciated that the user computing device (104) may not be restricted to the mentioned devices and various other devices may be used. A smart computing device may be one of the appropriate systems for storing data and other private/sensitive information.
In an exemplary embodiment, a network (106) may include, by way of example but not limitation, at least a portion of one or more networks having one or more nodes that transmit, receive, forward, generate, buffer, store, route, switch, process, or a combination thereof, etc. one or more messages, packets, signals, waves, voltage or current levels, some combination thereof, or so forth. A network may include, by way of example but not limitation, one or more of: a wireless network, a wired network, an internet, an intranet, a public network, a private network, a packet-switched network, a circuit-switched network, an ad hoc network, an infrastructure network, a public-switched telephone network (PSTN), a cable network, a cellular network, a satellite network, a fiber optic network, some combination thereof.
In another exemplary embodiment, the centralized server (112) may include or comprise, by way of example but not limitation, one or more of: a stand-alone server, a server blade, a server rack, a bank of servers, a server farm, hardware supporting a part of a cloud service or system, a home server, hardware running a virtualized server, one or more processors executing code to function as a server, one or more machines performing server-side functionality as described herein, at least a portion of any of the above, some combination thereof.
In an embodiment, the system (110) may include one or more processors coupled with a memory, wherein the memory may store instructions which when executed by the one or more processors may cause the system to overcome privacy threat while video streaming.
In an embodiment, the system (110)/VA PP module (114) may include an interface(s) 206. The interface(s) 206 may comprise a variety of interfaces, for example, interfaces for data input and output devices, referred to as I/O devices, storage devices, and the like. The interface(s) 206 may facilitate communication of the system (110). The interface(s) 206 may also provide a communication pathway for one or more components of the system (110). Examples of such components include, but are not limited to, processing engine(s) 208 and a database 210.
The processing engine(s) (208) may be implemented as a combination of hardware and programming (for example, programmable instructions) to implement one or more functionalities of the processing engine(s) (208). In examples described herein, such combinations of hardware and programming may be implemented in several different ways. For example, the programming for the processing engine(s) (208) may be processor executable instructions stored on a non-transitory machine-readable storage medium and the hardware for the processing engine(s) (208) may comprise a processing resource (for example, one or more processors), to execute such instructions. In the present examples, the machine-readable storage medium may store instructions that, when executed by the processing resource, implement the processing engine(s) (208). In such examples, the system (110) may comprise the machine-readable storage medium storing the instructions and the processing resource to execute the instructions, or the machine-readable storage medium may be separate but accessible to the system (110) and the processing resource. In other examples, the processing engine(s) (208) may be implemented by electronic circuitry.
The processing engine (208) may include one or more engines selected from any of a data acquisition engine (212), an artificial intelligence (AI) engine (214), and other engines (216). The other engines (216) may include an inference module, a scaling module, one or more deep learning modules and the like.
In an embodiment, the UE (108) may include an interface(s) 226. The interface(s) 226 may comprise a variety of interfaces, for example, interfaces for data input and output devices, referred to as I/O devices, storage devices, and the like. The interface(s) 226 may facilitate communication of the UE (108). Examples of such components include, but are not limited to, processing engine(s) 228 and a database (230).
The processing engine(s) (228) may be implemented as a combination of hardware and programming (for example, programmable instructions) to implement one or more functionalities of the processing engine(s) (228). In examples described herein, such combinations of hardware and programming may be implemented in several different ways. For example, the programming for the processing engine(s) (228) may be processor executable instructions stored on a non-transitory machine-readable storage medium and the hardware for the processing engine(s) (228) may comprise a processing resource (for example, one or more processors), to execute such instructions. In the present examples, the machine-readable storage medium may store instructions that, when executed by the processing resource, implement the processing engine(s) (228). In such examples, the UE (108) may comprise the machine-readable storage medium storing the instructions and the processing resource to execute the instructions, or the machine-readable storage medium may be separate but accessible to the UE (108) and the processing resource. In other examples, the processing engine(s) (228) may be implemented by electronic circuitry.
The processing engine (228) may include one or more engines selected from any of a data acquisition engine (232), an artificial intelligence (AI) engine (234), and other engines (236). The other engines (236) may include an inference module, a scaling module, one or more deep learning modules and the like.
As illustrated in
The frames are then sent to inference and Video Analytics with Privacy Preserving (VA-PP) block (308) for preserving the information obtained from the inferencing models and preserving the sensitive information using Up Gradient-False Color Conversion-with Noise and Adversarial Pixel.
The inferenced information with the sensitive data preserved then given to further applications and elements of the streaming media pipeline which includes deep learning models (310) followed by inference processes (312) and then post processing and encoding (314) to convert the sequence of frames into a video stream.
In an exemplary implementation, Grad represents gradient generated based on prediction probabilities, IoI represents Image of Interest, Delta max represents pixels with high importance in the view of prediction by the model. Px represents perturbed IoI of x class. The Px is obtained by subtracting the highlighted pixels from IoI and adding gradient pixels of other classes that changes the prediction class of the IoI. This Px is the added with noise generated which will result miss classifications and non-identification of the IoI from naked eyes. X here is the present class in IoI and y is the class different from that of x. The equations are given below
P
x=GradIoI−∇max GradIoI+∇max GradIoI eq (1)
where y≠x, x and y are classes and Px is perturbed image
GradIoI=∇(x), where x∈IoI eq(2)
VA PPG
x
=P
x
+N, N∈Gaussian Noise eq(3)
Bus 720 communicatively couples processor(s) 770 with the other memory, storage and communication blocks. Optionally, operator and administrative interfaces, e.g. a display, keyboard, and a cursor control device, may also be coupled to bus 720 to support direct operator interaction with a computer system. Other operator and administrative interfaces can be provided through network connections connected through communication port 760. Components described above are meant only to exemplify various possibilities. In no way should the aforementioned exemplary computer system limit the scope of the present disclosure.
While considerable emphasis has been placed herein on the preferred embodiments, it will be appreciated that many embodiments can be made and that many changes can be made in the preferred embodiments without departing from the principles of the invention. These and other changes in the preferred embodiments of the invention will be apparent to those skilled in the art from the disclosure herein, whereby it is to be distinctly understood that the foregoing descriptive matter to be implemented merely as illustrative of the invention and not as limitation.
A portion of the disclosure of this patent document contains material which is subject to intellectual property rights such as, but are not limited to, copyright, design, trademark, IC layout design, and/or trade dress protection, belonging to Jio Platforms Limited (JPL) or its affiliates (herein after referred as owner). The owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all rights whatsoever. All rights to such intellectual property are fully reserved by the owner.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202121060643 | Dec 2021 | IN | national |
