Patent Application
20200193868
The present invention relates generally to the analysis of multimedia content items, and more specifically to techniques for identifying a correct orientation of a multimedia content item.
Computing devices, such as mobile devices, tablets, smartphones, and the likes, frequently include an orientation sensor that indicates the orientation of the computing devices with respect to a reference point, such as gravitational pull or other orientation references. Current applications executed on these computing devices use the orientation information of the computing devices to adjust functions of each computing device. For example, such applications are configured to rotate a multimedia content item displayed on a user interface of a mobile device based on the orientation of the mobile device.
The problem with such applications is that the multimedia content item is not analyzed before it is displayed on the user interface. Thus, in a case where the orientation of the multimedia content item (e.g., an image), or portion of it (i.e., object shown in the image) is incorrect in the first place; the image will be displayed in an incorrect orientation on the user interface despite the orientation sensor the mobile device is equipped with. For example, in a case where an image is captured by a mobile device with an inarticulate camera angle, the view of the image (as captured) is not vertical or horizontal to the ground, hence using the existing orientation sensor to rotate the image will not solve the problem.
It would be therefore advantageous to provide an efficient solution to analyze multimedia content items. It would be further advantageous if such a solution would enable identification of a correct orientation of an object shown in the multimedia content item.
A summary of several example embodiments of the disclosure follows. This summary is provided for the convenience of the reader to provide a basic understanding of such embodiments and does not wholly define the breadth of the disclosure. This summary is not an extensive overview of all contemplated embodiments, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all embodiments. Its sole purpose is to present some concepts of one or more embodiments in a simplified form as a prelude to the more detailed description that is presented later. For convenience, the term some embodiments may be used herein to refer to a single embodiment or multiple embodiments of the disclosure.
Certain embodiments include a method for identifying a correct orientation of a multimedia content item. The method comprises receiving from a user device the multimedia content item; identifying at least one object shown in the multimedia content item; generating by a signature generator system (SGS) at least one signature for the at least one object shown in the multimedia content item; querying, using the at least one generated signature, a deep-content-classification (DCC) system to find at least one concept that matches the at least one object, wherein the querying of the DCC system is performed using the at least one signature generated for each object shown in the multimedia content item; determining a correct orientation of the at least one matching concept; and comparing an orientation of the at least one object to the determined correct orientation to determine if the at least one object is correctly oriented.
Certain embodiments include a system for identifying a correct orientation of a multimedia content item. The system comprises an interface to a network for receiving the multimedia content item; a processing unit; and a memory communicatively connected to the processing unit, wherein the memory contains instructions that, when executed by the processing unit, configures the system to: receive from a user device the multimedia content item; identify at least one object shown in the multimedia content item; generate by a signature generator system (SGS) at least one signature for the at least one object shown in the multimedia content item; query, using the at least one generated signature, a deep-content-classification (DCC) system to find at least one concept that matches the at least one object, wherein the querying of the DCC system is performed using the at least one signature generated for each object shown in the multimedia content item; determine a correct orientation of the at least one matching concept; and comparing an orientation of the at least one object to the determined correct orientation to determine if the at least one object is correctly oriented.
The subject matter disclosed herein is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other objects, features, and advantages of the disclosed embodiments will be apparent from the following detailed description taken in conjunction with the accompanying drawings.
It is important to note that the embodiments disclosed herein are only examples of the many advantageous uses of the innovative teachings herein. In general, statements made in the specification of the present application do not necessarily limit any of the various claimed embodiments. Moreover, some statements may apply to some inventive features but not to others. In general, unless otherwise indicated, singular elements may be in plural and vice versa with no loss of generality. In the drawings, like numerals refer to like parts through several views.
Certain exemplary embodiments disclosed herein include a method for analyzing the orientation of objects shown in a multimedia content item for detecting an incorrect orientation of the multimedia content item. In an embodiment, the multimedia content item is received from a user device. At least one signature is generated for at least one object shown in the multimedia content item. The signatures generated for the at least one object are matched to signatures generated for at least one concept. The signatures generated for each concept are retrieved from a data warehouse. Upon identifying a match between at least one object and at least one concept, the correct orientation of the concept is retrieved from the data warehouse. The orientation of the concept is correlated to the orientation of the object shown in the multimedia content item to determine whether the orientation of the object is the correct orientation.
Upon identification of an incorrect orientation of the object, the multimedia content item is rotated until the object is in the correct orientation. According to an embodiment, the correct multimedia content item is then sent to the user device for display.
Further connected to the network 110 is a user device 120 configured to execute at least one application 125. The application 125 may be, for example, a web browser, a script, an add-on, a mobile application (“app”), or any application programmed to interact with a server 130. The user device 120 may be, but not limited to, a personal computer (PC), a personal digital assistant (PDA), a mobile phone, a smart phone, a tablet computer, a laptop, a wearable computing device, or another kind of computing device equipped with browsing, viewing, listening, filtering, and managing capabilities that is enabled as further discussed herein below. It should be noted that one user device 120 and one application 125 are illustrated in
The network system 100 also includes a data warehouse 160 configured to store multimedia content items, previously generated signatures for concepts or concept structures, information respective of the concepts' orientation in space, and the like. The data warehouse 160 may be further connected to the network 110. In the embodiment illustrated in
The various embodiments disclosed herein are realized using the server 130, a signature generator system (SGS) 140 and a deep-content-classification (DCC) system 150. The SGS 140 may be connected to the server 130 directly or through the network 110. The server 130 is configured to receive and serve the at least one multimedia content item in which objects are shown and cause the SGS 140 to generate at least one signature respective thereof and query the DCC system 150. To this end, the server 130 is communicatively connected to the SGS 140 and the DCC system 150. The DCC system 150 may be further connected to the network 110.
The DCC system 150 is configured to generate concept structures (or concepts) and to identify concepts that match the objects. A concept is a collection of signatures representing an object and metadata describing the concept. The collection is a signature reduced cluster generated by inter-matching the signatures generated for the many objects, clustering the inter-matched signatures, and providing a reduced cluster set of such clusters. As a non-limiting example, a ‘Superman concept’ is a signature reduced cluster of signatures describing elements (such as objects) related to, e.g., a Superman cartoon: a set of metadata including textual representations of the Superman concept.
Techniques for generating concepts and concept structures are also described in the U.S. Pat. No. 8,266,185 (hereinafter the '185 Patent) to Raichelgauz, et al., which is assigned to a common assignee, and is incorporated by reference herein for all that it contains. In an embodiment, the DCC system 150 is configured and operates as the DCC system discussed in the '185 patent. The process of generating the signatures in the SGS 140 is explained in more detail below with respect to
It should be noted that each of the server 130, the SGS 140, and the DCC system 150 typically comprise a processing unit, such as a processor (not shown) or an array of processors coupled to a memory. In one embodiment, the processing unit may be realized through architecture of computational cores described in detail below. The memory contains instructions that can be executed by the processing unit. The instructions, when executed by the processing unit, cause the processing unit to perform the various functions described herein. The one or more processors may be implemented with any combination of general-purpose microprocessors, multi-core processors, microcontrollers, digital signal processors (DSPs), field programmable gate array (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic, discrete hardware components, dedicated hardware finite state machines, or any other suitable entities that can perform calculations or other manipulations of information. The server 130 also includes an interface (not shown) to the network 110.
According to the disclosed embodiments, the server 130 is configured to receive a multimedia content item showing objects from the user device 120. The multimedia content item may be, but is not limited to, an image, a graphic, a photograph, and/or combinations thereof and portions thereof. An object may be any element shown in the multimedia content item, for example, a tree, a car, a person, a table, and the like. In one embodiment, the server 130 is configured to receive a URL of a webpage viewed by the user device 120 and accessed by the application 125. The webpage is processed to extract the multimedia content item contained therein.
The request to analyze the multimedia content item can be sent by a script executed in the webpage such as the application 125 (e.g., a web server or a publisher server) when requested to upload one or more multimedia content items to the webpage. Such a request may include a URL of the webpage or a copy of the webpage. The application 125 can also send a picture taken by a user of the user device 120 to the server 130.
Responsive to receiving the multimedia content item, the server 130 is configured to rotate the multimedia content item until the multimedia content item is in the correct orientation and to return the correctly oriented multimedia content item. To this end, the server 130 is configured to analyze the multimedia content item to identify portions or objects in the multimedia content item. As an example, an image showing Central Park in New York is analyzed to identify the objects of a carriage way, a car, a streetlight, and a person. At least one signature is generated for each object using the SGS 140. The generated signatures may be robust to noise and distortion as discussed below.
In one embodiment, using the generated signatures, the DCC system 150 is queried to determine if there is a match to at least one concept maintained in the data warehouse 160. The DCC system 150 returns for each matching concept a concept's signature (signature reduced cluster (SRC)) and optionally the concept's metadata. Using the SRC of the matching concept and the signatures generated for the at least one object, the server 130 is configured to determine if there a difference between the orientation of the object in the multimedia content item and the matching concept. According to an embodiment, parameters such as orientation of an object and/or a concept in space respective of a reference point may be taken into account.
Specifically, when one match is identified, the server 130 is configured to retrieve from the data warehouse 160 information respective of the typical orientation of a concept in space. The information contained in the data warehouse 160 may have been entered by users, collected from external web sources connected to the network, saved from previous calculations of the disclosed method, and the like. In another embodiment, the information respective of the typical orientation of a concept in space may be determined respective of the relation between at least two elements shown constantly in multimedia elements stored in the database. As a non-limiting example, a tree is always perpendicular to grass. The correct orientation of such a concept is determined respective thereof. As an example, when a match to the concept “tree” is identified, the server 130 is configured to determine that the tree should be perpendicular to the ground (the ground in such case can be used as a reference point).
The server 130 is further configured to correlate the orientation of an object shown in the multimedia content item and the correct concept's orientation. This is performed by correlating the signatures generated for the object and the signatures of the concept retrieved from the data warehouse 160. Here it should be noted that the signatures generated for each object are generated respective of the spatial location of an object shown in the multimedia content item. Upon identification of an incorrect orientation of the object, the multimedia content item is rotated until the object is in the correct orientation. According to an embodiment, the correct multimedia content item is then sent to the user device 120 for display.
In another embodiment, the SGS 140 is configured to generate signatures for the objects shown in the received multimedia content item. The generated signatures are matched by the server 130 to previously generated signatures of concepts maintained in the data warehouse 160 to identify at least one object that matches to at least one concept. When such a match is identified, the server 130 is configured to correlate the orientation of the concept and the orientation of the object shown in the multimedia content item as noted above. Upon identification of an incorrect orientation of the object, the multimedia content item is rotated until the object is in the correct orientation. According to an embodiment, the correct multimedia content item is then sent to the user device for display.
In S210, a multimedia content item in which objects are shown is received. In an embodiment, the multimedia content item is received together with a request to analyze the orientation of the multimedia content item. Optionally, in S215, the received multimedia content item is analyzed to identify at least one object shown within.
In S220 at least one signature is generated for at least one object. The signatures are generated respective of the spatial location of the object shown in the multimedia content item. The signatures are generated by the SGS 140 as described in greater detail below with respect to
In S230, a DCC system (e.g., DCC system 150) is queried to find a match between at least one concept and the object using their respective signatures. In an embodiment, at least one signature generated for an object is matched against the signature (signature reduced cluster (SRC)) of each concept maintained by the DCC system 150. If the signature of the concept overlaps with the signature of the multimedia element more than a predetermined threshold level, a match exists. Various techniques for determining matching concepts are discussed in the '185 Patent. For each matching concept the respective multimedia element is determined to be identified and at least the concept signature (SRC) is returned.
In S240, the correct/typical orientation of the matching concept is determined respective of information related to the concept maintained in a database, such as the data warehouse 160. The information contained in the data warehouse 160 may have been entered by users, collected from external web sources connected to the network, saved from previous calculations of the disclosed method, and the like. In another embodiment, the correct/typical orientation of the matching concept is determined respective of the relation between at least two elements shown constantly in multimedia elements stored in the database, for example, a tree is always perpendicular to grass, etc.
In S250 the orientation of the object is correlated to the orientation of the concept. The correlation includes analyzing the signatures generated for the object and the signatures of the concept. Such correlation is performed based on the spatial location of the object and respective of information related to the concept maintained in the data warehouse 160. In another embodiment, if matching concepts are not found, the signatures generated in S220 are utilized to search the data warehouse 160.
In S260, it is checked whether the orientation of the object shown in the multimedia content item is the correct orientation, and if so, execution continues with S280; otherwise, execution continues with 270. In S270, the multimedia content item is rotated until the object is in the correct orientation. According to an embodiment, the corrected multimedia content item is sent to the user device 120 for display. In another embodiment, object is rotated until the object is in the correct orientation. In S280, it is checked whether additional multimedia content items are received, and if so, execution continues with S215; otherwise, execution terminates.
The objects of a house 310, the moon 320 and a tree 330 are identified in the drawing 300 and signatures are generated for each such object. The generated signatures are then used to search for matching concepts. The “house” concept is identified. Information related to the “house” concept is retrieved and the typical orientation of a house in space is determined (i.e., the house should be perpendicular to the ground). Specifically, the server 130 is configured to query a DCC system 150 to search a data warehouse 160 for information related to the “house” concept. This information includes a typical orientation of a house relative to the ground. In another embodiment, the typical orientation of the house is determined respective of the relation between at least two elements shown constantly in multimedia elements stored in the database, for example, a tree is always perpendicular to grass, etc. The orientation of the concept “house” is correlated to orientation of the house 310 shown in the drawing and it is determined that the orientation of the house 310 is incorrect, and therefore the orientation of the drawing is incorrect. According to this embodiment, the drawing 300 is rotated 340 until the house 310 is in the correct orientation.
Video content segments 2 from a Master database (DB) 6 and a Target DB 1 are processed in parallel by a large number of independent computational cores 3 that constitute an architecture for generating the signatures (hereinafter the “Architecture”). Further details on the generation of computational cores are provided below. The independent cores 3 generate a database of Robust Signatures and Signatures 4 for Target content-segments 5 and a database of Robust Signatures and Signatures 7 for Master content-segments 8. An exemplary and non-limiting process of signature generation for an audio component is shown in detail in
To demonstrate an example of the signature generation process, it is assumed, merely for the sake of simplicity and without limitation on the generality of the disclosed embodiments, that the signatures are based on a single frame, leading to certain simplification of the computational cores generation. The Matching System is extensible for signatures generation capturing dynamics in-between the frames.
The Signatures' generation process is now described with reference to
In order to generate Robust Signatures, i.e., Signatures that are robust to additive noise L (where L is an integer equal to or greater than 1) by the computational cores 3 a frame ‘i’ is injected into all the cores 3. Then, cores 3 generate two binary response vectors: {right arrow over (S)}, which is a Signature vector, and {right arrow over (RS)} which is a Robust Signature vector.
For generation of signatures robust to additive noise, such as White-Gaussian-Noise, scratch, etc., but not robust to distortions, such as crop, shift and rotation, etc., a core Ci={ni} (1≤i≤L) may consist of a single leaky integrate-to-threshold unit (LTU) node or more nodes. The node ni equations are:
where, is a Heaviside step function; wij is a coupling node unit (CNU) between node i and image component j (for example, grayscale value of a certain pixel j); kj is an image component ‘j’ (for example, grayscale value of a certain pixel j); THx is a constant Threshold value, where ‘x’ is ‘S’ for Signature and ‘RS’ for Robust Signature; and Vi is a Coupling Node Value.
The Threshold values ThX are set differently for Signature generation than for Robust Signature generation. For example, for a certain distribution of Vi values (for the set of nodes), the thresholds for Signature (ThS) and Robust Signature (ThRS) are set apart, after optimization, according to at least one or more of the following criteria:
1−p(V>ThS)−1−(1−ε)l<<1
i.e., given that I nodes (cores) constitute a Robust Signature of a certain image I, the probability that not all of these I nodes will belong to the Signature of same, but noisy image, Ĩ is sufficiently low (according to a system's specified accuracy).
i.e., approximately I out of the total L nodes can be found to generate a Robust Signature according to the above definition.
It should be understood that the generation of a signature is unidirectional, and typically yields lossless compression, where the characteristics of the compressed data are maintained but the uncompressed data cannot be reconstructed. Therefore, a signature can be used for the purpose of comparison to another signature without the need for comparison to the original data. The detailed description of the signature generation can be found in U.S. Pat. Nos. 8,326,775 and 8,312,031, assigned to common assignee, which are hereby incorporated by reference for all the useful information they contain.
A computational core generation is a process of definition, selection, and tuning of the parameters of the cores for a certain realization in a specific system and application. The process is based on several design considerations, such as:
(a) The cores should be designed so as to obtain maximal independence, i.e., the projection from a signal space should generate a maximal pair-wise distance between any two cores' projections into a high-dimensional space.
(b) The cores should be optimally designed for the type of signals, i.e., the cores should be maximally sensitive to the spatio-temporal structure of the injected signal, for example, and in particular, sensitive to local correlations in time and space. Thus, in some cases, a core represents a dynamic system, such as in state space, phase space, edge of chaos, etc., which is uniquely used herein to exploit its maximal computational power.
(c) The cores should be optimally designed with regard to invariance to a set of signal distortions, of interest in relevant applications.
A detailed description of the computational core generation and the process for configuring such cores is discussed in more detail in U.S. Pat. No. 8,655,801 referenced above.
The various embodiments disclosed herein can be implemented as hardware, firmware, software, or any combination thereof. Moreover, the software is preferably implemented as an application program tangibly embodied on a program storage unit or computer readable medium consisting of parts, or of certain devices and/or a combination of devices. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units (“CPUs”), a memory, and input/output interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU, whether or not such a computer or processor is explicitly shown. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit. Furthermore, a non-transitory computer readable medium is any computer readable medium except for a transitory propagating signal.
All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the principles of the disclosed embodiments and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, embodiments, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.
This application claims the benefit of U.S. provisional application No. 62/030,086 filed on Jul. 29, 2014. This application is also a continuation-in-part (CIP) of U.S. patent application Ser. No. 14/096,865 filed Dec. 4, 2013, now pending, which claims the benefit of U.S. provisional application No. 61/890,251 filed Oct. 13, 2013. The Ser. No. 14/096,865 Application is a continuation-in-part (CIP) of U.S. patent application Ser. No. 13/624,397 filed on Sep. 21, 2012, now pending. The Ser. No. 13/624,397 application is a CIP of: (a) U.S. patent application Ser. No. 13/344,400 filed on Jan. 5, 2012, U.S. Pat. No. 8,959,037, which is a continuation of U.S. patent application Ser. No. 12/434,221, filed May 1, 2009, now U.S. Pat. No. 8,112,376; (b) U.S. patent application Ser. No. 12/195,863, filed Aug. 21, 2008, now U.S. Pat. No. 8,326,775, which claims priority under 35 USC 119 from Israeli Application No. 185414, filed on Aug. 21, 2007, and which is also a continuation-in-part of the below-referenced U.S. patent application Ser. No. 12/084,150; and (c) U.S. patent application Ser. No. 12/084,150 having a filing date of Apr. 7, 2009, now U.S. Pat. No. 8,655,801, which is the National Stage of International Application No. PCT/IL2006/001235, filed on Oct. 26, 2006, which claims foreign priority from Israeli Application No. 171577 filed on Oct. 26, 2005 and Israeli Application No. 173409 filed on 29 Jan. 2006. All of the applications referenced above are herein incorporated by reference for all that they contain.
| Number | Date | Country | |
|---|---|---|---|
| 62030086 | Jul 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14638176 | Mar 2015 | US |
| Child | 16711686 | US |
