Patent Application
20130086075
The term “hashing” or (signatures) refers to a process wherein one computes a signature of a property, such that when the property stays the same, the signatures are equal and when the property changes, the signatures are “likely” to change (subject to a probabilistic principle). Hashing functions are related to (and often confused with) checksums, check digits, fingerprints, randomization functions, error correcting codes, and cryptographic hash functions. Although standard hashing functions for these various purposes do exist, they generally do not measure a particular property of a matrix or dimensional array, as opposed to merely getting a hash number of an input matrix or object.
Additionally, current hashing techniques operating on matrices to measure a particular property of the matrix (e.g., sparseness) often involve a processor and memory-intensive matrix algebra. For example, to measure sparseness with standard hashing techniques, a sparseness structure is created and a hash value of this sparseness structure is then obtained. But the creation of the sparseness structure often requires more processing overhead than the hashing operation itself.
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various embodiments. In the drawings:
Reference will now be made in detail to the present exemplary embodiments illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.
One or more user devices 161, 162, and 163, may be coupled to the signature system 105 via the network 140 or some other coupling. User devices 161, 162, or 163 may be, for example, personal computers, personal data devices, telephones, tablet personal computers, mobile phones, or other devices coupled to identification system 105 via network 140. Additionally, in some embodiments (not shown) users may directly operate signature system 105 (e.g., the user's device may embody the signature system 105). While portions of the specification may only refer to one user device 161, 162, or 163, this is for simplification purposes only and, unless noted otherwise, is not meant to limit the described embodiments in any way.
Data sources 151, 152, and 153 may be any form of database, including a proprietary database containing information about one or more users. Data sources 151, 152, and 153 may be “blogs” or websites, such as social networking websites or news agency websites. Data sources 151, 152, and 153 may, for example, be private party websites, company websites, or cached information stored in a search database, such as those maintained at Google™ or Yahoo™. Data sources 151, 152, and 153 may, in alternative examples, be a criminal database or listing, a credit agency data source, an insurance database, facial recognition database, a fingerprint database, a DNA database, a retinal database, or any electronic or other source of information about any user. There may be any number of data sources 151, 152, and 153. While portions of the specification may only refer to only one data source 151, 152, and/or 153, this has been done for simplification purposes only and, unless noted otherwise, is not meant to limit the described embodiments in any way. Documents contained in the search database may relate to a person, organization, or other entity and may include web pages, emails, Microsoft Word™ documents, plain text files, encoded documents, or any other appropriate form of unstructured electronic information.
Network 140 may be, for example, the Internet, an intranet, a local area network, a wide area network, a campus area network, a metropolitan area network, an extranet, a private extranet, any set of two or more coupled electronic devices, or a combination of any of these or other appropriate networks.
Signature system 105 may include one or more processors (not shown), a memory (not shown), and a data interface (not shown). The processor(s) may be a single or multiple microprocessors, field programmable gate arrays (FPGAs), or digital signal processors (DSPs) capable of executing particular sets of instructions. Computer-readable instructions may be stored on a tangible non-transitory computer-readable medium, such as a flexible disk, a hard disk, a CD-ROM (compact disk-read only memory), and MO (magneto-optical), a DVD-ROM (digital versatile disk-read only memory), a DVD RAM (digital versatile disk-random access memory), or a semiconductor memory. Alternatively, the methods may be implemented in hardware components or combinations of hardware and software such as, for example, ASICs, special purpose computers, or general purpose computers. Signature system 105 may be implemented on a single computer, or may be distributed across a plurality of computers. Signature system 105 may be coupled to multiple data sources, for example, data sources 151, 152, and 153 either via network 140 or via other coupling.
Input module 200 is a hardware interface, a software program, or a combination thereof configured to obtain a set of data from a network, for example, data from one or more user devices 161-163 or one or more data sources 151-153. The data set may include, for example, text based information, image based information, or some combination thereof. Additionally, input module 200 may be configured to receive a data set from a userinterfacing directly with signature system 105. Input module 200 can be configured to store the received information in data storage module 260. Input module 200 may be, for example, an Internet web server, such as Apache Tomcat™, Microsoft's Internet Information Services™, or Sun's Java System Web Server™. In some embodiments, input module 200 is combined with output module 270. Input module 200 may be coupled to one or more of collector module 210, data processing module 230, identification module 240, and data storage module 260.
Collector module 210 is a hardware component, a software program, or a combination thereof configured to formulate searches using the received information. Additionally, the collector module 210 may be configured to prepare queries to send to one or more data sources 151, 152, and 153. Collector module 210 may be configured to work with output module 250 for sending the queries to one or more data sources 151, 152, and 153. Collector module 210 may be configured to receive electronic information related to the target of the search from data sources 151, 152, and 153. Collector module 210 may be coupled to one or more of input module 200, characterization module 220, and data storage module 260.
Characterization module 220 is a hardware component, a software program, or a combination thereof configured to characterize the received data set creating one or more data matrices. For example, if the data set is text-based, characterization module 220 can break the stream of text up into tokens (e.g., each token representing a word, a phrase, or a symbol), which are used to populate a data matrix, such that each received document has its own associated data matrix. Similarly if the data set is image-based, characterization module 220 can break the images up into tokens (e.g., each token representing a numeric value associated with a characteristic of the image), which are used to populate a data matrix, such that each received image has its own associated data matrix. For example, fingerprints may be associated with a data matrix by looking at features associated with each finger. Fingerprints may contain specific patterns, such as whorls, arches, and loops. The presence of these features can be assigned a numeric value. The one or more data matrices may have the same dimensions, n×m.
Characterization module 220 can provide the one or more data matrices to data processing module 230. In some embodiments, characterization module 220 may provide the one or more data matrices to data storage module 260. Characterization module 220 may be coupled to one or more of collector module 210, data processing module 230, and data storage module 260.
Data processing module 230 is a hardware component, a software program, or a combination thereof configured to process the one or more data matrices, to determine a unique signature for each of the data matrices. Data processing module 230 may be configured to receive one or more data matrices from characterization module 220 or input module 200.
Data processing module 230 may be configured to determine the number of signatures to be calculated by signature system 105. In some embodiments, the number of signatures may be equal to or in excess of the total number of data matrices. In some embodiments, a user may input the number of desired signatures (e.g., via input module 200). Data processing module is configured to automatically determine a prime number, N, which is larger than the desired total number of signatures. Data processing module 230 is configured to automatically set N to a value greater than the number 3.
Data processing module 230 may be configured to determine a plurality of primitive roots of N. The process of determining primitive roots is discussed below with reference to
Data processing module 230 may be configured to calculate a template matrix that has the same dimensions as the one or more data matrices, n×m. The elements of the template matrix are determined by the formula: aibj, where a and b correspond to the primitive roots previously determined. The ‘i’ variable corresponds to the row position within the template matrix, and the ‘j’ variable corresponds to the column position within the template matrix. For example, for a 2×2 template matrix, where a=2 and b=3, the template matrix would equal [(6, 12), (18, 36)].
Data processing module 230 may be configured to characterize the one or more data matrices based on one or more properties of the received information. Properties may include: sparseness structures, positive numbers, negative numbers, repeated rows, repeated columns, irregular matrices, etc. For example, if the user is interested in creating signatures for sparseness structures, data processing module 230 may replace every non-zero entry in each data matrix with a ‘1,’ such that the set of one or more data matrices only contains ‘1’ and ‘0’ values. These characterized matrices may then be applied to the template matrix to create a shadow matrix for each of the one or more data matrices. For example if the template matrix has a value of [(15, 45), (75, 225)], and a characterized data matrix has a value of [(1, 0), (0, 1)], the associated shadow matrix has a value of [(15, 0), (0, 225)]. In some embodiments, the shadow matrices are temporary and are not stored in persistent memory.
Data processing module 230 may be configured to calculate a signature for each data matrix using the hash function:
The hash function is calculated for each of the data matrices, where Aij corresponds to the positional value within the data matrix being processed. The modifier, if Aij=condition, modifies the hash function such that it creates signatures respective of a particular property of interest. Additionally, this function may be tailored for the particular property of interest. For example, if the property of interest is sparseness structures, the hash function becomes:
where Ai,j refers to elements within each data matrix that are not equal to zero. Data processing module 230 may be configured to store the calculated signatures in data storage module 260. The process of calculating the hash function is discussed in detail below with reference to
Data processing module 230 may be configured to detect collisions in the calculated signatures. A collision occurs when, of the set of signatures, two or more signatures have the same value. Data processing module 230 may be configured to select new values from a and b, recalculate the template matrix with the new a and b values, re-calculate the signatures using the hash function, and then determine if any additional collisions occur. Additionally, data processing module 230 may be configured to choose a larger prime number (than the original N value) as N if collisions continue to occur. In some embodiments, data processing module 230 may be configured to select the new a and b values from the plurality of primitive roots previously selected.
In some embodiments, data processing module 230 may be configured to re-calculate the signatures for all the data matrices, using the new a and b values. In some embodiments, data processing module 230 may be configured to only re-calculate the signatures with the new a and b values for only the data matrices where the collision originally occurred.
Additionally, in some embodiments not shown, data processing module 230 may be coupled to one or more slave servers, where each slave server maintains a portion of the one or more data matrices. Data processing module may be coupled to one or more of input module 200, characterization module 220, identification module 240, and data processing module 240.
Identification module 240 is a hardware component, a software program, or a combination thereof configured to compare one or more signatures to the signatures associated with the main set of signatures to determine if a match occurs. Identification module 240 may be configured to receive one or more signatures from data processing module 230, input module 200, output module 250, and data storage module 260.
Output module 250 may also be configured to transmit data, via network 140, from identification module 240, data storage module 260, or some combination thereof, to one or more user devices 161, 162, and 163. In some embodiments, output module 250 acquires the one or more signatures associated with the one or more data matrices from data storage module 260 and provides them to one or more user devices 161, 162, and 163. Additionally, in some embodiments, output module 250 acquires any match information from identification module 240. This information may be displayed to the user through the user's access device. In some embodiments output module 260 may display the results to a human operator who may modify, correct, or alter the information, or to any other system or agent capable of interacting with the information, including an artificial intelligence system or agent (AI agent), before sending it to the user.
Additionally, output module 250 may be configured to provide the main set of signatures, the one or more data matrices, or a combination thereof, to one or more user devices 161, 162, and 163. Output module 250 may be coupled to one or more of identification module 240 and data storage module 260.
Data storage module 260 may comprise a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), a field programmable read-only memory (FPROM), or other dynamic storage device for storing information and instructions to be used by input module 200, collector module 210, characterization module 220, data processing module 230, identification module 240, and output module 250. For example, data storage module 260 may store data received by input module 200. Data storage module 260 may also include a database, one or more computer files in a directory structure, or any other appropriate data storage mechanism such as a memory. In some embodiments, data storage module 260 is distributed across a plurality of different data storage mechanisms.
The coupling between modules, or between modules and network 140, may include, but is not limited to, electronic connections, coaxial cables, copper wire, and fiber optics, including the wires that comprise network 140. The coupling may also take the form of acoustic or light waves, such as lasers and those generated during radio-wave and infra-red data communications. Coupling may also be accomplished by communicating control information or data through one or more networks to other data devices. In some embodiments input module 200, collector module 210, characterization module 220, data processing module 230, identification module 240, output module 250, and data storage module 260 may be coupled in a manner such that each module is logically connected to all of the other modules in information system 105.
Each of the logical or functional modules described above may comprise multiple modules. The modules may be implemented individually or their functions may be combined with the functions of other modules. Further, each of the modules may be implemented on individual components, or the modules may be implemented as a combination of components. For example, input module 200, collector module 210, characterization module 220, data processing module 230, identification module 240, and output module 250 may each be implemented by a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a printed circuit board (PCB), a combination of programmable logic components and programmable interconnects, single CPU chip, a CPU chip combined on a motherboard, a general purpose computer, or any other combination of devices or modules capable of performing the tasks of modules 200, 210, 220, 230, 240, 250, and 260.
In step 310, the signature system 105 determines a prime number N. The N value is also greater than 3, greater than the n×m, and greater than or equal to the total number of signatures. In this example, signature system 105 sets N at 17. In step 315, signature system 105 determines primitive roots a and b. A primitive root is any number g with the property that any number coprime (relative prime) to N is congruent to a power of g modulo N. That is, for every integer a coprime to n, there is an integer k such that gk≡a (mod n). For example, the primitive roots of 17 are 3, 5, 6, 7, 10, 11, 12, 14. In step 315, signature system 105 automatically determines a plurality of primitive roots of N. In some embodiments, signature system 105 may calculate all of the primitive roots for N. Signature system 105 selects a and b from the calculated primitive roots. In this example, signature system 105 selects 3 and 5 as the values for a and b, respectively. In some embodiments, a and b may simply be prime numbers and not primitive roots.
In step 320, signature system 105 calculates a template matrix 321. Template matrix 321 has the same dimensions as the data matrices. In this example, the dimensions of each data matrix is 2×2, thus, template matrix 321 has the same dimensions. The positional values within template matrix 321 correspond to the value ai bj, where a and b correspond to the primitive roots previously determined (e.g., 3 and 5, respectively). The ‘i’ variable corresponds to the row position within template matrix 321, and the ‘j’ variable corresponds to the column position within template matrix 321. For example, for a 2×2 template matrix, where a=3 and b=5, the template matrix would equal [(15, 45), (75, 225)].
In step 325, signature system 105 characterizes the set of data matrices based on one or more properties of the received information. Properties may include: sparseness structures, positive numbers, negative numbers, repeated rows, repeated columns, irregular matrices, etc. In this example, the property of interest is sparseness structures, accordingly the signature system 105 may replace every non-zero entry in data matrix 306 and data matrix 307 with a ‘1,’ such that the set of data matrices only contains ‘1’ and ‘0’ values. The characterized matrices may then be applied to the template matrix to create a shadow matrix for each of the one or more data matrices. For example if the template matrix has a value of [(15, 45), (75, 225)], and a characterized data matrix has a value of [(1, 0), (0, 1)], the associated shadow matrix has a value of [(15, 0), (0, 225)]. In some embodiments, the shadow matrices are temporary and are not stored in persistent memory. In some embodiments, signature system 105 may omit the characterization step.
In step 330, signature system 105 calculates a hash function 331:
where Aij corresponds to the positional value within the data matrix being processed. Signature system 105 uses the pre-computed values of the positional elements within the template matrix to solve for 3i5j values. For large sets of data matrices, this may save a lot of system resources, as the values of aibj would not need to be recomputed each time the hash function operates on a new data matrix. The calculated signatures for data matrices 306 and 307 are 2 and 4, respectively. Additionally, in some embodiments, signature system 105 uses the one or more shadow matrices to calculate the signatures for each of their associated data matrices. These results may then be provided to the user. Detailed embodiments of the process discussed above are described with respect to
Additionally, in some embodiments, information is received from the user and the information is used to formulate searches to obtain the data set. In some embodiments, a set of search terms may be received from the user or through web crawling or spidering. In some embodiments, the server selects which data sources will be queried in the search process. For example, the server can select categories where the search may be performed, e.g., private party websites, company websites, cached information stored in a search database (such as those maintained at Google™ or Yahoo™), criminal databases, a credit agency data source, an insurance database, etc. The query may be submitted as Hypertext Transfer Protocol (HTTP) POST or GET mechanism, hypertext markup language (HTML), extensible markup language (XML), structured query language (SQL), plain text, Google Base, as terms structured with Boolean operators, or in any appropriate format using any appropriate query or natural language interface. The query may be submitted via the Internet, an intranet, or via any other appropriate coupling to a query processing engine associated with or contained within any of the data sources. The searches may be performed via “screen scraping” on a publicly available search engine, such as Google search or Yahoo search, or private search engines, such as Westlaw search or LexisNexis search. Searches may also be performed via a searching application program interface (API) or structured data exchange (such as eXtensible Markup Language).
In step 405, the obtained data is converted into one or more matrices. If the obtained data set is text based, the stream of text is broken up into tokens (e.g., each token representing a word, a phrase, or a symbol), which may be used to populate a data matrix, such that each received document has its own associated data matrix. Similarly if the data set is image based, the images may be broken up into tokens (e.g., each token representing a characteristic of the image), which are used to populate a data matrix, such that each received image has its own associated data matrix. For example, fingerprints may be associated with a data matrix by looking at features associated with each finger. Fingerprints may contain specific patterns, e.g., whorls, arches, and loops. The presence of these features can then be assigned a numeric value. The one or more data matrices may have the same dimensions, n×m.
In step 410, a prime number N is determined.
In step 515, a prime number that is larger than the total number of signatures is determined. For example if the total number of signatures is 100, potential values for N include 101, 103, etc. The potential value of N is selected and designated as N′. In step 520, it is determined whether N′ is greater than the dimensions of the data matrix, n×m. For example, if each data matrix is a 2×2 matrix, then N′ must be larger than 4. If N′<n×m, a new N′ that is greater than n×m (step 530) is automatically selected. If N′>n×m, then a determination is automatically made whether N′ is greater than 3 (step 530). If N′>3, the N′ value is selected for N (step 540). If N′<3, a prime number larger than 3 is automatically selected as the value for N′ (step 535). Then the new N′ value is selected as the N value (step 540).
Referring back to
In step 420, a template matrix that has the same dimensions as the one or more data matrices, n×m, is calculated. The elements of the template matrix are determined by the formula: aibij, where a and b correspond to the primitive roots previously determined. The ‘i’ variable corresponds to the row position within the template matrix, and the ‘j’ variable corresponds to the column position within the template matrix. For example, for a 2×2 template matrix, where a=2 and b=3, the template matrix would equal [(6, 12), (18, 36)], assuming all non-zero entries.
In step 425, the set of data matrices are characterized based on one or more properties of the received information. Properties may include: sparseness structures, positive numbers, negative numbers, repeated rows, repeated columns, irregular matrices, etc. For example, if the property of interest is sparseness structures, every non-zero entry in the one or more data matrices can be replaced with a ‘1,’ such that the set of data matrices only contains ‘1’ and ‘0’ values. Similarly, if the property of interest were positive numbers, every positive number in the one or more data matrices can be replaced with a ‘1,’ and any negative with a ‘0’ values. In some embodiments (not shown), the characterization step may be omitted. The characterized matrices may then be applied to the template matrix to create a shadow matrix for each of the one or more data matrices. For example if the template matrix has a value of [(15, 45), (75, 225)], and a characterized data matrix has a value of [(1, 0), (0, 1)], the associated shadow matrix has a value of [(15, 0), (0, 225)]. In some embodiments, the shadow matrices are temporary and are not stored in persistent memory.
In step 430, a hash function is calculated. A general form of the hash function is:
The hash function is calculated for each of the data matrices, where Aij corresponds to the positional value within the data matrix being processed. The modifier, if Aij=condition, modifies the hash function such that it creates signatures respective of the particular property of interest. For example, if the user is interested in sparseness structures, the condition may read “if Aij≠0.” In some embodiments, the pre-computed values of the positional elements within the template matrix are used to solve for aibj values. For large sets of data matrices, this may save a lot of system resources, as the values of ai bj would not need to be recomputed each time the hash function operates on a new data matrix. Additionally, in some embodiments, the one or more shadow matrices are used to calculate the signatures for each of their associated data matrices.
In step 435, it is determined if any collisions have occurred between the calculated set of signatures. A collision occurs when of the set of signatures, two or more signatures have the same value. If no collision occurs, then a unique signature has been created for each of the data matrices in the one or more data matrices, the set of signatures is referred to as a main set of signatures, and the process ends (440). If a collision occurs, in step 445, new values for a and b may be selected. The new values of a and b may be selected from the set of primitive roots of N previously determined. In some embodiments, a and b may be prime numbers and not primitive roots of N. For example, referring to
In step 450, a template matrix is calculated using the new a and b values. In step 455, the signatures for all the data matrices are re-calculated using the new a and b values. In some embodiments, the signatures are re-calculated with the new a and b values for only the data matrices where the collision originally occurred. In step 435, it is determined whether a collision occurs. If no collision occurs, then a unique signature has been created for each of the data matrices in the one or more data matrices, as noted previously the set of signatures is referred to as the main set of signatures, and the process ends (440). If a collision occurs again, the method may repeat steps 435, 445, 450, and 455 until a unique set of signatures is determined.
In step 605, the obtained one or more data items are converted into one or more data matrices, such that each data item has a corresponding data matrix. If the obtained data set is text based, the stream of text may be broken up into tokens (e.g., each token representing a word, a phrase, or a symbol), which may be used to populate a data matrix, such that each received document has its own associated data matrix. Similarly if the data set is image based, the images may be broken up into tokens (e.g., each token representing a characteristic of the image), which are used to populate a data matrix, such that each received image has its own associated data matrix. The one or more data matrices may have the same dimensions, n×m.
In step 610, the one or more data matrices are characterized based on one or more properties used to generate the signatures of the main data set. Properties may include: sparseness structures, positive numbers, negative numbers, repeated rows, repeated columns, irregular matrices, etc. For example, if signatures in the main signature set were created for the main data set based on sparseness structures, then the one or more data matrices would be characterized in a similar fashion, e.g., setting all non-zero elements to 1.
In step 615, a hash function is calculated for each of the one or more data matrices to be verified. A general form of the hash function is:
where Aij corresponds to the positional value within the data matrix being processed. The modifier, if Aij=condition, modifies the hash function such that it creates signatures respective of the particular property of interest. The a and b correspond to the primitive roots previously used, when calculating the main signature set. N corresponds to the prime number previously used when the main signature set was calculated. The template matrix corresponds to the template matrix used when determining the main signature set. The ‘i’ variable corresponds to the row position within the template matrix, and the ‘j’ variable corresponds to the column position within the template matrix. The template matrix has the same dimensions as the one or more data matrices to be verified, n×m. The elements of the template matrix are determined by the formula: aibj. In some embodiments, the template matrix is temporary and is not stored in persistent memory.
In step 620, the one or more signatures to be verified are compared against the main set of signatures. If a signature to be verified matches a signature from the main set of signatures, the signature is verified (step 635). In contrast, if a signature to be verified does not match a signature from the main set of signatures, no match occurs (step 640.
Other embodiments of the claimed inventions will be apparent to those skilled in the art from consideration of the specification and practice of the inventions disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the inventions being indicated by the following claims.
