Patent Grant
11244238
A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
The subject matter discussed in the background section should not be assumed to be prior art merely as a result of its mention in the background section. Similarly, a problem mentioned in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognized in the prior art. The subject matter in the background section merely represents different approaches, which in and of themselves may also be inventions.
A database system can retrieve digital objects' information in response to a user's query. For example, when a user submits a query that specifies a company, a database system responds with the company's name, web site, number of employees, annual revenue, industry, sub-industry, phone number, street address, city, zip, state, and country that is stored by electronic records for business accounts, the accounts' contacts, and business leads in a customer relationship management (CRM) database. A database query may have an AND-OR structure, in which the query is an AND of clauses, with each clause corresponding to a particular attribute, and inside a clause is an OR of literals, with each literal corresponding to a particular value of that attribute. For example, the database query Company=(Salesforce OR Google) AND Job Level=(C-level or VP-level) has two clauses that are connected by an AND, with two literals connected by an OR in each clause. The term segment denotes a particular tuple of values for a corresponding subset of attributes, such as the segment (Company=Salesforce, Job Level=C-level).
In the following drawings like reference numbers are used to refer to like elements. Although the following figures depict various examples, the one or more implementations are not limited to the examples depicted in the figures.
General Overview
A model may be trained and used to enable a database system to respond to a user's search query by estimating the number of search results the query would yield for a given data set. Model training involves training a model from a full data set. Model use involves using the trained model to generate an overall result count (or a sufficiently good approximation) for a given query.
A basic trained model can count and persist the number of results in a data set for every possible segment. Next, the basic trained model can expand a given query into its segments, which is equivalent to the basic trained model taking a conjunctive normal form of a query and transforming the query into its disjunctive normal form. Then the basic trained model can sum up the counts of these segments to obtain the overall result count. For example, the basic trained model expands the query (Company=Salesforce OR Google, Department=Sales or Marketing) to its expanded segments (Company=Salesforce, Department=Sales), (Company=Salesforce, Department=Marketing), (Company=Google, Department=Sales), and (Company=Google, Department=Marketing). However, the basic trained model may have a significant number of segments, such as millions of segments for a realistic data set of companies. Furthermore, since the number of segments in the disjunctive normal form of a complex query (a query containing many clauses and many literals) may be significantly large, the basic trained model's process of identifying and summing up the segments' counts can consume a significant amount of system resources and time. The basic trained model's size may be reduced by maintaining counts for only complete segments, which are segments that involve all of the attributes. However, in addition to expanding a query to its complete segments, this reduced size model would have to also expand each segment to its complete extensions. Consequently, restricting segments to complete segments would not generally decrease the reduced model size exponentially, and expanding the query to complete segments could exponentially increase the time to get the overall results count.
A data science model can provide a significantly more compact model, significantly faster model training, and significantly faster results count computation (or at least approximation) for a given query than the basic and reduced size models. The data science model can also enable trading off the accuracy of a result set size approximation for model simplicity in a continuum.
In accordance with embodiments described herein, there are provided systems and methods for search query result set count estimation. A system parses a data set query that includes a first query attribute and a second query attribute. The system identifies a first hierarchy of connected nodes including a first node representing the first query attribute, and a second hierarchy of other connected nodes including a second node representing the second query attribute. The system identifies a directed arc connecting a first correlated node in the first hierarchy to a second correlated node in the second hierarchy. The system identifies cross-hierarchy probabilities of correlations between values of a first attribute that is represented by the first correlated node in the first hierarchy and values of a second attribute that is represented by the second correlated node in the second hierarchy. The system outputs an estimated count of a query result set, the estimated count generated from the cross-hierarchy probabilities, probabilities that the values of the first attribute are associated with values corresponding to the first node in the first hierarchy, and probabilities that the values of the second attribute are associated with values corresponding to the second node in the second hierarchy.
For example, a database system parses a user's query for a database's records of healthcare industry companies located in New York state. The database system identifies a geolocation hierarchy of connected nodes which include a state node and an industrial hierarchy of connected nodes which include an industry node. The database system identifies a directed arc connecting a sub-industry node in the industrial hierarchy to a city node in the geolocation hierarchy and another directed arc connecting the sub-industry node to a zip code node in the geolocation hierarchy. The database system identifies sub-industry node values which specify probabilities that healthcare industry companies are primarily correlated with specific zip code node values for the city node values of Chicago, Houston, San Francisco, and New York. Without having to execute the query, the database system uses the probabilities that specify healthcare sub-industries are located in New York city zip codes, probabilities that the healthcare industry attribute encompasses the identified sub-industry node values, and probabilities that the New York state attribute encompasses the identified city and zip code nodes' values, to estimate that the database contains records for 3,600 healthcare companies in New York state. The user who submitted the query can use this estimate to revise the query parameters to request fewer or more results than the estimated number of results would provide, such as revising the query to request the database records for small-sized healthcare companies located in New York state.
In an illustrative mathematical example, a table has two independent attributes X and Y and is comprised of n rows. A data science model counts the number of rows in the table in which X=a and Y=b. Since P(X=a, Y=b)=P(X=a)*P(Y=b), then the equation for n is: nab=n*P(X=a)*P(Y=b)=n*(na/n)*(nb/n)=na*nb/n, where na and nb are the number of rows, with X=a and Y=b, respectively. The count of a segment whose attributes are independent in the data set may be obtained by multiplying the counts of the components of the segment. Thus, counts of combinations need not be explicitly stored, nor even computed. By assuming that attributes are independent (even when they are not) the model can efficiently obtain an approximation to the exact count. When n>>2, the equation above for n may be generalized to yield a segment count under the assumption that all of the attributes are independent. Since this assumption is often too strong, and may result in an (approximate) count that is very inaccurate, a Bayesian network may be used to model situations which span the two extremes of “all attributes are independent” and “no two attributes are independent.” A Bayesian network is a directed graph whose nodes denote random variables and whose arcs encode certain dependencies among the random variables. This directed graph must be acyclic. The word “certain” is used intentionally, to avoid having to be precise, which is quite complex.
In an illustrative example, the variable Xi is a direct cause of the variable Xj, in the sense that certain values of Xi directly cause certain values of Xj to become more probable or less probable. A Bayesian network can capture this relationship between Xi and Xj by creating an arc from the node for Xi to the node for Xj. In an alternative example, if the values of Xi and Xj are correlated, but there is no known direction of causality, a Bayesian network might add an arc from the Xi node to the Xj node, or an arc from the Xj node to the Xi node, but not two arcs between both of these nodes because the directed graph is required to be acyclic. For any node Xi in the directed graph, π(Xi) denotes the nodes with arcs coming into Xi. Attached to the node Xi is a probability distribution over the values of Xi. The probabilities of the various values of Xi are allowed to depend only on the values of the attributes at the nodes in π(Xi). That is, P(Xi|X1, . . . , Xj≠i, . . . Xn)=P(Xi|π(Xi)). Consequently, the joint distribution over all the nodes in the Bayes network has the factored form P(X1, . . . Xn)=Πi P(Xi|π(i)). In this setting, a node corresponds to a searchable attribute, which is a clause in the afore-mentioned AND-OR query. The values at this node are the values possible for this attribute. In this notation, the AND-OR query may be represented by the equation Xi1=(vi1,2, . . . vi1,ni1), Xi2=(vi2,1, . . . vi2,ni2), . . . .
In this equation, as in the search query that this equation models, not all attributes need be present, nor all values of those of the attributes are present. The model's goal is to efficiently estimate the number of results this query would yield in a given data set. The estimate is described by the following formula: n*P(Xi1=(vi1,2, . . . vi1,ni1), Xi2=(vi2,1, . . . vi2,ni2), . . . ), where n is the total number of objects in the data set, and P(Xi1=(vi1,2, . . . vi1,ni1), Xi2=(vi2,1, . . . vi2,ni2), . . . is the probability that any particular object is in the result set of the query. To simplify notation, the values of the various attributes may be suppressed, keeping the values implicit, and the attributes may be renamed to χ1, . . . χk. Scripted notation here indicates the fact that the random variables take on sets of values, those in their corresponding clauses. Therefore, the model will compute n*P(χ1, . . . χk). P(χ1, . . . χk) may be computed using either of two extremes, plus various networks that span these two extremes.
The first extreme is when all attributes are independent. In this case, P(χ1, . . . χk)=P(χ1)*P(χ2)* . . . *P(χk). Each of the terms on the right-hand side of this equation may be computed independently and then multiplied together. Note that P(χi) may be computed efficiently, as it involves just summing up values of the probabilities of various values of X1i. This may be done very quickly, so long as no χi has a large set of values. Under this independent variable assumption, the model is very compact and trains very fast. This compact model models all distributions P(χ1), P(χn) independently. As an example, if each of the attributes has two values, storing this model involves storing just n numbers, one number for each P(Xi). If an attribute has only two values, and the model records the probability of one value, then the model also has the probability of the other value. Since only the above-mentioned n probabilities need to be calculated, the model trains very fast.
At the other extreme, there are no independencies among the attributes. In this case, P(χ1, . . . , χk) equals Σvk+1 Σvk+2 . . . Σvn P(χ1, . . . , χk|Xk+1 . . . n=vk+1 . . . n)*P(Xk+1 . . . n=vk+1 . . . n) This no independence formula is complex because it accounts for the possibility that any of the values of the attributes Xk+1, Xk+2, . . . Xn not mentioned in the query can influence the probability P(χ1, . . . , χk). At this extreme, the model might do poorly in model complexity, training complexity, and query time results count estimation. To be able to compute P(χ1, . . . , χk) for any subset of attributes χ1, . . . , χk and for any particular value set to each of these attributes, the model needs to effectively store P(X1 . . . n=v1 . . . n) for every tuple of values v1 . . . n=(v1, v2, . . . , vn) of these n attributes. Even when each of the n attributes contains only two values (which is the minimum possible), the model needs to store 2n probabilities. Therefore, the model's size may be huge for a large n. Since the models effectively needs to compute P(X1 . . . n=v1 . . . n) for every tuple of values v1 . . . n of these n attributes, the training time is exponential in n. As noted above, to compute P(χ1, . . . , χk), the model has n−k nested summations, each involving summing over the values of its corresponding attribute. So even when all attributes are binary-valued, and even if the model spends just one unit of time in the inner-most summation, computing P(χ1, . . . , ×k|Xk+1 . . . n=vk+1 . . . n)*P(Xk+1 . . . n=vk+1 . . . n), the model will be spending 2n−k units of time overall. The model cannot do any better at this extreme when computing exact result set counts. The model can generate approximate counts if such an amount of time cannot be spent on computations, or function under the assumption that most real-world use cases are not this extreme.
In some use cases, a system administrator may be able to design a reasonably good Bayes network structure manually, from domain knowledge. For a simple example, a model is based on the three attributes city, industry, and phone.
The following example is based on a globally comprehensive data set on companies at various locations. This data set mainly has the following firmographic attributes: company name, web site, number of employees (binned), annual revenue (binned), industry, sub-industry, phone number, street address, city, zip, state, and country. A system administrator can design a portion of a Bayes network based on the following domain knowledge.
In response to the query (city=CI, state=ST, country=CO), the model estimates the number of results in the data set that match this query. In view of the constraints in
Consequently, the portion of the model needed to compute P(city, state, country) is compact. Furthermore, P(city, state, country) for any one triplet (city=CI, state=ST, country=CO) may be computed very quickly. This just involves calculating each of the three factors in the right-hand side of the equation for P(city, state, country), and multiplying these factors together. Collectively this may be executed in practice in constant time, assuming the model is stored in a medium—such as RAM memory—supporting very fast lookups of each of these three factors.
Note that the equation for P(city, state, country) is rich enough to accommodate a city with the same name being in multiple states, possibly spanning multiple countries. As an example, Paris is not only a city in the country France, Paris is also a city in many different states in the United States of America. The query P(cityϵC, stateϵS), where C is some set of cities and S is some set of states, may be equated to Σc ϵc, sϵSP(state=s|city=c)*P(city=c). To execute this P(cityϵC, stateϵS) equation efficiently, the model takes advantage of the near-hierarchical relationship city→state. Specifically, for every city c, the model assumes that there are only a few states s for which P(state=s|city=c) is greater than 0, and usually there is just one state for each city. To leverage this property, a system administrator can create a forward index from the node city which maps every city c to the states s having a non-zero probability for P(state=s|city=c). The revised P(cityϵC, stateϵS) equation makes explicit how the model leverages this property efficiently: P(cityϵC, stateϵS)=ΣcϵC, sϵπ(c)Σsϵπ(c)∩S P(state=s|city=c) P(city=c), where π(c) denotes the states in which city c appears. In view of the assumed sparsity of π(c) the revised P(cityϵC, stateϵS) equation executes significantly faster than the previous P(cityϵC, stateϵS) equation. This revised equation may be generalized to any query on which the variables are on a directed path. The following illustrative example is based on the query P(xi1ϵSi1, xi2ϵSi2, . . . , xikϵSik), where (xi1, . . . , xik) forms a directed path in the Bayes network, and Si1, . . . , Sik are arbitrary subsets of the value sets at their corresponding nodes. For the query P(cityϵC, stateϵS, countryϵCTR), the generalized form of the previous revised P(cityϵC, stateϵS) equation is P(cityϵC, stateϵS, countryϵCTR)=Σcϵc, sϵπ(c)∩S ΣsϵS,crϵπ(s)∩CTR ΣcrϵCTR P(country=cr|state=s)*P(state=s|city=c)*P(city=c).
The directed graph is a collection of one or more weakly connected components, each modeling a near-hierarchy on its nodes. For example,
A system administrator may have two types of prior beliefs about two nodes A and B. ηA, B denotes a positive number capturing the strength of conviction in this belief. The model generates ηA, B ordered pairs (a, b) to capture this strength, where a denotes a value of A, and b denotes a value of B. The first type of prior belief is that B is an ancestor of A. The model generates ηA distinct synthetic values for A, and ηB distinct synthetic values for B. The system administrator chooses η and ηB to satisfy ηA=dBηB, where dB>1 denotes the average number of values in A that are descendants of any one value in B. If the system administrator has a prior point belief on what this value should be, the system administrator sets dB to that value. Otherwise, the system administrator somewhat arbitrarily sets dB=10. Next, to each value a of A, the model randomly assigns a value b of B. B=b will serve as the ancestor for A=a. Next, the model generates the ηA, B ordered pairs (a, b) as follows. For the values a1, . . . , aηA, B, the model samples from the distinct values of A with replacement. Next, for every i=1, . . . , ηA, B. the model sets bi to the value of B deemed the ancestor of ai. For example, since a system administrator has a prior belief that Country is an ancestor of City, and a rough belief that a country on average has 40 distinct cities, the system administrator sets dB=40. If ηA, B=1,000, then the model needs to generate 1,000 pseudo-examples to reflect this belief. Next, the model generates 400 different synthetic values for city, c1, . . . , c400, and 10 different synthetic values for country, ctr1, . . . , ctr10, respectively. Next, the model assigns each of the 400 distinct cities to a country, randomly chosen from ctr1, . . . ctr10. Finally, the model generates the 1,000 cities in the pseudo-sample by first sampling from the 400 distinct cities with replacement 1,000 times, and next setting the value of country to the country associated with this city.
The second type of prior belief is that neither A nor B is an ancestor of the other. As in the first type of prior belief, the model generates ηA distinct synthetic values for A and ηB distinct synthetic values for B, but ηA is not constrained by ηB in this case. Next, the model generates ηAB ordered pairs (a, b) by sampling ηAB values from the distinct values of A with replacement, by independently sampling ηAB values from the distinct values of B with replacement and pairing up the two samples. The intent is that the bivariate sample provide sufficient evidence for the many-to-many map between A and B to avoid the inference that one is an ancestor of the other. For example, the variable City has its 400 distinct synthetic values generated by the model, as in the previous example, and the variable Industry has 15 generated distinct synthetic values, and ηAB=1,000. The model generates 1,000 cities from the 400 distinct cities with replacement, and generates 1,000 industries from the 15 distinct industries with replacement. For i=1, . . . , 1,000, the model pairs the ith city with the ith industry.
The model adds the real data to the pseudo-examples, and learns the structure of the Bayes network of the desired form from the combined data. For each attribute, the model computes its set of distinct values in the combined data set, and creates an ordered partition ψ1, . . . , ψk of the attributes as follows. Two attributes are in the same partition if they have the same number of distinct values. The size of a partition is defined as the size of the value set of any of its attributes. Next, the model sorts the partitions in increasing order of size. The next step is better described in pseudo-code in view of its complexity:
The method is_near_IS_A(u, v) operates on a data set D={(xu, xv)}, where xu is a value of u and xv is a value of v. These pairs are assumed to be sampled from the marginal joint distribution P(Xu, Xv) of the two random variables Xu and Xv. Such a data set may be constructed by selecting just the columns Xu and Xv in the afore-mentioned combined data set intended for structure learning. From D, is_near_IS_A(u, v) first computes nv(xu), the number of distinct values of v that co-occur at least once in D with value xu of u. If nv(xu) equals 1 for every xu, this implies that every xu maps uniquely to a value of u. Also note that u contains more distinct values than v. From these two a model can conclude that u IS_A v. The is_near_IS_A(u, v) involves relaxing this hard constraint. In more detail, in the sample {nv(xu)|xuj is a distinct value of u, muv denotes the mean of this sample, and suv denotes its standard deviation. The method is_near_IS_A(u, v) inputs two thresholds m and s, and returns true if and only if muv≤m and suv≤s. Plausible examples of these thresholds include m=1.1 and s=1.
The structure induced to this point, via a combination of domain knowledge and data, is a collection of one or more weakly connected components, each a near-hierarchy of nodes. If there is only one component, the system administrator is done. If there are at least two components, the system administrator considers adding arcs to capture dependencies among these components, should there be any significant ones. A concrete example can clarify some subtle issues. In the structure of
If the city and sub-industry attributes are strongly dependent, and the city and industry attributes are also dependent, the algorithm presented below will add an arc between the nodes for the city and sub-industry attributes, suitably oriented, and stop. The influence of the city attribute on the industry attribute will get modeled by rippling the influence of the city attribute and the sub-industry attributes, which is explicitly modeled by the arc between their two corresponding nodes to the parent node industry of the child node sub-industry. The idea is to favor simpler models while remaining sensitive to dependencies.
The network input to this algorithm is a directed acyclic graph assumed to contain at least two weakly connected components. The algorithm does not change the structure within any component. The algorithm only adds arcs (as needed) that cross components as needed while maintaining the acyclicity of the resulting directed graph. The algorithm orders the weakly connected components in order of least number of nodes first, breaking ties arbitrarily. Next, the algorithm considers every pair of components (i, j), i<j in order of increasing i, and (for the i) increasing j. Note that i and j are the indices of the components in the afore-mentioned ordering. On a given pair (i, j) the algorithm tries to add at least one arc from component i to component j as described below.
The algorithm renames the two components the above-mentioned inner loop is operating on, A and B. The algorithm tries to add arcs from certain nodes in A to certain nodes in B. The algorithm creates partial orderings of the nodes in A and the nodes in B (separately). This is done using the so-called topological sort algorithm. As and Bs denote the partial ordering of each set of nodes respectively. Note that each partial ordering is a sequence of sets. The first element of the sequence is the set of nodes with indegree 0. The second element is the set of nodes at distance 1 from those in the first element. The third element is the set of nodes at distance 2 from those in the first element, and so on. Next, the algorithm works as described below.
As a test for “sufficient dependence” the model can use the mutual information measure from information theory, with a suitable threshold. In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, mutual information quantifies the “amount of information” (in units such as shannons, more commonly called bits) obtained about one random variable, through the other random variable. The concept of mutual information is intricately linked to that of entropy of a random variable, a fundamental notion in information theory, that defines the “amount of information” held in a random variable. Not limited to real-valued random variables like the correlation coefficient, mutual information is more general and determines how similar the joint distribution P(X, Y) is to the products of factored marginal distribution P(X)P(Y). The model can use a form of “normalized mutual information,” (NMI) specifically I(X;Y)/sqrt(H(X)*H(Y)). If the value of NMI(X, Y) is 0, then X and Y are independent. If the value of NMI(X, Y) is sufficiently positive, then X and Y are deemed to be sufficiently dependent. Choosing a suitable threshold is a modeling decision for a specific use case.
Simulating a run of the algorithm on the example of
The algorithm orders the weakly connected components to ensure that the resulting directed graph remains acyclic. The algorithm exits from the inner loop as soon as at least one arc has been added in a particular next-level iteration to keep the model from becoming overly complex, in particular to avoid modeling “higher-level” dependencies that may be inferred from a modeled ‘lower-level” dependency. When attempting to add arcs from one weakly connected component to another, the algorithm proceeds in the sequence of the partial orderings of the nodes to create arcs among the elements earlier in the partial orderings first, so their effects are easier to ripple down the partial orderings. For example, if the sub-industry attribute and the zip attribute were sufficiently dependent, and the sub-industry attribute and the city attribute were sufficiently dependent, then the algorithm would add two arcs to the network, resulting in the following extension of
In order to estimate the number of records in a data set that match the query (state=NY, industry=Healthcare), the model needs to compute P(state=NY, industry=Healthcare):
P(state=N, industry=H)=ΣsiΣcΣz(industry=H, sub=s, city=c, zip=z, state=N), where P(industry=H, sub=s, city=c, zip=z, state=N) equals P(sub=s) P(industry=H|sub=s) P(city=c|sub=s) P(zip=z|sub=s) P(state=N|city=c) P(state=N|zip=z).
In these equations, sub denotes sub-industry, and New York and Healthcare have been abbreviated to N and H, respectively. Since the query identifies the industry is H, the model follows the sub-industry→industry arc 126 in the reverse direction to find the sub-industries of industry H, as inferred by the model of
The following example is based on the industry Healthcare having two main sub-industries, Healthcare Institutions and Medical Devices, with the sub-industry Healthcare Institutions located primarily in two cities, San Francisco and New York, and located primarily in four zip codes, 10001, 11104, 94016, and 94188, while the sub-industry Medical Devices is located primarily in two cities, Houston and Chicago, and located primarily in three zip codes, 77001, 60007, and 60827. The combinations may be succinctly listed as
For this data, when a field contains a set of values, then all combinations of those values are taken with all combinations of the values from the other field in the same row. Therefore, the first row actually provides eight combinations since there are two cities and four zip codes, and the second row provides six combinations since there are two cities and three zip codes.
From the structure of the Bayes network in
Once the model determines the structure, the parameters are easy to determine. The model takes one more pass over the data set and from it compute, for every node i in the network, its local probability distribution P(Xi|π(i)). Here, the data set is the real data, minus the pseudo-examples. The pseudo-examples used generated synthetic values, which was for structure induction, but those values are not included in the local probability distributions. Some computations involve marginalizing over variables not in the query. Having an inverted index, such as an index that maps industries to their sub-industries, will help speed up such a query. To help speed up various queries, the model can create, for each node in the graph having at least one parent node, an inverted index that maps every value at node i to the tuples of values at the parents π(i) each of which has non-zero probability.
The following example of how these inverted indices help speed up the computations, based on the Bayes network of
The inverted indices will help the model compute the various sums efficiently. That is, for country C, its set of states st will be found efficiently, for a given state st, the set of pairs (city=ct, zip=zp) may be found efficiently, and sub-industries for industry I, may be found efficiently.
Using the example of
Since computing P(country=USA) requires the involved computation of P(state=ST), the model can cache the marginals P(state=ST) for states ST in the USA. Note that the model is not caching the full marginal distribution P(state), P(state), only its restriction to US states. This caching will make future queries needing calculation of P(state=ST) execute faster when ST is a state in the USA. The model can use the least recently used (LRU) scheme as the caching policy. That is, when memory needs to be re-claimed, the model ejects the least recently used marginals from the cache. Thus, probabilities needed in the computation of a recent query have a higher chance of being in the cache. This caching scheme tends to favor caching of marginals at nodes deep in a hierarchy, since these nodes get involved relatively more frequently in queries.
After the Bayes network has been trained, but before any queries have been executed, the model can “cold start” the cache by preloading the cache with marginals likely to have a relatively high hit rate. A sensible strategy is to favor loading marginals at nodes deeper in a hierarchy than shallower ones because the marginals at deeper nodes are involved in more queries than the marginals at shallower nodes. The model can implement this policy deterministically, such as starting loading from the deepest nodes first until the model exhausts the cache budget.
An alternative is to use a prior on the distribution of the queries. The model can then preload the cache by generating queries from this prior, such as simulated queries, and letting the query-time caching play out, which will tend to favor loading of marginals at deeper nodes if all nodes are roughly equally likely to be involved in a query. On the other hand, if the prior, which can leverage domain knowledge, favors certain nodes over others, the cache warming policy adjusts accordingly. Moreover, the prior can capture finer non-uniformities, such as those at the level of individual values, for example, that queries on (country=USA) are more popular than queries on certain other countries. Therefore, the prior-based approach is more general.
Systems and methods are provided for search query result set count estimation. As used herein, the term multi-tenant database system refers to those systems in which various elements of hardware and software of the database system may be shared by one or more customers. For example, a given application server may simultaneously process requests for a great number of customers, and a given database table may store rows for a potentially much greater number of customers. As used herein, the term query plan refers to a set of steps used to access information in a database system. Next, methods and systems for search query result set count estimation will be described with reference to example embodiments. The following detailed description will first describe a method for search query result set count estimation.
While one or more implementations and techniques are described with reference to an embodiment in which search query result set count estimation is implemented in a system having an application server providing a front end for an on-demand database service capable of supporting multiple tenants, the one or more implementations and techniques are not limited to multi-tenant databases nor deployment on application servers. Embodiments may be practiced using other database architectures, such as ORACLE®, DB2® by IBM and the like without departing from the scope of the embodiments claimed.
Any of the embodiments described herein may be used alone or together with one another in any combination. The one or more implementations encompassed within this specification may also include embodiments that are only partially mentioned or alluded to or are not mentioned or alluded to at all in this brief summary or in the abstract. Although various embodiments may have been motivated by various deficiencies with the prior art, which may be discussed or alluded to in one or more places in the specification, the embodiments do not necessarily address any of these deficiencies. In other words, different embodiments may address different deficiencies that may be discussed in the specification. Some embodiments may only partially address some deficiencies or just one deficiency that may be discussed in the specification, and some embodiments may not address any of these deficiencies.
After an influence between two attributes' values is identified, a hierarchy is optionally created, the hierarchy including a node representing an attribute, and another node representing another attribute, and a directed arc connecting the node representing the attribute to the other node representing the other attribute, block 204. The system creates a hierarchy of attributes. By way of example and without limitation, this can include the database system creating the hierarchy 118 that includes a zip code node 128, a state node 130, and a directed arc 132 connecting the zip code node 128 to the state node 130, as depicted in
Following the identification of an influence between two attributes' values, an additional influence by values of one of the attributes on probabilities of values of an additional attribute is optionally identified, block 206. The system identifies additional hierarchical attributes. In embodiments, this can include the database system identifying an influence by the state attribute's values on probabilities of the country attribute's values.
Having created a hierarchy which includes a node an attribute, the hierarchy is optionally modified to include an additional node representing an additional attribute, and an additional directed arc connecting the node representing the attribute to the additional node representing the additional attribute, block 208. The system modifies the hierarchy of attributes to include additional attributes. For example, and without limitation, this can include the database system modifying the hierarchy 118 which includes the state node 130 to include a country node 142, and a directed arc 144 connecting the state node 130 to the country node 142. While this example describes and
After hierarchies are created, a correlation between values of a first attribute that is represented by a first correlated node in a first hierarchy of connected nodes, and probabilities of values of a second attribute that is represented by a second correlated node in a second hierarchy of other connected nodes is optionally identified, block 210. The system identifies correlations between hierarchies of attributes. By way of example and without limitation, this can include the database system using a mutual information measure to identify a sufficient dependency between the sub-industry attribute's values and the probabilities of the city attribute's values. In another example, the database system uses a mutual information measure to identify a sufficient dependency between the sub-industry attribute's values and the probabilities of the zip code attribute's values. A correlation can be a relationship between data. A correlated node can be a connecting point that is arranged or classified according to relative importance or inclusiveness, the connecting point representing data that has a relationship with other data. A hierarchy of connected nodes can be an arrangement classification of connecting points according to relative importance or inclusiveness.
Identifying a correlation may be based on determining whether values associated with a lowest correlated node in a hierarchy of connected nodes have any correlation before determining whether values associated with a higher correlated node in the hierarchy of connected nodes have any correlation, determining whether values associated with a highest correlated node in the hierarchy of connected nodes have any correlation before determining whether values associated with a lowest correlated node in another hierarchy of other connected nodes have any correlation, determining whether values associated with a higher correlated node in the other hierarchy of other connected nodes have any correlation before determining whether values associated with a highest correlated node in the other hierarchy of other connected nodes have any correlation, and terminating the determining when a hierarchical level of correlation is identified. For example, the database system determines whether sufficient dependency exists between the sub-industry attribute's values and the city attribute's values and/or the zip code attribute's values to add a directed arc from the sub-industry node 138 to the city node 134 and/or add a directed arc from the sub-industry node 138 to the zip code node 128. If a sufficient dependency exists among the corresponding values to add either or both of these directed arcs, then the database system adds any required directed arcs and stops. If no directed arc is added, the database system determines whether sufficient dependency exists between the sub-industry attribute's values and the state attribute's values to add a directed arc from the sub-industry node 138 to the state node 130. If a sufficient dependency exists among the corresponding values to add this directed arc, then the database system adds the required directed arc and stops. If no directed arc is added, the database system determines whether sufficient dependency exists between the sub-industry attribute's values and the country attribute's values to add a directed arc from the sub-industry node 138 to the country node 142. If a sufficient dependency exists among the corresponding values to add this directed arc, then the database system adds the required directed arc and stops.
If no directed arc is added, the database system determines whether sufficient dependency exists between the industry attribute's values and the city attribute's values and/or the zip code attribute's values to add directed arcs from the industry node 140 to the city node 134 and/or from the industry node 140 to the zip code node 128. If a sufficient dependency exists among the corresponding values to add either or both of these directed arcs, then the database system adds any required directed arcs and stops. If no directed arc is added, the database system determines whether sufficient dependency exists between the industry attribute's values and the state attribute's values to add a directed arc from the industry node 140 to the state node 130. If a sufficient dependency exists among the corresponding values to add this directed arc, then the database system adds the required directed arc and stops. If no directed arc is added, the database system determines whether sufficient dependency exists between the industry attribute's values and the country attribute's values to add a directed arc from the industry node 140 to the country node 142. If a sufficient dependency exists among the corresponding values to add this directed arc, then the database system adds the required directed arc and stops.
The database system stops determining whether sufficient dependency exists between attributes' values as soon as one level of directed arcs has been added, which prevents the model from becoming overly complex, in particular to avoid modeling higher-level dependencies that may be inferred from a modeled lower-level dependency. When attempting to add directed arcs from one hierarchy of connected nodes to another hierarchy of connected nodes, the database system proceeds in the sequence of the partial orderings of the nodes to create directed arcs among the nodes earlier in the partial orderings first, so their effects are easier to ripple down the partial orderings.
Following the identification of a correlation between correlated nodes in different hierarchies, a directed arc connecting a first correlated node in a first hierarchy to a second correlated node in a second hierarchy is optionally created, block 212. The system records a correlation between hierarchies of attributes. In embodiments, this can include the database system creating the directed arc 122 connecting the sub-industry node 138 to the city node 134. In another example, the database system creates the directed arc 124 connecting the sub-industry node 138 to the zip code node 128.
Having modeled a data set, a data set query that includes a first query attribute and a second query attribute is parsed, block 214. The system processes queries' attributes. For example, and without limitation, this can include the database system parsing a user's query for a database's records of healthcare industry companies located in New York state. A data set query can be a request for information from a computer. A query attribute can be a piece of information that is identified in a data request and that determines the properties of a field in a database.
After parsing query attributes, a first hierarchy of connected nodes including a first node representing the first query attribute, and a second hierarchy of other connected nodes including a second node representing the second query attribute, are identified, block 216. The system identifies the queried attributes in hierarchies of attributes. By way of example and without limitation, this can include the database system identifying the geolocation hierarchy 118 of connected nodes which include the state node 130 and the industrial hierarchy 120 of connected nodes which include the industry node 140.
Following identification of hierarchies of connected nodes, a directed arc connecting a first correlated node in a first hierarchy to a second correlated node in a second hierarchy is identified, block 218. The system identifies a recorded correlation between hierarchies of attributes. In embodiments, this can include the database system identifying the directed arc 122 connecting the sub-industry node 138 in the industrial hierarchy 120 to the city node 134 in the geolocation hierarchy 118. In another example, the database system identifies the directed arc 124 connecting the sub-industry node 138 in the industrial hierarchy 120 to the zip code node 128 in the geolocation hierarchy 118. Although these examples describe and
Having identified directed arcs that cross hierarchies of connected nodes, cross-hierarchy probabilities of correlations between values of a first attribute that is represented by a first correlated node in a first hierarchy and values of a second attribute that is represented by a second correlated node in a second hierarchy are identified, block 220. The system identifies probabilities of the recorded correlation between hierarchies of attributes. For example, and without limitation, this can include the database system identifying sub-industry node 138 values which specify probabilities that healthcare industry companies are primarily correlated with specific zip code node 128 values for the city node 134 values of Chicago, Houston, San Francisco, and New York. A cross-hierarchy probability can be can be the extent to which something is likely to occur, measured by the ratio of the favorable cases to the whole number of cases possible, within arrangements or classifications of things according to relative importance or inclusiveness.
After identifying cross-hierarchy probabilities, an estimated count of a query result set is output, the estimated count generated from cross-hierarchy probabilities, probabilities that values of a first attribute are associated with values corresponding to a first node in a first hierarchy, and probabilities that values of a second attribute are associated with values corresponding to a second node in a second hierarchy, block 222. The system uses the probabilities of the recorded correlation between hierarchies of attributes to estimate a query result set count. By way of example and without limitation, this can include the database system using the probabilities that specify healthcare sub-industries are located in New York city zip codes, probabilities that the healthcare industry attribute encompasses the identified sub-industry node values, and probabilities that the New York state attribute encompasses the identified city and zip code nodes' values, to estimate that the database contains records for 3,600 healthcare companies in New York state, without having to execute the query. An estimated count can be an approximate calculation of the number of something. A query result set can be a group of information obtained in response to a data request.
For this simplified example, the database stores 100,000 company records, 6,000 (6%) of these records are for companies located in New York state, and 5,000 (5%) of these records are for companies in the healthcare industry. An actual count of the database records for healthcare companies indicates the following distributions of data:
When the database system computes the probabilities for healthcare companies, the combinations that are incompatible with the query attribute for New York state will have zero probability. For example, since the zip code 94016 is in the city of San Francisco, then the probability of New York state given the zip code 94016 is 0 because the dependency between the zip code attribute and the state attribute is modeled, and the zip code 94016 is in the state of California, not in the state of New York. For this example, one probability that values of an attribute are associated with values corresponding to a node in a hierarchy is the 0% probability that the correlated San Francisco value of the correlated city attribute is associated with the queried New York value corresponding to the queried state attribute. Continuing this example, another probability is the 100% probability that the correlated New York City value of the correlated city attribute is associated with the queried New York value corresponding to the queried state attribute. In another example, if the query specified New York City instead of specifying New York state, one probability that values of an attribute are associated with values corresponding to a node in a hierarchy is the 0% probability that the correlated San Francisco value of the correlated city attribute is associated with the queried New York City value corresponding to the queried city attribute. Continuing this other example, another probability is the 100% probability that the correlated New York City value of the correlated city attribute is associated with the queried New York City value corresponding to the queried city attribute. Alternatively, if the query specified New York City and the state attribute was correlated, one probability that values of an attribute are associated with values corresponding to a node in a hierarchy is the 0% probability that the correlated California value of the correlated state attribute is associated with the queried New York City value corresponding to the queried city attribute. Continuing this alternative example, another probability is the 100% probability that the correlated New York value of the correlated state attribute is associated with the queried New York City value corresponding to the queried city attribute. Therefore, these probabilities may be for correlated values of a correlated attribute that is hierarchically higher, lower, or the same as the queried value of the queried attribute.
Consequently, the database system identifies the combined 72.0% (48.0%+24.0%) probabilities that a healthcare industry company is located in one of the zip codes (10001 and 11104) in New York City, which is in New York state. The database system multiplies the combined probability of 72.0% by 5,000, which is the number of healthcare company records in the database, to estimate the count of 3,600 records in the query's result set, without having to execute the query.
If a database system erroneously assumed that the state attribute and the industry attribute were attribute that are completely independent of each other, then such a database system could multiply the 6% probability that companies are located in New York state by the 5,000 companies are in the healthcare industry to estimate the count of only 300 records in the query's result set.
The estimated count may be for external data set results, internal data set results, relational database table results, and/or a comparison of external data set results to internal data set results. For example, the database system estimates that the query result set from the user's CRM system will include 3,600 healthcare companies in New York state, and estimates that the query result set from a subscription data service will include 10,000 healthcare companies in New York state. Such a comparison may enable a user to estimate the number of external data set results that may already be stored in an internal data set, and revise the original query if fewer or more external data set results are desired relative to the number of these results that may already be stored in the internal data set.
The data set query can include a third query attribute that lacks any connection to any hierarchy of connected nodes, and generating the estimated count may be further based on independent probabilities associated with the third query attribute. For example, the user can revise the original query to also specify a small company for the company size attribute so that the query of the subscription data service returns fewer than the estimated count of 10,000 records in the query result set. Since the company size attribute is completely independent of the state attribute and the industry attribute, the database system multiplies the 50% independent probability for the small company size value by the original estimate of 10,000 records for healthcare companies in New York state to estimate the count of 5,000 records for the revised query's result set. An independent probability can be the extent to which something is likely to occur, measured by the ratio of the favorable cases to the whole number of cases possible, and which is not influenced or affected by other possibilities.
The method 200 may be repeated as desired. Although this disclosure describes the blocks 202-222 executing in a particular order, the blocks 202-222 may be executed in a different order. In other implementations, each of the blocks 202-222 may also be executed in combination with other blocks and/or some blocks may be divided into a different set of blocks.
System Overview
The environment 310 is an environment in which an on-demand database service exists. A user system 312 may be any machine or system that is used by a user to access a database user system. For example, any of the user systems 312 may be a handheld computing device, a mobile phone, a laptop computer, a work station, and/or a network of computing devices. As illustrated in
An on-demand database service, such as the system 316, is a database system that is made available to outside users that do not need to necessarily be concerned with building and/or maintaining the database system, but instead may be available for their use when the users need the database system (e.g., on the demand of the users). Some on-demand database services may store information from one or more tenants stored into tables of a common database image to form a multi-tenant database system (MTS). Accordingly, the “on-demand database service 316” and the “system 316” will be used interchangeably herein. A database image may include one or more database objects. A relational database management system (RDMS) or the equivalent may execute storage and retrieval of information against the database object(s). The application platform 318 may be a framework that allows the applications of the system 316 to run, such as the hardware and/or software, e.g., the operating system. In an embodiment, the on-demand database service 316 may include the application platform 318 which enables creation, managing and executing one or more applications developed by the provider of the on-demand database service, users accessing the on-demand database service via user systems 312, or third party application developers accessing the on-demand database service via the user systems 312.
The users of the user systems 312 may differ in their respective capacities, and the capacity of a particular user system 312 might be entirely determined by permissions (permission levels) for the current user. For example, where a salesperson is using a particular user system 312 to interact with the system 316, that user system 312 has the capacities allotted to that salesperson. However, while an administrator is using that user system 312 to interact with the system 316, that user system 312 has the capacities allotted to that administrator. In systems with a hierarchical role model, users at one permission level may have access to applications, data, and database information accessible by a lower permission level user, but may not have access to certain applications, database information, and data accessible by a user at a higher permission level. Thus, different users will have different capabilities with regard to accessing and modifying application and database information, depending on a user's security or permission level.
The network 314 is any network or combination of networks of devices that communicate with one another. For example, the network 314 may be any one or any combination of a LAN (local area network), WAN (wide area network), telephone network, wireless network, point-to-point network, star network, token ring network, hub network, or other appropriate configuration. As the most common type of computer network in current use is a TCP/IP (Transfer Control Protocol and Internet Protocol) network, such as the global internetwork of networks often referred to as the “Internet” with a capital “I,” that network will be used in many of the examples herein. However, it should be understood that the networks that the one or more implementations might use are not so limited, although TCP/IP is a frequently implemented protocol.
The user systems 312 might communicate with the system 316 using TCP/IP and, at a higher network level, use other common Internet protocols to communicate, such as HTTP, FTP, AFS, WAP, etc. In an example where HTTP is used, the user systems 312 might include an HTTP client commonly referred to as a “browser” for sending and receiving HTTP messages to and from an HTTP server at the system 316. Such an HTTP server might be implemented as the sole network interface between the system 316 and the network 314, but other techniques might be used as well or instead. In some implementations, the interface between the system 316 and the network 314 includes load sharing functionality, such as round-robin HTTP request distributors to balance loads and distribute incoming HTTP requests evenly over a plurality of servers. At least as for the users that are accessing that server, each of the plurality of servers has access to the MTS' data; however, other alternative configurations may be used instead.
In one embodiment, the system 316, shown in
One arrangement for elements of the system 316 is shown in
Several elements in the system shown in
According to one embodiment, each of the user systems 312 and all of its components are operator configurable using applications, such as a browser, including computer code run using a central processing unit such as an Intel Pentium® processor or the like. Similarly, the system 316 (and additional instances of an MTS, where more than one is present) and all of their components might be operator configurable using application(s) including computer code to run using a central processing unit such as the processor system 317, which may include an Intel Pentium® processor or the like, and/or multiple processor units. A computer program product embodiment includes a machine-readable storage medium (media) having instructions stored thereon/in which may be used to program a computer to perform any of the processes of the embodiments described herein. Computer code for operating and configuring the system 316 to intercommunicate and to process webpages, applications and other data and media content as described herein are preferably downloaded and stored on a hard disk, but the entire program code, or portions thereof, may also be stored in any other volatile or non-volatile memory medium or device as is well known, such as a ROM or RAM, or provided on any media capable of storing program code, such as any type of rotating media including floppy disks, optical discs, digital versatile disk (DVD), compact disk (CD), microdrive, and magneto-optical disks, and magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of media or device suitable for storing instructions and/or data. Additionally, the entire program code, or portions thereof, may be transmitted and downloaded from a software source over a transmission medium, e.g., over the Internet, or from another server, as is well known, or transmitted over any other conventional network connection as is well known (e.g., extranet, VPN, LAN, etc.) using any communication medium and protocols (e.g., TCP/IP, HTTP, HTTPS, Ethernet, etc.) as are well known. It will also be appreciated that computer code for implementing embodiments may be implemented in any programming language that may be executed on a client system and/or server or server system such as, for example, C, C++, HTML, any other markup language, Java™, JavaScript, ActiveX, any other scripting language, such as VBScript, and many other programming languages as are well known may be used. (Java™ is a trademark of Sun Microsystems, Inc.).
According to one embodiment, the system 316 is configured to provide webpages, forms, applications, data and media content to the user (client) systems 312 to support the access by the user systems 312 as tenants of the system 316. As such, the system 316 provides security mechanisms to keep each tenant's data separate unless the data is shared. If more than one MTS is used, they may be located in close proximity to one another (e.g., in a server farm located in a single building or campus), or they may be distributed at locations remote from one another (e.g., one or more servers located in city A and one or more servers located in city B). As used herein, each MTS could include one or more logically and/or physically connected servers distributed locally or across one or more geographic locations. Additionally, the term “server” is meant to include a computer system, including processing hardware and process space(s), and an associated storage system and database application (e.g., OODBMS or RDBMS) as is well known in the art. It should also be understood that “server system” and “server” are often used interchangeably herein. Similarly, the database object described herein may be implemented as single databases, a distributed database, a collection of distributed databases, a database with redundant online or offline backups or other redundancies, etc., and might include a distributed database or storage network and associated processing intelligence.
The user systems 312, the network 314, the system 316, the tenant data storage 322, and the system data storage 324 were discussed above in
The application platform 318 includes the application setup mechanism 438 that supports application developers' creation and management of applications, which may be saved as metadata into the tenant data storage 322 by the save routines 436 for execution by subscribers as one or more tenant process spaces 404 managed by the tenant management process 410 for example. Invocations to such applications may be coded using the PL/SOQL 434 that provides a programming language style interface extension to the API 432. A detailed description of some PL/SOQL language embodiments is discussed in commonly owned U.S. Pat. No. 7,730,478 entitled, METHOD AND SYSTEM FOR ALLOWING ACCESS TO DEVELOPED APPLICATIONS VIA A MULTI-TENANT ON-DEMAND DATABASE SERVICE, by Craig Weissman, filed Sep. 21, 2007, which is incorporated in its entirety herein for all purposes. Invocations to applications may be detected by one or more system processes, which manages retrieving the application metadata 416 for the subscriber making the invocation and executing the metadata as an application in a virtual machine.
Each application server 400 may be communicably coupled to database systems, e.g., having access to the system data 325 and the tenant data 323, via a different network connection. For example, one application server 4001 might be coupled via the network 314 (e.g., the Internet), another application server 400N-1 might be coupled via a direct network link, and another application server 400N might be coupled by yet a different network connection. Transfer Control Protocol and Internet Protocol (TCP/IP) are typical protocols for communicating between application servers 400 and the database system. However, it will be apparent to one skilled in the art that other transport protocols may be used to optimize the system depending on the network interconnect used.
In certain embodiments, each application server 400 is configured to handle requests for any user associated with any organization that is a tenant. Because it is desirable to be able to add and remove application servers from the server pool at any time for any reason, there is preferably no server affinity for a user and/or organization to a specific application server 400. In one embodiment, therefore, an interface system implementing a load balancing function (e.g., an F5 Big-IP load balancer) is communicably coupled between the application servers 400 and the user systems 312 to distribute requests to the application servers 400. In one embodiment, the load balancer uses a least connections algorithm to route user requests to the application servers 400. Other examples of load balancing algorithms, such as round robin and observed response time, also may be used. For example, in certain embodiments, three consecutive requests from the same user could hit three different application servers 400, and three requests from different users could hit the same application server 400. In this manner, the system 316 is multi-tenant, wherein the system 316 handles storage of, and access to, different objects, data and applications across disparate users and organizations.
As an example of storage, one tenant might be a company that employs a sales force where each salesperson uses the system 316 to manage their sales process. Thus, a user might maintain contact data, leads data, customer follow-up data, performance data, goals and progress data, etc., all applicable to that user's personal sales process (e.g., in the tenant data storage 322). In an example of a MTS arrangement, since all of the data and the applications to access, view, modify, report, transmit, calculate, etc., may be maintained and accessed by a user system having nothing more than network access, the user can manage his or her sales efforts and cycles from any of many different user systems. For example, if a salesperson is visiting a customer and the customer has Internet access in their lobby, the salesperson can obtain critical updates as to that customer while waiting for the customer to arrive in the lobby.
While each user's data might be separate from other users' data regardless of the employers of each user, some data might be organization-wide data shared or accessible by a plurality of users or all of the users for a given organization that is a tenant. Thus, there might be some data structures managed by the system 316 that are allocated at the tenant level while other data structures might be managed at the user level. Because an MTS might support multiple tenants including possible competitors, the MTS should have security protocols that keep data, applications, and application use separate. Also, because many tenants may opt for access to an MTS rather than maintain their own system, redundancy, up-time, and backup are additional functions that may be implemented in the MTS. In addition to user-specific data and tenant specific data, the system 316 might also maintain system level data usable by multiple tenants or other data. Such system level data might include industry reports, news, postings, and the like that are sharable among tenants.
In certain embodiments, the user systems 312 (which may be client systems) communicate with the application servers 400 to request and update system-level and tenant-level data from the system 316 that may require sending one or more queries to the tenant data storage 322 and/or the system data storage 324. The system 316 (e.g., an application server 400 in the system 316) automatically generates one or more SQL statements (e.g., one or more SQL queries) that are designed to access the desired information. The system data storage 324 may generate query plans to access the requested data from the database.
Each database can generally be viewed as a collection of objects, such as a set of logical tables, containing data fitted into predefined categories. A “table” is one representation of a data object, and may be used herein to simplify the conceptual description of objects and custom objects. It should be understood that “table” and “object” may be used interchangeably herein. Each table generally contains one or more data categories logically arranged as columns or fields in a viewable schema. Each row or record of a table contains an instance of data for each category defined by the fields. For example, a CRM database may include a table that describes a customer with fields for basic contact information such as name, address, phone number, fax number, etc. Another table might describe a purchase order, including fields for information such as customer, product, sale price, date, etc. In some multi-tenant database systems, standard entity tables might be provided for use by all tenants. For CRM database applications, such standard entities might include tables for Account, Contact, Lead, and Opportunity data, each containing pre-defined fields. It should be understood that the word “entity” may also be used interchangeably herein with “object” and “table”.
In some multi-tenant database systems, tenants may be allowed to create and store custom objects, or they may be allowed to customize standard entities or objects, for example by creating custom fields for standard objects, including custom index fields. U.S. Pat. No. 7,779,039, filed Apr. 2, 2004, entitled “Custom Entities and Fields in a Multi-Tenant Database System”, which is hereby incorporated herein by reference, teaches systems and methods for creating custom objects as well as customizing standard objects in a multi-tenant database system. In certain embodiments, for example, all custom entity data rows are stored in a single multi-tenant physical table, which may contain multiple logical tables per organization. It is transparent to customers that their multiple “tables” are in fact stored in one large table or that their data may be stored in the same table as the data of other customers.
While one or more implementations have been described by way of example and in terms of the specific embodiments, it is to be understood that one or more implementations are not limited to the disclosed embodiments. To the contrary, it is intended to cover various modifications and similar arrangements as would be apparent to those skilled in the art. Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5577188 | Zhu | Nov 1996 | A |
| 5608872 | Schwartz | Mar 1997 | A |
| 5649104 | Carleton | Jul 1997 | A |
| 5715450 | Ambrose et al. | Feb 1998 | A |
| 5761419 | Schwartz | Jun 1998 | A |
| 5819038 | Carleton | Oct 1998 | A |
| 5821937 | Tonelli et al. | Oct 1998 | A |
| 5831610 | Tonelli et al. | Nov 1998 | A |
| 5873096 | Lim et al. | Feb 1999 | A |
| 5918159 | Fomukong et al. | Jun 1999 | A |
| 5963953 | Cram et al. | Oct 1999 | A |
| 6092083 | Brodersen et al. | Jul 2000 | A |
| 6161149 | Achacoso et al. | Dec 2000 | A |
| 6169534 | Raffel et al. | Jan 2001 | B1 |
| 6178425 | Brodersen et al. | Jan 2001 | B1 |
| 6189011 | Lim et al. | Feb 2001 | B1 |
| 6216135 | Brodersen et al. | Apr 2001 | B1 |
| 6233617 | Rothwein et al. | May 2001 | B1 |
| 6266669 | Brodersen et al. | Jul 2001 | B1 |
| 6295530 | Ritchie et al. | Sep 2001 | B1 |
| 6324568 | Diec et al. | Nov 2001 | B1 |
| 6324693 | Brodersen et al. | Nov 2001 | B1 |
| 6336137 | Lee et al. | Jan 2002 | B1 |
| D454139 | Feldcamp et al. | Mar 2002 | S |
| 6367077 | Brodersen et al. | Apr 2002 | B1 |
| 6393605 | Loomans | May 2002 | B1 |
| 6405220 | Brodersen et al. | Jun 2002 | B1 |
| 6434550 | Warner et al. | Aug 2002 | B1 |
| 6446089 | Brodersen et al. | Sep 2002 | B1 |
| 6535909 | Rust | Mar 2003 | B1 |
| 6549908 | Loomans | Apr 2003 | B1 |
| 6553563 | Ambrose et al. | Apr 2003 | B2 |
| 6560461 | Fomukong et al. | May 2003 | B1 |
| 6574635 | Stauber et al. | Jun 2003 | B2 |
| 6577726 | Huang et al. | Jun 2003 | B1 |
| 6601087 | Zhu | Jul 2003 | B1 |
| 6604117 | Lim et al. | Aug 2003 | B2 |
| 6604128 | Diec | Aug 2003 | B2 |
| 6609150 | Lee et al. | Aug 2003 | B2 |
| 6621834 | Scherpbier | Sep 2003 | B1 |
| 6654032 | Zhu | Nov 2003 | B1 |
| 6665648 | Brodersen et al. | Dec 2003 | B2 |
| 6665655 | Warner et al. | Dec 2003 | B1 |
| 6684438 | Brodersen et al. | Feb 2004 | B2 |
| 6711565 | Subramaniam et al. | Mar 2004 | B1 |
| 6724399 | Katchour et al. | Apr 2004 | B1 |
| 6728702 | Subramaniam et al. | Apr 2004 | B1 |
| 6728960 | Loomans et al. | Apr 2004 | B1 |
| 6732095 | Warshavsky et al. | May 2004 | B1 |
| 6732100 | Brodersen et al. | May 2004 | B1 |
| 6732111 | Brodersen et al. | May 2004 | B2 |
| 6754681 | Brodersen et al. | Jun 2004 | B2 |
| 6763351 | Subramaniam et al. | Jul 2004 | B1 |
| 6763501 | Zhu | Jul 2004 | B1 |
| 6768904 | Kim | Jul 2004 | B2 |
| 6772229 | Achacoso et al. | Aug 2004 | B1 |
| 6782383 | Subramaniam et al. | Aug 2004 | B2 |
| 6804330 | Jones et al. | Oct 2004 | B1 |
| 6826565 | Ritchie et al. | Nov 2004 | B2 |
| 6826582 | Chatterjee et al. | Nov 2004 | B1 |
| 6826745 | Coker | Nov 2004 | B2 |
| 6829655 | Huang et al. | Dec 2004 | B1 |
| 6842748 | Warner et al. | Jan 2005 | B1 |
| 6850895 | Brodersen et al. | Feb 2005 | B2 |
| 6850949 | Warner et al. | Feb 2005 | B2 |
| 7062502 | Kesler | Jun 2006 | B1 |
| 7340411 | Cook | Mar 2008 | B2 |
| 7356482 | Frankland et al. | Apr 2008 | B2 |
| 7401094 | Kesler | Jul 2008 | B1 |
| 7620655 | Larsson | Nov 2009 | B2 |
| 7698160 | Beaven et al. | Apr 2010 | B2 |
| 7779475 | Jakobson et al. | Aug 2010 | B2 |
| 7851004 | Hirao et al. | Dec 2010 | B2 |
| 8010663 | Firminger et al. | Aug 2011 | B2 |
| 8014943 | Jakobson | Sep 2011 | B2 |
| 8015495 | Achacoso et al. | Sep 2011 | B2 |
| 8032297 | Jakobson | Oct 2011 | B2 |
| 8082301 | Ahlgren et al. | Dec 2011 | B2 |
| 8095413 | Beaven et al. | Jan 2012 | B1 |
| 8095594 | Beaven et al. | Jan 2012 | B2 |
| 8209308 | Jakobson et al. | Jun 2012 | B2 |
| 8275836 | Beaven et al. | Sep 2012 | B2 |
| 8484111 | Frankland et al. | Jul 2013 | B2 |
| 8490025 | Jakobson et al. | Jul 2013 | B2 |
| 8504945 | Jakobson et al. | Aug 2013 | B2 |
| 8510664 | Rueben et al. | Aug 2013 | B2 |
| 8566301 | Rueben et al. | Oct 2013 | B2 |
| 8646103 | Jakobson et al. | Feb 2014 | B2 |
| 9244976 | Zhang | Jan 2016 | B1 |
| 10387578 | Xiao | Aug 2019 | B1 |
| 20010044791 | Richter et al. | Nov 2001 | A1 |
| 20020072951 | Lee et al. | Jun 2002 | A1 |
| 20020082892 | Raffel | Jun 2002 | A1 |
| 20020129352 | Brodersen et al. | Sep 2002 | A1 |
| 20020140731 | Subramaniam et al. | Oct 2002 | A1 |
| 20020143997 | Huang et al. | Oct 2002 | A1 |
| 20020162090 | Parnell et al. | Oct 2002 | A1 |
| 20020165742 | Robins | Nov 2002 | A1 |
| 20030004971 | Gong | Jan 2003 | A1 |
| 20030018705 | Chen et al. | Jan 2003 | A1 |
| 20030018830 | Chen et al. | Jan 2003 | A1 |
| 20030066031 | Laane et al. | Apr 2003 | A1 |
| 20030066032 | Ramachandran et al. | Apr 2003 | A1 |
| 20030069936 | Warner et al. | Apr 2003 | A1 |
| 20030070000 | Coker et al. | Apr 2003 | A1 |
| 20030070004 | Mukundan et al. | Apr 2003 | A1 |
| 20030070005 | Mukundan et al. | Apr 2003 | A1 |
| 20030074418 | Coker et al. | Apr 2003 | A1 |
| 20030120675 | Stauber et al. | Jun 2003 | A1 |
| 20030151633 | George et al. | Aug 2003 | A1 |
| 20030159136 | Huang et al. | Aug 2003 | A1 |
| 20030187921 | Diec et al. | Oct 2003 | A1 |
| 20030189600 | Gune et al. | Oct 2003 | A1 |
| 20030204427 | Gune et al. | Oct 2003 | A1 |
| 20030206192 | Chen et al. | Nov 2003 | A1 |
| 20040001092 | Rothwein et al. | Jan 2004 | A1 |
| 20040015981 | Coker et al. | Jan 2004 | A1 |
| 20040027388 | Berg et al. | Feb 2004 | A1 |
| 20040128001 | Levin et al. | Jul 2004 | A1 |
| 20040186860 | Lee et al. | Sep 2004 | A1 |
| 20040193510 | Catahan et al. | Sep 2004 | A1 |
| 20040199489 | Barnes-Leon et al. | Oct 2004 | A1 |
| 20040199536 | Barnes-Leon et al. | Oct 2004 | A1 |
| 20040249854 | Barnes-Leon et al. | Dec 2004 | A1 |
| 20040260534 | Pak et al. | Dec 2004 | A1 |
| 20040260659 | Chan et al. | Dec 2004 | A1 |
| 20040268299 | Lei et al. | Dec 2004 | A1 |
| 20050050555 | Exley et al. | Mar 2005 | A1 |
| 20050091098 | Brodersen et al. | Apr 2005 | A1 |
| 20090063415 | Chatfield et al. | Mar 2009 | A1 |
| 20090100342 | Jakobson | Apr 2009 | A1 |
| 20090177744 | Marlow et al. | Jul 2009 | A1 |
| 20090259679 | Thiesson | Oct 2009 | A1 |
| 20120233137 | Jakobson et al. | Sep 2012 | A1 |
| 20120278321 | Traub | Nov 2012 | A1 |
| 20130218948 | Jakobson | Aug 2013 | A1 |
| 20130218949 | Jakobson | Aug 2013 | A1 |
| 20130218966 | Jakobson | Aug 2013 | A1 |
| 20140040269 | Sundaresan | Feb 2014 | A1 |
| 20140359537 | Jakobson et al. | Dec 2014 | A1 |
| 20150007050 | Jakobson et al. | Jan 2015 | A1 |
| 20150095162 | Jakobson et al. | Apr 2015 | A1 |
| 20150142807 | Hofmann | May 2015 | A1 |
| 20150172563 | Jakobson et al. | Jun 2015 | A1 |
| 20180336202 | Jahanbakhsh | Nov 2018 | A1 |
| 20180376000 | Molander | Dec 2018 | A1 |
| Entry |
|---|
| Selectivity Estimation using Probabilistic Models ACM 1581133324/01/05 ACM SIGMOD May 21-24, 2001 (Year: 2001). |
| U.S. Appl. No. 13/986,251, filed Apr. 16, 2013. |
| Number | Date | Country | |
|---|---|---|---|
| 20190236475 A1 | Aug 2019 | US |
