Patent Grant
7131060
The present invention relates to computer graphics displays. More specifically, the present invention relates to systems and methods for placing of icons in graphics displays.
Interactive computer graphics applications frequently require that two-dimensional labels be attached to objects that are displayed on a screen. Such labels provide information about the objects they are related to and are useful for users of such systems. These labels are typically drawn as a two-dimensional overlay on top of the two-dimensional or three-dimensional graphical object rendition that otherwise appears on the graphics display screen. One example of such an application is an air traffic control system where a graphical display of aircraft in flight is rendered on a display screen for use by an air traffic controller in controlling air traffic within a given region of interest. Within the region, for each real aircraft, there is rendered a displayed aircraft icon. Each aircraft icon has a two-dimensional label is attached to it, providing crucial information such as flight number, speed and altitude.
Since the airplanes move in and out of the region of interest over time, so do the aircraft icons and their respective labels. Because the air traffic controllers need ready access to all the information presented on their computer graphical display, it is important to prevent the labels from overlapping each other and other crucial features in the display such as the aircraft themselves. This can cause the information presented to become very difficult to read. In certain circumstances, overlap problems become so severe that the controllers cannot read the labels or other features on the display.
In some air traffic control computer graphical displays, the users must manually reposition the labels so that they do not overlap or hide other useful information. This step requires additional work on the user's part and is, in fact, not even a practical solution in the case of a three-dimensional perspective graphical display, where changing the viewpoint causes large changes in the spatial relationship of the displayed graphical objects.
The problem of how to reposition labels to alleviate the forgoing problems may appear intuitively simple at first, but it is actually quite difficult. The general problem is known to be ‘NP-hard’ by those of skill in the art. That means that for large problems it is not possible (within the current understanding of computer systems) to generate a system that can find the optimal solution to the problem in a reasonable amount of time. This is because finding the optimal solution requires searching through the entire potential search space. Previous automated label placement schemes do exist, however, all have limitations that result in them being less effective than is needed and desired.
In addition to the air traffic control systems, object label placement on computer graphical displays can be problematic in other applications as well. For example, similar labeling problems occur in command and control displays, which show the location of friendly and enemy units on two-dimensional or three-dimensional displays. Automatic label placement may also eventually be needed in ‘Augmented Reality’ displays (which display virtual labels over three-dimensional real objects in the environment as a person looks around). Aircraft head-up displays are another example of where object label placement may becomes increasingly important.
The object label placement problem for computer graphics applications is closely related to placement problems for static displays, such as maps. Cartographers have long faced the problem of assigning text labels to map features while avoiding label overlap and covering other important features. There are a number of references, but a good summary and evaluation of existing techniques is provided in: “An Empirical Study of Algorithms for Point-Feature Label Placement”, published by Jon Christensen, Joe Marks, and Stuart Shieber in ACM Transactions on Graphics 14, 3 (July 1995). An extended version by the same author is provided in Labeling Point Features on Maps and Diagrams. Technical Report TR-25-92. Center for Computer Technology, Harvard University, 1992 (revised June 1994). Another version of this discussion is available in Graphics Gems IV, Paul Heckbert (editor), Cambridge Academic Press, 1994.
In fact, label placement problems have proven to be NP-hard. For example, in a situation where each aircraft label can be placed in one of twenty positions adjacent to the aircraft, and where there are one-hundred total aircraft in the region of interest and on the graphical display screen, the search space consists of 20100 total sets of possible positions to evaluate. Clearly, this is far too many to evaluate in any reasonable time frame given the current state of computational technologies. Thus, an NP-hard problem means that there is no way to sufficiently prune this search space to avoid exponential growth caused by the combinatorics of the problem. A conceptual way of thinking about this is to imagine placing one label down on one aircraft. That takes up space that is no longer available for any other label. It is possible that, as labels continue to be placed, the situation will eventually arise where another aircraft label requires the space occupied by the first aircraft's label. Thus, decisions made early on in the label placement process can have ramifications much later in the placement process. The problem grows exponentially as the number of aircraft and possible label positions increase.
Existing label placement algorithms can be divided into two categories: exhaustive search and greedy search. Exhaustive search techniques essentially perform Depth First Searching with various bracketing and pruning heuristics. They are effective only on a small data set and are known to be less practical than greedy search algorithms. Greedy algorithms do not guarantee an optimal solution, however, finish in a practical amount of time by limiting the search space. A wide variety of greedy algorithms exist, ranging from random placement, to force-based methods, to mathematic optimization techniques, and discrete gradient descent (i.e. repositioning one or more randomly-chosen labels simultaneously and keeping the new configuration if it results in an overall improvement). Greedy algorithms in general have trouble escaping from local minima. Local minima are effectively non-optimal solutions that require repositioning large numbers of labels simultaneously to overcome the dangers associated with the solution. One particular algorithm is called ‘Simulated Annealing’ yields the best results. It is essentially a stochastic gradient descent search that sometimes allows the label placement to change to a worse solution, according to a particular schedule. ‘Simulated Annealing’ has the characteristic that as the time it is allowed to run approaches infinity, the solution generated approaches the optimum. The references above compare all published algorithms and conclude that Simulated Annealing finds a better solution than all other algorithms for all sizes of data sets.
The problem domain for label placement in an interactive computer graphics environment differs significantly from label placement for static maps and displays. In an interactive computer graphics environment, the three main differences are that; (1) the labeled objects change position with time, (2) little time is available for computing the label positions (a fraction of a second), and, (3) a human user observes the labels as they move around the display. Therefore, the object label placement system should minimize performing actions that distract the user from the underlying tasks. In the case of a static map, a placement system can run for several minutes, or longer, to generate a good solution. In an interactive computer graphics application, the time allotted for such calculations may have to be limited to 100 milliseconds, or less per iteration. Thus, the problem becomes finding a ‘good enough’ solution within the available time. Simulated annealing is not an appropriate solution for the interactive computer graphics application. The computation time required is too large to support in an interactive application, and it does not give any consideration to the cognitive distraction potentially caused by the solution.
Heuristic methods for interactive computer graphics applications focus on generating solutions with minimal computation and limited exploration of the total search space. Two main types of solutions have been tried, and they are force-based methods and fixed-position methods.
Force-based methods are commonly used in visualization of graphs and other objects. Labels and objects that are not to be covered are given a repulsion force. Other labels that encroach on such an object are ‘pushed away’ by a repulsion force inversely proportional to the separation distance. These forces are often modeled on springs or gravitational forces. In practice, force-based methods perform poorly respecting the label placement problem. They are vulnerable to getting stuck in local minima, even in simple cases. For example, when two aircraft cross paths, their associated labels can get stuck on the opposite sides because the repulsive forces prevent them from crossing paths. The optimum solution is for the two labels to swap positions, but the force-based technique instead pushes the labels in the opposite directions. Force-based schemes also perform poorly in the area of minimizing cognitive distractions. Labels stuck in between other labels or objects are jostled about; resulting in oscillations and wiggling that attract unwarranted user attention.
Another approach is the examination of a fixed number of label positions about an object. In general, this is a locally greedy approach with no backtracking. Each aircraft has a fixed number of positions where the labels can be placed in positions spaced equally about and relative to the aircraft icon. For example, there may be eight positions equally spaced about a circle with the aircraft icon located at the center of the circle. The system then iterates through all the aircraft icons on the display, one by one, in some order. For each aircraft, it checks all eight spaces to see which are empty, and puts the label in an empty position (if available, in the preferred location, such as the upper right corner). If none are empty, it picks one of the positions that causes the least amount of overlap. This method is efficient because of its simplicity and can generate reasonable results. However, it is very susceptible to local minima, and in practice the labels placed according to these solutions tend to oscillate.
Thus there is a need in the art for a method and apparatus for placing two-dimensional labels on an interactive computer graphics display.
The need in the art is addressed by the apparatus and methods of the present invention. In one embodiment, the invention comprises an apparatus for positioning labels among graphical elements on a computer graphics display and includes a means for identifying at least a first cluster of overlapping labels and graphical elements, a means for calculating new display coordinates for at least one label in the cluster, and a means for moving the label in accordance with the new display coordinates.
In a refinement of this invention, the means for identifying further comprises a means for sequentially selecting labels from a plurality of labels on the display, a means for testing each of the selected labels for overlap with other labels and graphical elements in the display, a means for accumulating an overlap score for each of the selected labels, a means for generating a list of other labels and graphical elements that overlap each of the selected labels, a means for comparing a plurality of the lists and accumulating cluster lists of overlapping labels and graphical elements, and a means for sorting a plurality of the cluster lists according to the number of entries in each.
In another refinement, the overlap score is based on the degree of severity of overlap between labels and graphical elements. In another refinement, the labels are determined to be overlapping other labels or graphical elements when they are mutually overlapping. In another refinement, the labels are determined to be overlapping other labels or graphical elements when they are mutually or transitively overlapping. In another refinement, the means for sorting orders the sort from largest cluster list to smallest cluster list. In another refinement, the means for calculating further comprises a means for comparing the degree of overlap of labels and graphical elements with the new display coordinates and the existing degree of overlap of labels and graphical elements.
In another refinement, the new display coordinates are calculated according to a stochastic method. In another refinement, the new display coordinates are calculated according to a heuristic method. In yet another refinement, the means for moving further comprises a means for interpolating a plurality of intermediate display coordinates between the existing display coordinates and the new display coordinates, and a means for sequentially placing the labels at each of the intermediate display coordinates before placing the labels at the new display coordinates, thereby smoothing the movements of the labels on the display.
The present invention also teaches several methods of positioning labels among graphical elements on a computer graphics display. A first method comprises the steps of identifying at least a first cluster of overlapping labels and graphical elements, calculating new display coordinates for at least one label in the cluster, and moving the label in accordance with the new display coordinates.
Another method of positioning labels among graphical elements on a computer graphics display is also taught. It comprises the steps of sequentially selecting labels from a plurality of labels on the display, testing each of the selected labels for overlap with other labels and graphical elements in the display, accumulating an overlap score for each of the selected labels, generating a list of other labels and graphical elements that overlap each of the selected labels, comparing a plurality of the lists and accumulating cluster lists of overlapping labels and graphical elements, sorting a plurality of the cluster lists according to the number of entries in each, calculating new display coordinates for the labels on a cluster by cluster basis, comparing, on a cluster by cluster basis, the degree of overlap of labels and graphical elements with the new display coordinates and the existing degree of overlap of labels and graphical elements, and if the new coordinates result in a reduction of the degree of overlap, moving the graphical elements to new positions according to the calculated display coordinates.
In another embodiment, the overlap score is based on the degree of severity of overlap between labels and graphical elements. In another refinement, the labels are determined to be overlapping other labels or graphical elements when they are mutually overlapping. In another refinement, the labels are determined to be overlapping other labels or graphical elements when they are mutually or transitively overlapping. In another refinement, the sorting is ordered from largest cluster list to smallest cluster list. In another refinement, the new display coordinates are calculated according to a stochastic method. In another refinement, the new display coordinates are calculated according to a heuristic method. In another refinement, the calculating of new display coordinates is ordered according to the cluster list. In yet another refinement, the moving step further comprises the steps of interpolating a plurality of intermediate display coordinates between the existing display coordinates and the new display coordinates, and sequentially placing the labels at each of the intermediate display coordinates before placing the labels at the new display coordinates, thereby smoothing the movements of the labels on the display. In yet another refinement, the method further includes the step of repeating the foregoing sequence of steps through a plurality of iterations.
These and other objects, aspects, features and advantages of the invention will become more apparent from the following drawings, description and claims.
Illustrative embodiments and exemplary applications will now be described with reference to the accompanying drawings to disclose the advantageous teachings of the present invention. While the present invention is described herein with reference to illustrative embodiments for particular applications, it should be understood that the invention is not limited thereto. Those having ordinary skill in the art and access to the teachings provided herein will recognize additional modifications, applications, and embodiments within the scope thereof and additional fields in which the present invention would be of significant utility.
The present invention teaches a novel apparatus and method for automatically placing two-dimensional labels that are associated with certain graphical objects in an interactive computer graphics display, so that the labels avoid overlapping each other and other objects on the display. This saves the user the effort of doing the task manually. The present invention comprises a computer-implemented software system that cooperates with a processor or controller at interactive rates in an interactive computer graphics application. One aspect of the present invention is the way in which it approaches the search space of potential locations for each label. As was discussed hereinbefore, the entire search space is typically too large to examine exhaustively in the time available. To alleviate this limitation, the present invention analyzes the positions of the labels recognizing clusters of labels that it can efficiently search, thereby reducing the total search space to a tractable level. In one embodiment, the present invention also minimizes the cognitive distractions to the user caused by changing label positions by changing their position smoothly and in one embodiment, generally avoiding moving labels unless it is advantageous to do so to prevent overlaps.
Reference is directed to
In a computer graphics display, the computer draws the aircraft icons and labels onto the display according to values provided through calculation. Reference is directed to
The relationship between each label and its icon is described mathematically, according to the distance from the nose of the aircraft icon to the label, and the angle of rotation of the label position relative to a reference angle (usually up or north). Thus the relative label position is described by the variables:
Given the known coordinates of the nose position of the aircraft icon, the relative location of the label, expressed in a Cartesian coordinate system (x,y) of pixels is:
There could be any number of positions for labels with respect to the nose of the aircraft icon. Since the number of positions is one dimension in the search space, it is useful to select a number of alternative positions that meets the needs of label placement systems without unduly expanding the search space dimensions.
Reference is directed to
One aspect of the present invention is an analysis of screen area problem locations, resulting in clusters of labels and graphic objects that are individually processed such most of the local minima problems are avoided. In one embodiment, the cluster approach advantageously avoids moving any labels, relative to their respective graphical objects, that are judged to not currently be causing an overlap problem. Labels that are moved to new locations are moved smoothly to minimize user distractions. While other systems simply jump the label from the old position to the new position, this routine smoothly moves the label from the old to the new position over time by an interpolation scheme.
The present invention does not always generate an optimum label placement solution, and there are situations where its performance is limited. For example, if the density of the objects to be labeled is so high that there is simply not enough display screen space to place the labels without significant overlap, then the present invention can, in one embodiment, continually move the labels around with a low probability of finding a good solution. However, no scheme that is practical to run at interactive rates will perform well under those circumstances, and in real applications users will not typically allow object densities to become this high because they will not be able to perform their tasks.
Reference is directed to
In an illustrative embodiment of the present invention for each aircraft, a label data structure of position variables is maintained for drawing the label associated with that aircraft to the display scene. Four pertinent variables in this data structure are:
All four of these variables are floating point numbers. The first two represent the start and destination values for the radius of the label with respect to the nose of the aircraft icon, in pixels. The last two represent the start and destination values for the angle, in radians. In one embodiment, the label radius has a minimum value of 20 pixels and a maximum value of 75 pixels, and the radius valves range from 0 to 2π radians. When an aircraft first becomes visible on the display scene, the start radius and start angle of the label are initialized to default base values such as, for example 30 pixels and π/4 radians, however, it is to be appreciated that other default base values can be chosen. The relative position with respect to the aircraft nose (in pixels) is specified in x and y offsets as follows:
x=radius*cos(angle)
y=radius*sin(angle)
Reference is directed to
Referring again to
The philosophy behind the present invention is to identify clusters of overlapping aircraft icons, labels and other graphical objects such as, for example, markers, collision points, the jetway line, etcetera, which are then searched for solutions to prevent overlap in a stochastic manner. The system of the present invention can solve the more difficult overlap problems first, such as, for example, problems involving clusters of aircraft icons with label and object overlaps. In one embodiment of the present invention, these clusters can be identified and solved individually without affecting all other aircraft icons on the display.
Again referring to
The details of how clusters are identified will be discussed with reference to
It is important to note, however, that the cluster lists may not include all of the visible aircraft. For example, if an aircraft does not overlap another aircraft or graphical object, and it itself is not overlapped by another aircraft or graphical object, then, in the present embodiment, it will not exist in a cluster and its label position will not be moved. This feature of the present embodiment allows the system to avoid moving labels that are not currently causing any problems, thus minimizing user distraction.
After the above described iterations are done, in the present embodiment, the list of clusters can be sorted by the number of aircraft in each cluster, so that the largest clusters are accorded a higher priority, and the smallest clusters are accorded a lesser priority, for purposes of order of later processing. The system can then proceed to manage the overlaps existing in the larger clusters first. It is to be appreciated that in alternate embodiments of the invention the clusters can be sorted according to different requirements or considerations, such as, for example, the extent to which an aircraft overlaps with graphical objects, or the extent to which clusters are in proximity to certain areas of the airport. It is to be further appreciated that in alternate embodiments of the invention the clusters need not be sorted.
The IDENTIFY CLUSTERS subroutine, step 68 in
What remains is to set new values for the label_r_dest and label_a_dest. In one embodiment the initial settings are simply the ones that previously existed. That is, initially:
The system can then process each cluster to determine whether the repositioning of the labels associated with the clusters is preferable. In the present embodiment, the system processes the largest clusters prior to processing the smaller clusters. For each cluster, several new sets of label positions, label_r_dest, label_a_dest are chosen and evaluated for all the aircraft icons in that cluster. These numbers of new sets chosen can vary, depending on the desired number of alternate destination values the user configures the system to supply. In one embodiment the number of sets created can be set to equal 25 if there is only one aircraft in the cluster. One aircraft can exist in a cluster in a situation where one aircraft icon or its label overlaps with graphical objects, such as, for example in the case of an aircraft overlapping with markers or collision points. In another embodiment, the number of sets created can equal 40 if 2 or 3 aircraft exist in the cluster, and the number of sets can equal 75 if 4 or more aircraft exist in a cluster. It is to be appreciated by those skilled in the art that the number of sets can differ from those above while remaining within the scope of the present invention.
In another embodiment, one of the sets of label positions can include only the base or default positions for each of the aircraft labels in the cluster, and another set of label positions can include only the current position of the aircraft labels in the cluster. By including a default set and a current position set, the system of the present invention insures that a new solution to reposition the labels having a greater overlap cost than the current solution will not be chosen. In the present embodiment, the remaining sets can be chosen randomly. In alternate embodiments, certain non-random algorithmic methods can be used to generate sets. In another embodiment, for each aircraft in the cluster, the proposed new label values for radius and angle can be chosen such that the radius stays within a predetermined minimum and maximum radius value, and the angle ranges from 0 to 2π radians.
After generating the sets of label positions, each set can then evaluated for a total penalty cost. This cost is the sum of all the penalty costs computed for each aircraft in the cluster, as computed by the ‘compute overlaps’ subroutine. The system can then selects the set that has the lowest overall cost and sets the label_r_dest and label_a_dest values for all aircraft in the cluster to the values in that set. With respect to the label positions of the aircraft not in the cluster currently being examined, such positions can remain, until the system processes such clusters, at their current label_r_dest and label_a_dest values. In the present embodiment, to minimize user distraction, the aircraft icons that are not part of any cluster need not have their labels repositioned. It is important to note however, that in alternate embodiment, all aircraft labels can be repositioned including those not existing in a cluster.
By using the foregoing approach, the system can complete the repositioning task in a reasonable amount of time. In the event, however, that the system does not find a preferred solution to the overlapping of icons in an iteration, in one embodiment, the system can reset the labels to their existing position. By doing so, the system will have a greater likelihood of finding a good solution during a subsequent iteration. For example, N iterations later, when the system reexamines the clusters, it will generate a new group of randomly chosen solution sets, thus increasing the probability that a preferred repositioning solution will be found. In one embodiment, the system spreads out the computation load over time, preserving the real time interaction while progressively refining the solution over time.
Referring again to
At step 74, the movement routine smoothly moves labels form their original position to their destination positions. In the present embodiment, during the iterations in which the new label positions are not being computed, the display smoothly moves the labels from their initial to their final positions. This is done by computing intermediate label positions based on the current iteration number, N. In an illustrative embodiment, the intermediate label positions are generated based on a linear interpolation from the start to the destination values. First, a fraction value is computed. The fraction is computed as:
Fraction=2.0*(cycle/N)
If (fraction>1.0)then fraction=1.0
Initially, the cycle variable is equal to one and then increments with N, where N is the number of iterations it takes to choose the destination label position, which in the present example can equal 30. When new label positions are chosen, the cycle variable can be set to zero.
Thereafter, each visible aircraft's label radius and angle is set by linearly interpolating between label_r_start and label_r_dest, and label_a_start and label_a_dest, using the fraction value. A fraction of 0.0 means that the radius and angle values are set to the start values and a fraction of 1.0 means they are set to the destination values. Fractions in between result in radii and angle values that are linear interpolations between the start and destination values. Since there is a discontinuity in angles between 0 and 2π, in one embodiment, the system can avoid potential discontinuity and select a rotation direction that minimizes the total rotation angle.
Reference is directed to
Reference is directed to
At step 136, the process checks to determine if the selected aircraft is a member of any other clusters. This could occur if a previously selected aircraft overlapped with the presently selected aircraft, causing it to be included in the previous aircraft's cluster list. If there is an overlap with another cluster at step 138, the clusters are merged at step 140. This is because of the transitive nature of clusters discussed hereinbefore. Flow then proceeds to step 146. On the other hand, at step 138, if there was no cluster overlap, then flow proceeds to step 142 where the process checks to determine if another aircraft remains to be tested. If so, then such aircraft is selected at step 144 and flow recycles to step 124. If no other aircraft remain, flow proceeds to step 146. At step 146, the clusters are sorted according to the number of graphical objects they contain. Then, the process returns at step 148.
Thus, the present invention has been described herein with reference to a particular embodiment for a particular application. It is therefore intended by the appended claims to cover any and all such applications, modifications and embodiments within the scope of the present invention.
Accordingly,
| Number | Name | Date | Kind |
|---|---|---|---|
| 5307455 | Higgins et al. | Apr 1994 | A |
| 5428733 | Carr | Jun 1995 | A |
| 5450535 | North | Sep 1995 | A |
| 5684940 | Freeman et al. | Nov 1997 | A |
| 5963731 | Sagawa et al. | Oct 1999 | A |
| 6081764 | Varon | Jun 2000 | A |
| 6091424 | Madden et al. | Jul 2000 | A |
| 6154219 | Wiley et al. | Nov 2000 | A |
| 6154220 | Prakriya et al. | Nov 2000 | A |
| 6295517 | Roy et al. | Sep 2001 | B1 |
| 6507838 | Syeda-Mahmood | Jan 2003 | B1 |
| 6525723 | Deering | Feb 2003 | B1 |
| 6562077 | Bobrow et al. | May 2003 | B1 |
| 6748447 | Basani et al. | Jun 2004 | B1 |
| 20040068758 | Daily et al. | Apr 2004 | A1 |
| Number | Date | Country |
|---|---|---|
| 2000047577 | Feb 2000 | JP |
