NSF Award
1919055
High quality forecasting is essential to any area where people make decisions based on uncertain information, such as in geopolitics, the environment, and medical and financial domains. Subjective judgements of probabilities that quantify uncertainty are often used to produce such predictions. Previous research has differentiated two qualities of forecasts: accuracy, and coherence, or the extent of logical and probabilistic consistency in judgments. Recent work has suggested that coherent judgments tend to also be more accurate. Yet there exists no measure of coherence that might be used to identify a "select crowd" of judges who are more accurate than the average. The purpose of this research is to develop a psychological measure of coherence and determine how it predicts accuracy. Specifically, the research will compare the relative accuracy of judgments that are weighed by this new measure with judgments weighed by existing methods of judgment aggregation. Results of this work can help improve judgment quality in highly consequential fields. <br/><br/>To answer this question, the researchers will create items that measure various aspects of probabilistic coherence. For example, items would test whether the probabilities assigned to all possible outcomes add to 1, as expected. Researchers will assess the psychometric properties - reliability, convergent and divergent validity - of the newly devised measure. To test the hypothesis that individuals with higher coherence scores are also more accurate, the researchers will weigh judgments from an incentivized forecasting tournament with this newly devised coherence measure. They will compare the accuracy of judgments weighed by this new method with other existing methods, such as linear and logarithmic averaging. The coherence measure may also produce a new, out-of-sample way of identifying the experts without additional knowledge of forecast performance, which will further our understanding of the role that individual-level coherence plays in maximizing judgment quality.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
