I want your feedback to make the book better for you and other readers. If you find typos, errors, or places where the text may be improved, please let me know. The best ways to provide feedback are by GitHub or hypothes.is annotations.
Opening an issue or submitting a pull request on GitHub: https://github.com/isaactpetersen/Fantasy-Football-Analytics-Textbook
Adding an annotation using hypothes.is.
To add an annotation, select some text and then click the
symbol on the pop-up menu.
To see the annotations of others, click the
symbol in the upper right-hand corner of the page.
22 Decision Making in the Context of Uncertainty
22.1 Getting Started
22.1.1 Load Packages
22.2 Overview of Decision Making in Uncertainty
22.3 Wisdom of the Crowd
In many domains, the average of forecasters’ predictions is more accurate than the accuracy of the constituent individuals. In some domains, the average of non-expert forecasts is more accurate than the forecasts by individual experts. This phenomenon is called “collective intelligence”, the “wisdom of the crowd”, or the “wisdom of crowds” (Larrick et al., 2024; Rader et al., 2017; Simoiu et al., 2019; Surowiecki, 2005; Wagner & Vinaimont, 2010).
Aggregation of predictions from multiple people leverages several important features, including cognitive diversity and error cancellation.
Cognitive diversity refers to the representation of individuals with different perspectives because of their “differences in knowledge, training, experience, or thinking styles” (Rader et al., 2017, p. 8). Cognitive diversity is important because judgments from a cognitively homogeneous group will tend to err systematically. That is, they tend to err in the same direction—either consistently above or below the truth; thus, their errors are correlated. By contrast, a cognitively diverse group will not tend to err systematically. Individuals of a cognitively diverse group will bring different areas of expertise (e.g., determination of player skill, player opportunity, matchup strength, etc.) to bear in making their predictions and will thus make different mistakes (Larrick et al., 2024). Consequently, for the people comprising a cognitively diverse group, their judgments will tend to err randomly—where some people’s predictions fall above the truth and some people’s predictions fall below the truth—i.e., individual judgments “bracket” the truth (Mannes et al., 2014). Thus, judgments from a cognitively diverse group tend to have uncorrelated errors.
Error cancellation deals with the idea that, when individuals’ judgments bracket the truth and show random rather than systematic error, the average of the predictions will “cancel out” some of the errors so that the predictions average out to more closely approximate the truth. However, when individuals’ judgments do not bracket the truth, the average of the predictions will not cancel out the errors.
Averaging projections from individuals tends to yield predictions that are more accurate than the accuracy of most forecasters in the group (Mannes et al., 2014). Indeed, when at least some of of the projections bracket the truth, averaged predictions must be more accurate than the average individual forecaster—in terms of mean absolute error (MAE)—and averaged predictions are often much more accurate (Larrick et al., 2024). When referring to the accuracy of the “average individual forecaster”, we are referring to accuracy in terms of mean absolute error (MAE)—not to the accuracy of the forecaster at the 50th percentile. If none of the projections bracket the truth—e.g., all projections overestimate the truth—averaged predictions will be as accurate than the average individual forecaster in terms of mean absolute error (Larrick et al., 2024). In sum, “Averaging the answers of a crowd, therefore, ensures a level of accuracy no worse than the average member of the crowd and, in some cases, a level better than nearly all members” (Larrick et al., 2024, p. 126). Moreover, averaged projections tend to be more accurate than consensus-based judgments from groups of people that interact and discuss, due to cognitive biases associated with the social interaction among groups, such as herding in which people align their behavior with others (Mannes et al., 2014; Simoiu et al., 2019), though discussion can be helpful in some contexts (Larrick et al., 2024).
There are well-known prediction markets, in which people bet money to make predictions for various events, which allows determining the crowd-averaged prediction for events:
There are also betting markets for football:
- https://www.rotowire.com/betting/nfl/player-futures.php
- https://vegasprojections.com
- https://www.actionnetwork.com/nfl/props
Crowd-averaged projections tend to be most accurate when:
- the crowd consists of individuals who hold expertise in the domain such that they will make predictions that fall close to the truth
- there is relatively low variability in the expertise of the individual forecasters in terms of their ability to make accurate forecasts
- there is cognitive diversity among the forecasters
- the projections are made independently—i.e., the forecasters are not aware of others’ forecasts and do not discuss or interact with the other forecasters
- the bracketing rate—i.e., the frequency with which any two forecasters’ predictions fall on opposite sides of the truth—is high
- there are at least 5–10 sources of projections
However, the crowd is not more accurate than the expert or best forecaster in all situations or domains. For instance, the crowd tends to be less accurate than the (prospectively identified) best forecaster when there is great variability in forecasters’ expertise (in terms of the forecasters’ ability to forecast accurately) and when the bracketing rate is low (Mannes et al., 2014). Some forecasters may provide terrible projections; thus, including them in an average may make the average projections substantially less accurate. Thus, it may be necessary to examine the average of a “select crowd”, by aggregating the projections of the most consistently accurate forecasters (Mannes et al., 2014). Incorporating at least 5–10 forecasters leverages most of the benefits of the crowd; adding additional forecasters tends to result in diminishing returns (Larrick et al., 2024). However, to the extent that those who are most accurate in a given period reflects luck, you are better off averaging the predictions of all forecasters than selecting the forecasters who were most accurate in the most recent period (Larrick et al., 2024).
22.4 Accuracy of Fantasy Football Crowd Projections
Even though the crowd tends to be more accurate than individual forecasters, crowd-averaged projections (at least among experts) are not necessarily highly accurate. In fantasy football, [DESCRIBE THEIR ACCURACY–BOTH FOR ALL PLAYERS AND WHEN SUBSETTING TO THE TOP X PLAYERS]. Nevertheless, individual sources tend to be even less accurate. [DESCRIBE THEIR ACCURACY].
The petersenlab package (Petersen, 2024) has the wisdomOfCrowd() function that computes the overall accuracy of the crowd-averaged projections, including the bracketing rate of the individual projections.
Code
[1] -40 -30 -20 -10 10 20 30 40[1] 0[1] 50Code
[1] -61 -42 -42 -31 -11 -1 9 19[1] -21[1] 51Code
[1] -90 -80 -70 -60 -40 -30 -20 -10[1] -50[1] 50Even though some sources are more accurate than the average in a given year, they are not consistently more accurate than the average. Prediction involves a combination of luck and skill. In some years, a prediction will invariably do better than others, in part, based on luck. However, luck is unlikely to continue systematically into future years, so a source that got lucky in a given year is likely to regress to the mean in subsequent years. That is, determinining the most accurate source in a given year, after the fact, is not necessarily the same as identifying the most skilled forecaster. It is easy to identify the most accurate source after the fact, but it is challenging to predict, in advance, who the best forecaster will be (Larrick et al., 2024). It requires a large sample of predictions to determine whether a given forecaster is reliably (i.e., consistently) more accurate than other forecasters and to identify the most accurate forecaster (Larrick et al., 2024). Thus, it can be challenging to know, in advance, who the most accurate forecasters will be. Because average projections are as or more accurate than the average forecaster’s prediction, averaging projections across all forecasters is superior to choosing individual forecasters when the forecasters are roughly similar in forecasting ability or when it is hard to distinguish their ability in advance (Larrick et al., 2024).
The relatively modest accuracy of the projections by so-called fantasy “experts’” and of their average of their projections could occur for a number of reasons. One possibility is that the level of expertise of the “expert” forecasters in terms of being able to provide accurate forecasts is not strong. That is, because football performance and injuries are so challenging to predict, individual forecasters’ projections may not be particularly close to the truth.
A second possibility is that the bracketing rate of the predictions is not particularly high (Mannes et al., 2014). Even if the individual forecasters’ projections are not close to the truth, if ~50% of them overestimate the truth and the other 50% of the underestimate the truth, the average will more closely approximate the truth. However, if all forecasters overestimate the truth for a given player, averaging the projections will not necessarily lead to more accurate projections.
A third possibility is that the forecasts of the different experts are not independent.
Each of these possibilities is likely true to some degree. First, individuals’ predictions are unlikely to be highly accurate consistently. Second, there are many players who are systematically overpredicted (e.g., due to their injury) or underpredicted (e.g., due to their becoming the starter after a teammate becomes injured, is traded, etc.)—an example of overextremity miscalibration. In general, it is likely for players who are projected to score more points to be overpredicted and for players who are projected to score fewer points to be underpredicted [PRESENT CALIBRATION STATS]. Third, the experts may interact and discuss with one another. Interaction and discussion among experts may lead them to follow the herd and conform their projections to what each other predict. This has been terms informational influence and may reflect the anchoring and adjustment heuristic (Larrick et al., 2024). In any case, they are able to see each other’s projections and make change their projections, accordingly.
22.5 How Well Do People Incorporate Advice From Others?
An important question is how well people incorporate advice from others. In the context of fantasy football, advice might be sources of projections. In general, evidence from social psychology suggests that people tend to underweight how much weight they give to advice from others relative to their own opinions, a phenomeon called egocentric discounting (Larrick et al., 2024; Rader et al., 2017). People tend to weight others’ advice around 30% in terms of the proportion of a shift a person makes toward another person’s perspective, though this depends on the perceived accuracy of the advisor. Moreover, people frequently ignores others’ advice entirely. In general, people make use of crowds too little—they put too much weight on their own prediction and not enough weight on others’ predictions whose diversity can be leveraged for error cancellation (Larrick et al., 2024).
22.6 Suggestions
Based on the above dicussion, here are some suggestions for decision making in the context of uncertainty:
- seek advice from diverse perspectives and incorporate it into your decision making
- try to get opinions from others before you state your perspective and before the various sources of advice discuss with each other, to ensure independence of advice
- As noted by Kahneman (2011), “before an issue is discussed, all members of the committee should be asked to write a very brief summary of their position… The standard practice of open discussion gives too much weight to the opinions of those who speak early and assertively, causing others to line up behind them.” (p. 85).
- if some sources of advice (or projections) are clearly more skilled and accurate than others, you can average this “select” crowd of projections or give them greater weight in a weighted average
- if it is unclear whether or which sources are reliably more accurate than others, using a simple average across all sources (i.e., crowd-averaged projections) can be a useful approach that is as accurate as—if not more accurate than—the average individual forecaster
- incorporate at least 5–10 sources of projections
22.7 Conclusion
22.8 Session Info
R version 4.4.2 (2024-10-31)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 22.04.5 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so; LAPACK version 3.10.0
locale:
[1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
[4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
[7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
time zone: UTC
tzcode source: system (glibc)
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] petersenlab_1.1.0
loaded via a namespace (and not attached):
[1] utf8_1.2.4 generics_0.1.3 lattice_0.22-6 stringi_1.8.4
[5] digest_0.6.37 magrittr_2.0.3 evaluate_1.0.1 grid_4.4.2
[9] RColorBrewer_1.1-3 mvtnorm_1.3-2 fastmap_1.2.0 plyr_1.8.9
[13] jsonlite_1.8.9 nnet_7.3-19 backports_1.5.0 DBI_1.2.3
[17] Formula_1.2-5 gridExtra_2.3 purrr_1.0.2 fansi_1.0.6
[21] viridisLite_0.4.2 scales_1.3.0 pbivnorm_0.6.0 mnormt_2.1.1
[25] cli_3.6.3 mitools_2.4 rlang_1.1.4 munsell_0.5.1
[29] Hmisc_5.2-0 base64enc_0.1-3 mix_1.0-12 parallel_4.4.2
[33] tools_4.4.2 reshape2_1.4.4 checkmate_2.3.2 htmlTable_2.4.3
[37] dplyr_1.1.4 colorspace_2.1-1 ggplot2_3.5.1 vctrs_0.6.5
[41] R6_2.5.1 rpart_4.1.23 stats4_4.4.2 lifecycle_1.0.4
[45] stringr_1.5.1 htmlwidgets_1.6.4 psych_2.4.6.26 foreign_0.8-87
[49] cluster_2.1.6 pkgconfig_2.0.3 pillar_1.9.0 gtable_0.3.6
[53] Rcpp_1.0.13-1 glue_1.8.0 data.table_1.16.2 xfun_0.49
[57] tibble_3.2.1 tidyselect_1.2.1 rstudioapi_0.17.1 knitr_1.49
[61] xtable_1.8-4 nlme_3.1-166 htmltools_0.5.8.1 rmarkdown_2.29
[65] lavaan_0.6-19 compiler_4.4.2 quadprog_1.5-8