I need your help!

I want your feedback to make the book better for you and other readers. If you find typos, errors, or places where the text may be improved, please let me know. The best ways to provide feedback are by GitHub or hypothes.is annotations.

You can leave a comment at the bottom of the page/chapter, or open an issue or submit a pull request on GitHub: https://github.com/isaactpetersen/Fantasy-Football-Analytics-Textbook

Hypothesis Alternatively, you can leave an annotation using hypothes.is. To add an annotation, select some text and then click the symbol on the pop-up menu. To see the annotations of others, click the symbol in the upper right-hand corner of the page.

24  Decision Making in the Context of Uncertainty

24.1 Getting Started

24.1.1 Load Packages

Code
library("petersenlab")
library("tidyverse")

24.2 Overview of Decision Making in Uncertainty

24.3 Wisdom of the Crowd

In many domains, the average of forecasters’ predictions is more accurate than the accuracy of the constituent individuals. In some domains, the average of non-expert forecasts is more accurate than the forecasts by individual experts. This phenomenon is called “collective intelligence”, the “wisdom of the crowd”, or the “wisdom of crowds” (Larrick et al., 2024; Rader et al., 2017; Simoiu et al., 2019; Surowiecki, 2005; Wagner & Vinaimont, 2010).

Aggregation of predictions from multiple people leverages several important features, including cognitive diversity and error cancellation.

Cognitive diversity refers to the representation of individuals with different perspectives because of their “differences in knowledge, training, experience, or thinking styles” (Rader et al., 2017, p. 8). Cognitive diversity is important because judgments from a cognitively homogeneous group will tend to err systematically. That is, they tend to err in the same direction—either consistently above or below the truth; thus, their errors are correlated. By contrast, a cognitively diverse group will not tend to err systematically. Individuals of a cognitively diverse group will bring different areas of expertise (e.g., determination of player skill, player opportunity, matchup strength, etc.) to bear in making their predictions and will thus make different mistakes (Larrick et al., 2024). Consequently, for the people comprising a cognitively diverse group, their judgments will tend to err randomly—where some people’s predictions fall above the truth and some people’s predictions fall below the truth—i.e., individual judgments “bracket” the truth (Mannes et al., 2014). Thus, judgments from a cognitively diverse group tend to have uncorrelated errors.

Error cancellation deals with the idea that, when individuals’ judgments bracket the truth and show random rather than systematic error, the average of the predictions will “cancel out” some of the errors so that the predictions average out to more closely approximate the truth. However, when individuals’ judgments do not bracket the truth, the average of the predictions will not cancel out the errors.

Averaging projections from individuals tends to yield predictions that are more accurate than the accuracy of most forecasters in the group (Mannes et al., 2014). Indeed, when at least some of of the projections bracket the truth, averaged predictions must be more accurate than the average individual forecaster—in terms of mean absolute error (MAE)—and averaged predictions are often much more accurate (Larrick et al., 2024). When referring to the accuracy of the “average individual forecaster”, we are referring to accuracy in terms of mean absolute error (MAE)—not to the accuracy of the forecaster at the 50th percentile. If none of the projections bracket the truth—e.g., all projections overestimate the truth—averaged predictions will be as accurate than the average individual forecaster in terms of mean absolute error (Larrick et al., 2024). In sum, “Averaging the answers of a crowd, therefore, ensures a level of accuracy no worse than the average member of the crowd and, in some cases, a level better than nearly all members” (Larrick et al., 2024, p. 126). Moreover, averaged projections tend to be more accurate than consensus-based judgments from groups of people that interact and discuss, due to cognitive biases associated with the social interaction among groups, such as herding in which people align their behavior with others (Mannes et al., 2014; Simoiu et al., 2019), though discussion can be helpful in some contexts (Larrick et al., 2024).

There are well-known prediction markets, in which people bet money to make predictions for various events, which allows determining the crowd-averaged prediction for events:

There are also betting markets for football:

Crowd-averaged projections tend to be most accurate when:

  • the crowd consists of individuals who hold expertise in the domain such that they will make predictions that fall close to the truth
  • there is relatively low variability in the expertise of the individual forecasters in terms of their ability to make accurate forecasts
  • there is cognitive diversity among the forecasters
  • the projections are made independently—i.e., the forecasters are not aware of others’ forecasts and do not discuss or interact with the other forecasters
  • the bracketing rate—i.e., the frequency with which any two forecasters’ predictions fall on opposite sides of the truth—is high
  • there are at least 5–10 sources of projections

However, the crowd is not more accurate than the expert or best forecaster in all situations or domains. For instance, the crowd tends to be less accurate than the (prospectively identified) best forecaster when there is great variability in forecasters’ expertise (in terms of the forecasters’ ability to forecast accurately) and when the bracketing rate is low (Mannes et al., 2014). Some forecasters may provide terrible projections; thus, including them in an average may make the average projections substantially less accurate. Thus, it may be necessary to examine the average of a “select crowd”, by aggregating the projections of the most consistently accurate forecasters (Mannes et al., 2014). Incorporating at least 5–10 forecasters leverages most of the benefits of the crowd; adding additional forecasters tends to result in diminishing returns (Larrick et al., 2024). However, to the extent that those who are most accurate in a given period reflects luck, you are better off averaging the predictions of all forecasters than selecting the forecasters who were most accurate in the most recent period (Larrick et al., 2024).

24.4 Accuracy of Fantasy Football Crowd Projections

Even though the crowd tends to be more accurate than individual forecasters, crowd-averaged projections (at least among experts) are not necessarily highly accurate. In fantasy football, [DESCRIBE THEIR ACCURACY–BOTH FOR ALL PLAYERS AND WHEN SUBSETTING TO THE TOP X PLAYERS]. Nevertheless, individual sources tend to be even less accurate. [DESCRIBE THEIR ACCURACY].

The petersenlab package (Petersen, 2025) has the wisdomOfCrowd() function that computes the overall accuracy of the crowd-averaged projections, including the bracketing rate of the individual projections.

Code
# insert example using historical projections: wisdomOfCrowd()

Below are examples where the predictions a) are centered on the truth, b) bracket the truth, and c) do not bracket the truth. They are illustrated in Figure 24.1.

Three distributions of individual judgments in which judgments bracket and are centered on the truth (top left panel), bracket but are not centered on the truth (top right panel), or do not bracket the truth (i.e., are biased below the truth; bottom panel). The distributions depict the predictions. The dashed lines represent the true value. (Figure and caption adapted from Larrick, R. P., Mannes, A. E., & Soll, J. B. (2024). The social psychology of the wisdom of crowds (with a new section on recent advances). In F. M. Federspiel, G. Montibeller, & M. Seifert (Eds.), *Behavioral decision analysis* (pp. 121–143). Springer. https://doi.org/10.1007/978-3-031-44424-1_7)
Figure 24.1: Three distributions of individual judgments in which judgments bracket and are centered on the truth (top left panel), bracket but are not centered on the truth (top right panel), or do not bracket the truth (i.e., are biased below the truth; bottom panel). The distributions depict the predictions. The dashed lines represent the true value. (Figure and caption adapted from Larrick et al. (2024); Larrick, R. P., Mannes, A. E., & Soll, J. B. (2024). The social psychology of the wisdom of crowds (with a new section on recent advances). In F. M. Federspiel, G. Montibeller, & M. Seifert (Eds.), Behavioral decision analysis (pp. 121–143). Springer. https://doi.org/10.1007/978-3-031-44424-1_7).

Here is a code example of when the individual predictions (from which the crowd-averaged predictions are derived) are centered on the truth for a player. The distribution of projected points versus truth is depicted in Figure 24.2.

Code
predictedValues <- c(10,20,30,40,60,70,80,90)
actualValue <- 50
errors <- predictedValues - actualValue

errors
[1] -40 -30 -20 -10  10  20  30  40
Code
mean(predictedValues)
[1] 50
Code
petersenlab::wisdomOfCrowd(
  predicted = predictedValues,
  actual = actualValue
)
Code
bracketingCentered <- petersenlab::wisdomOfCrowd(
  predicted = predictedValues,
  actual = actualValue
)

bracketingCentered_bracketingRate <- bracketingCentered[which(row.names(bracketingCentered) == "crowdAveraged"), "bracketingRate"] * 100

maeBracketingCentered_individual <- bracketingCentered[which(row.names(bracketingCentered) == "individual"), "MAE"]
maeBracketingCentered_crowd <- bracketingCentered[which(row.names(bracketingCentered) == "crowdAveraged"), "MAE"]

mdaeBracketingCentered_individual <- bracketingCentered[which(row.names(bracketingCentered) == "individual"), "MdAE"]
meBracketingCentered_crowd <- bracketingCentered[which(row.names(bracketingCentered) == "crowdAveraged"), "ME"]

ggplot2::ggplot(,
  mapping = aes(
    x = predictedValues)
) +
  geom_density(
    fill = "lightgray"
  ) +
  geom_vline(
    xintercept = mean(predictedValues),
    color = "darkgray",
    linewidth = 1.5) +
    geom_vline(
    xintercept = actualValue,
    linetype = "dashed",
    linewidth = 1.5
  ) +
  #annotate(
  #  "segment",
  #  x = mean(predictedValues),
  #  xend = actualValue,
  #  y = 0,
  #  yend = 0,
  #  linewidth = 1.5,
  #  arrow = arrow(
  #    angle = 20,
  #    ends = "both",
  #    type = "closed")
  #)
  annotate(
    "text",
    x = mean(predictedValues) + (maeBracketingCentered_crowd / 2) + 1,
    y = 0.0003,
    label = paste("Mean Error: ", meBracketingCentered_crowd, sep = ""),
    hjust = 0 # left-justify
  ) +
  labs(
    x = "Fantasy Points",
    y = "Density",
    title = "Density Plot of Predicted Fantasy Points for a Player",
    subtitle = "Crowd-Averaged Projection (Vertical Gray Line) Vs.\nTruth, i.e., Actual Fantasy Points (Dashed Line)"
  ) +
  theme_classic() +
  theme(axis.title.y = element_text(angle = 0, vjust = 0.5)) # horizontal y-axis title
Example Distribution of Projected Fantasy Points that Brackets and is Centered on the Truth.
Figure 24.2: Example Distribution of Projected Fantasy Points that Brackets and is Centered on the Truth.

As demonstrated above, when the individual predictions are centered on the truth, the bracketing rate is very high (in this case, 57.14%), and the crowd-averaged predictions are perfectly accurate and are more accurate (MAE/MdAE = 0) than the accuracy of the typical individual forecaster (MAE = 25; MdAE = 25) in terms of mean absolute error and median absolute error.

Here is a code example of when the individual predictions bracket but are not centered on the truth for a player, with a high bracketing rate and no strong outliers. The distribution of projected points versus truth is depicted in Figure 24.3.

Code
predictedValues <- c(10,29,29,40,60,70,80,90)
actualValue <- 71
errors <- predictedValues - actualValue

errors
[1] -61 -42 -42 -31 -11  -1   9  19
Code
mean(predictedValues)
[1] 51
Code
petersenlab::wisdomOfCrowd(
  predicted = predictedValues,
  actual = actualValue
)
Code
bracketingNotCentered1 <- petersenlab::wisdomOfCrowd(
  predicted = predictedValues,
  actual = actualValue
)

bracketingNotCentered1_bracketingRate <- bracketingNotCentered1[which(row.names(bracketingNotCentered1) == "crowdAveraged"), "bracketingRate"] * 100

maeBracketingNotCentered1_individual <- bracketingNotCentered1[which(row.names(bracketingNotCentered1) == "individual"), "MAE"]
maeBracketingNotCentered1_crowd <- bracketingNotCentered1[which(row.names(bracketingNotCentered1) == "crowdAveraged"), "MAE"]

mdaeBracketingNotCentered1_individual <- bracketingNotCentered1[which(row.names(bracketingNotCentered1) == "individual"), "MdAE"]
meBracketingNotCentered1_crowd <- bracketingNotCentered1[which(row.names(bracketingNotCentered1) == "crowdAveraged"), "ME"]

ggplot2::ggplot(,
  mapping = aes(
    x = predictedValues)
) +
  geom_density(
    fill = "lightgray"
  ) +
  geom_vline(
    xintercept = mean(predictedValues),
    color = "darkgray",
    linewidth = 1.5) +
    geom_vline(
    xintercept = actualValue,
    linetype = "dashed",
    linewidth = 1.5
  ) +
  annotate(
    "segment",
    x = mean(predictedValues),
    xend = actualValue,
    y = 0,
    yend = 0,
    linewidth = 1.5,
    arrow = arrow(
      angle = 20,
      ends = "both",
      type = "closed")
  ) +
  annotate(
    "text",
    x = actualValue + (meBracketingNotCentered1_crowd / 2),
    y = 0.0007,
    label = paste("Mean Error: ", meBracketingNotCentered1_crowd, sep = ""),
    hjust = 0.5 # center-justify
  ) +
  labs(
    x = "Fantasy Points",
    y = "Density",
    title = "Density Plot of Predicted Fantasy Points for a Player",
    subtitle = "Crowd-Averaged Projection (Vertical Gray Line) Vs.\nTruth, i.e., Actual Fantasy Points (Dashed Line)"
  ) +
  theme_classic() +
  theme(axis.title.y = element_text(angle = 0, vjust = 0.5)) # horizontal y-axis title
Example Distribution of Projected Fantasy Points that Brackets But is Not Centered on the Truth.
Figure 24.3: Example Distribution of Projected Fantasy Points that Brackets But is Not Centered on the Truth.

As demonstrated above, when the individual predictions bracket but are not centered on the truth, and the bracketing rate is high (in this case, 42.86%) with no strong outliers, the crowd-averaged predictions (MAE/MdAE = 20) are more accurate than the accuracy of the typical individual forecaster (MAE = 27; MdAE = 25) in terms of mean absolute error and median absolute error.

Here is a code example of when the individual predictions bracket but are not centered on the truth for a player, with a high bracketing rate but with an outlier projection that is far away from the truth. The distribution of projected points versus truth is depicted in Figure 24.4.

Code
predictedValues <- c(40,60,70,80,90,100,110,250)
actualValue <- 74
errors <- predictedValues - actualValue

errors
[1] -34 -14  -4   6  16  26  36 176
Code
mean(predictedValues)
[1] 100
Code
petersenlab::wisdomOfCrowd(
  predicted = predictedValues,
  actual = actualValue
)
Code
bracketingnotCentered2 <- petersenlab::wisdomOfCrowd(
  predicted = predictedValues,
  actual = actualValue
)

bracketingnotCentered2_bracketingRate <- bracketingnotCentered2[which(row.names(bracketingnotCentered2) == "crowdAveraged"), "bracketingRate"] * 100

maeBracketingnotCentered2_individual <- bracketingnotCentered2[which(row.names(bracketingnotCentered2) == "individual"), "MAE"]
maeBracketingnotCentered2_crowd <- bracketingnotCentered2[which(row.names(bracketingnotCentered2) == "crowdAveraged"), "MAE"]

mdaeBracketingnotCentered2_individual <- bracketingnotCentered2[which(row.names(bracketingnotCentered2) == "individual"), "MdAE"]
meBracketingnotCentered2_crowd <- bracketingnotCentered2[which(row.names(bracketingnotCentered2) == "crowdAveraged"), "ME"]

ggplot2::ggplot(,
  mapping = aes(
    x = predictedValues)
) +
  geom_density(
    fill = "lightgray"
  ) +
  geom_vline(
    xintercept = mean(predictedValues),
    color = "darkgray",
    linewidth = 1.5) +
    geom_vline(
    xintercept = actualValue,
    linetype = "dashed",
    linewidth = 1.5
  ) +
  annotate(
    "segment",
    x = mean(predictedValues),
    xend = actualValue,
    y = 0,
    yend = 0,
    linewidth = 1.5,
    arrow = arrow(
      angle = 20,
      ends = "both",
      type = "closed")
  ) +
  annotate(
    "text",
    x = actualValue + (meBracketingnotCentered2_crowd / 2),
    y = 0.0007,
    label = paste("Mean Error: ", meBracketingnotCentered2_crowd, sep = ""),
    hjust = 0.5 # center-justify
  ) +
  labs(
    x = "Fantasy Points",
    y = "Density",
    title = "Density Plot of Predicted Fantasy Points for a Player",
    subtitle = "Crowd-Averaged Projection (Vertical Gray Line) Vs.\nTruth, i.e., Actual Fantasy Points (Dashed Line)"
  ) +
  theme_classic() +
  theme(axis.title.y = element_text(angle = 0, vjust = 0.5)) # horizontal y-axis title
Example Distribution of Projected Fantasy Points that Brackets But is Not Centered on the Truth.
Figure 24.4: Example Distribution of Projected Fantasy Points that Brackets But is Not Centered on the Truth.

As demonstrated above, when the individual predictions bracket but are not centered on the truth, and the bracketing rate is high (in this case, 53.57%) but with an outlier projection that is far away from the truth, the crowd-averaged predictions (MAE/MdAE = 26) are more accurate than the accuracy of the typical individual forecaster (MAE = 39) in terms of mean absolute error, but they can still be less accurate—in terms of median absolute error—than the individual forecaster who is at the 50th percentile in accuracy (MdAE = 21).

Here is another code example of when the individual predictions bracket but are not centered on the truth for a player, with a low bracketing rate. The distribution of projected points versus truth is depicted in Figure 24.5.

Code
predictedValues <- c(10,29,29,40,60,70,80,90)
actualValue <- 14
errors <- predictedValues - actualValue

errors
[1] -4 15 15 26 46 56 66 76
Code
mean(predictedValues)
[1] 51
Code
petersenlab::wisdomOfCrowd(
  predicted = predictedValues,
  actual = actualValue
)
Code
bracketingnotCentered3 <- petersenlab::wisdomOfCrowd(
  predicted = predictedValues,
  actual = actualValue
)

bracketingnotCentered3_bracketingRate <- bracketingnotCentered3[which(row.names(bracketingnotCentered3) == "crowdAveraged"), "bracketingRate"] * 100

maeBracketingnotCentered3_individual <- bracketingnotCentered3[which(row.names(bracketingnotCentered3) == "individual"), "MAE"]
maeBracketingnotCentered3_crowd <- bracketingnotCentered3[which(row.names(bracketingnotCentered3) == "crowdAveraged"), "MAE"]

mdaeBracketingnotCentered3_individual <- bracketingnotCentered3[which(row.names(bracketingnotCentered3) == "individual"), "MdAE"]
meBracketingnotCentered3_crowd <- bracketingnotCentered3[which(row.names(bracketingnotCentered3) == "crowdAveraged"), "ME"]

ggplot2::ggplot(,
  mapping = aes(
    x = predictedValues)
) +
  geom_density(
    fill = "lightgray"
  ) +
  geom_vline(
    xintercept = mean(predictedValues),
    color = "darkgray",
    linewidth = 1.5) +
    geom_vline(
    xintercept = actualValue,
    linetype = "dashed",
    linewidth = 1.5
  ) +
  annotate(
    "segment",
    x = mean(predictedValues),
    xend = actualValue,
    y = 0,
    yend = 0,
    linewidth = 1.5,
    arrow = arrow(
      angle = 20,
      ends = "both",
      type = "closed")
  ) +
  annotate(
    "text",
    x = actualValue + (meBracketingnotCentered3_crowd / 2),
    y = 0.0003,
    label = paste("Mean Error: ", meBracketingnotCentered3_crowd, sep = ""),
    hjust = 0.5 # center-justify
  ) +
  labs(
    x = "Fantasy Points",
    y = "Density",
    title = "Density Plot of Predicted Fantasy Points for a Player",
    subtitle = "Crowd-Averaged Projection (Vertical Gray Line) Vs.\nTruth, i.e., Actual Fantasy Points (Dashed Line)"
  ) +
  theme_classic() +
  theme(axis.title.y = element_text(angle = 0, vjust = 0.5)) # horizontal y-axis title
Example Distribution of Projected Fantasy Points that Brackets But is Not Centered on the Truth.
Figure 24.5: Example Distribution of Projected Fantasy Points that Brackets But is Not Centered on the Truth.

As demonstrated above, when the individual predictions bracket but are not centered on the truth and the bracketing rate is low (in this case, 25.00%), the crowd-averaged predictions (MAE/MdAE = 37) are more accurate than the accuracy of the typical individual forecaster (MAE = 38) in terms of mean absolute error, but they can still be less accurate—in terms of median absolute error—than the individual forecaster who is at the 50th percentile in accuracy (MdAE = 36).

Here is a code example of when the individual predictions do not bracket the truth for a player, and there are no strong outliers. The distribution of projected points versus truth is depicted in Figure 24.6.

Code
predictedValues <- c(10,20,30,40,60,70,80,90)
actualValue <- 100
errors <- predictedValues - actualValue

errors
[1] -90 -80 -70 -60 -40 -30 -20 -10
Code
mean(predictedValues)
[1] 50
Code
petersenlab::wisdomOfCrowd(
  predicted = predictedValues,
  actual = actualValue
)
Code
noBracketing1 <- petersenlab::wisdomOfCrowd(
  predicted = predictedValues,
  actual = actualValue
)

maeNoBracketing1_individual <- noBracketing1[which(row.names(noBracketing1) == "individual"), "MAE"]
maeNoBracketing1_crowd <- noBracketing1[which(row.names(noBracketing1) == "crowdAveraged"), "MAE"]

mdaeNoBracketing1_individual <- noBracketing1[which(row.names(noBracketing1) == "individual"), "MdAE"]
meNoBracketing1_crowd <- noBracketing1[which(row.names(noBracketing1) == "crowdAveraged"), "ME"]

ggplot2::ggplot(,
  mapping = aes(
    x = predictedValues)
) +
  geom_density(
    fill = "lightgray"
  ) +
  geom_vline(
    xintercept = mean(predictedValues),
    color = "darkgray",
    linewidth = 1.5) +
    geom_vline(
    xintercept = actualValue,
    linetype = "dashed",
    linewidth = 1.5
  ) +
  annotate(
    "segment",
    x = mean(predictedValues),
    xend = actualValue,
    y = 0,
    yend = 0,
    linewidth = 1.5,
    arrow = arrow(
      angle = 20,
      ends = "both",
      type = "closed")
  ) +
  annotate(
    "text",
    x = actualValue + (meNoBracketing1_crowd / 2),
    y = 0.0003,
    label = paste("Mean Error: ", meNoBracketing1_crowd, sep = ""),
    hjust = 0.5 # center-justify
  ) +
  labs(
    x = "Fantasy Points",
    y = "Density",
    title = "Density Plot of Predicted Fantasy Points for a Player",
    subtitle = "Crowd-Averaged Projection (Vertical Gray Line) Vs.\nTruth, i.e., Actual Fantasy Points (Dashed Line)"
  ) +
  theme_classic() +
  theme(axis.title.y = element_text(angle = 0, vjust = 0.5)) # horizontal y-axis title
Example Distribution of Projected Fantasy Points that Does Not Bracket the Truth.
Figure 24.6: Example Distribution of Projected Fantasy Points that Does Not Bracket the Truth.

As demonstrated above, when the individual predictions do not bracket the truth (i.e., the bracketing rate is 0%) and there are no strong outliers, the crowd-averaged predictions (MAE/MdAE = 50) are as accurate as the accuracy of the typical individual forecaster (MAE = 50; MdAE = 50) in terms of mean absolute error and median absolute error.

Here is another code example of when the crowd-averaged predictions do not bracket the truth for a player and there is an outlier prediction that is far away from the truth. The distribution of projected points versus truth is depicted in Figure 24.7.

Code
predictedValues <- c(0,65,67,70,80,85,90,95)
actualValue <- 100
errors <- predictedValues - actualValue

errors
[1] -100  -35  -33  -30  -20  -15  -10   -5
Code
mean(predictedValues)
[1] 69
Code
petersenlab::wisdomOfCrowd(
  predicted = predictedValues,
  actual = actualValue
)
Code
noBracketing2 <- petersenlab::wisdomOfCrowd(
  predicted = predictedValues,
  actual = actualValue
)

maeNoBracketing2_individual <- noBracketing2[which(row.names(noBracketing2) == "individual"), "MAE"]
maeNoBracketing2_crowd <- noBracketing2[which(row.names(noBracketing2) == "crowdAveraged"), "MAE"]

mdaeNoBracketing2_individual <- noBracketing2[which(row.names(noBracketing2) == "individual"), "MdAE"]
meNoBracketing2_crowd <- noBracketing2[which(row.names(noBracketing2) == "crowdAveraged"), "ME"]

ggplot2::ggplot(,
  mapping = aes(
    x = predictedValues)
) +
  geom_density(
    fill = "lightgray"
  ) +
  geom_vline(
    xintercept = mean(predictedValues),
    color = "darkgray",
    linewidth = 1.5) +
    geom_vline(
    xintercept = actualValue,
    linetype = "dashed",
    linewidth = 1.5
  ) +
  annotate(
    "segment",
    x = mean(predictedValues),
    xend = actualValue,
    y = 0,
    yend = 0,
    linewidth = 1.5,
    arrow = arrow(
      angle = 20,
      ends = "both",
      type = "closed")
  ) +
  annotate(
    "text",
    x = actualValue + (meNoBracketing2_crowd / 2),
    y = 0.0015,
    label = paste("Mean Error: ", meNoBracketing2_crowd, sep = ""),
    hjust = 0.5 # center-justify
  ) +
  labs(
    x = "Fantasy Points",
    y = "Density",
    title = "Density Plot of Predicted Fantasy Points for a Player",
    subtitle = "Crowd-Averaged Projection (Vertical Gray Line) Vs.\nTruth, i.e., Actual Fantasy Points (Dashed Line)"
  ) +
  theme_classic() +
  theme(axis.title.y = element_text(angle = 0, vjust = 0.5)) # horizontal y-axis title
Example Distribution of Projected Fantasy Points that Does Not Bracket the Truth.
Figure 24.7: Example Distribution of Projected Fantasy Points that Does Not Bracket the Truth.

As demonstrated above, when the crowd-averaged predictions do not bracket the truth and there are one or more predictions that are far away from the truth, the crowd-averaged predictions (MAE/MdAE = 31) are as accurate as the accuracy of the typical individual forecaster (MAE = 31) in terms of mean absolute error, but they can be less accurate—in terms of median absolute error—than the individual forecaster who is at the 50th percentile in accuracy (MdAE = 25).

In conclusion, as demonstrated from these examples, when the crowd-averaged predictions are centered on the truth, the crowd-averaged predictions are perfectly accurate and are more accurate than the accuracy of the typical individual forecaster in terms of mean absolute error and of most individual forecasters in terms of median absolute error. When the crowd-averaged predictions bracket but are not centered on the truth, the crowd-averaged predictions are more accurate than the accuracy of the typical individual forecaster in terms of mean absolute error. If the bracketing rate is high and there are no strong outliers, the crowd-averaged predictions will also be more accurate—in terms of median absolute error—than most individual forecasters; however, if the bracketing rate is low or there are strong outliers, the crowd-averaged predictions can be less accurate than than most individual forecasters. Outliers could make the crowd-averaged prediction more or less accurate, depending on whether the outlying prediction is closer to or farther away from the truth, compared to the other predictions. When the crowd-averaged predictions do not bracket the truth, the crowd-averaged predictions are as accurate as the accuracy of the typical individual forecaster in terms of mean absolute error. When the crowd-averaged predictions do not bracket the truth and there are outliers, the crowd-averaged projections can be less accurate—in terms of median absolute error—than most individual forecasters (again, depending on whether the outlying prediction is closer to or farther away from the truth, compared to the other predictions).

In sum, going with crowd-averaged projections will always yield predictions that are as or more accurate (in terms of mean absolute error) than the (collective) accuracy of the individuals’ projections. However, the crowd-averaged projections may be less accurate than any individual projection, including the projection of the forecaster who is at the 50th percentile in accuracy. That is, if there is a low bracketing rate or inaccurate outliers, the crowd-averaged projections can be less accurate than most individual projections. In general, the higher the bracketing rate and the more closely that the projections are centered on the truth, the more accurate the crowd-averaged projections will be.

There may be multiple ways of handling outlier predictions. For instance, instead of computing a simple average of the predictions, one could calculate a weighted average that weights each prediction according to the historical accuracy of the prediction source. Alternatively, one could calculate the crowd-averaged prediction using an index of the center of a distribution that is less sensitive to outliers, such as a median or robust average such as the Hodges-Lehmann statistic (aka pseudomedian).

Even though some sources are more accurate than the average in a given year, they are not consistently more accurate than the average. Prediction involves a combination of luck and skill. In some years, a prediction will invariably do better than others, in part, based on luck. However, luck is unlikely to continue systematically into future years, so a source that got lucky in a given year is likely to regress to the mean in subsequent years. That is, determining the most accurate source in a given year, after the fact, is not necessarily the same as identifying the most skilled forecaster. It is easy to identify the most accurate source after the fact, but it is challenging to predict, in advance, who the best forecaster will be (Larrick et al., 2024). It requires a large sample of predictions to determine whether a given forecaster is reliably (i.e., consistently) more accurate than other forecasters and to identify the most accurate forecaster (Larrick et al., 2024). Thus, it can be challenging to know, in advance, who the most accurate forecasters will be. Because average projections are as or more accurate than the average forecaster’s prediction, averaging projections across all forecasters is superior to choosing individual forecasters when the forecasters are roughly similar in forecasting ability or when it is hard to distinguish their ability in advance (Larrick et al., 2024).

The relatively modest accuracy of the projections by so-called fantasy “experts’” and of their average of their projections could occur for a number of reasons. One possibility is that the level of expertise of the “expert” forecasters in terms of being able to provide accurate forecasts is not strong. That is, because football performance and injuries are so challenging to predict, individual forecasters’ projections may not be particularly close to the truth.

A second possibility is that the bracketing rate of the predictions is not particularly high (Mannes et al., 2014). Even if the individual forecasters’ projections are not close to the truth, if ~50% of them overestimate the truth and the other 50% of the underestimate the truth, the average will more closely approximate the truth. However, if all forecasters overestimate the truth for a given player, averaging the projections will not necessarily lead to more accurate projections.

A third possibility is that the forecasts of the different experts are not independent.

Each of these possibilities is likely true to some degree. First, individuals’ predictions are unlikely to be highly accurate consistently. Second, there are many players who are systematically overpredicted (e.g., due to their injury) or underpredicted (e.g., due to their becoming the starter after a teammate becomes injured, is traded, etc.)—an example of overextremity miscalibration. In general, it is likely for players who are projected to score more points to be overpredicted and for players who are projected to score fewer points to be underpredicted [PRESENT CALIBRATION STATS]. Third, the experts may interact and discuss with one another. Interaction and discussion among experts may lead them to follow the herd and conform their projections to what each other predict. This has been terms informational influence and may reflect the anchoring and adjustment heuristic (Larrick et al., 2024). In any case, they are able to see each other’s projections and make change their projections, accordingly.

24.5 How Well Do People Incorporate Advice From Others?

An important question is how well people incorporate advice from others. In the context of fantasy football, advice might be sources of projections. In general, evidence from social psychology suggests that people tend to underweight how much weight they give to advice from others relative to their own opinions, a phenomenon called egocentric discounting (Larrick et al., 2024; Rader et al., 2017). People tend to weight others’ advice around 30% in terms of the proportion of a shift a person makes toward another person’s perspective, though this depends on the perceived accuracy of the advisor. Moreover, people frequently ignores others’ advice entirely. In general, people make use of crowds too little—they put too much weight on their own prediction and not enough weight on others’ predictions whose diversity can be leveraged for error cancellation (Larrick et al., 2024).

24.6 Strategies for Managing Risk and Uncertainty

Sports analyst Tanney (2021) encourages people to consider framing your decisions as bets, to use a probabilistic mindset (rather than a determininistic mindset). Even if something is perceived as likely to occur with 100% certainty, it is not guaranteed that such an event will occur. Other things can and do happen, and overconfidence can get in the way of effective decision-making.

A key goal is to identify decisions that provide a net advantage in expected value. Although there is considerable variability and the outcome is not guaranteed, small advantages can yield considerable gains in the long run Tanney (2021); however, one must be able to survive the costs of losses to obtain enough long-term trials. So, do not bet more than you are willing to lose.

24.7 Risk Management Principles from Cognitive Psychology

24.8 Sports Betting/Gambling

[overconfidence; confirmation bias; variable ratio reinforcement; addiction/delay discounting; wisdom of the crowd, and the challenge of beating “the market”]

The National Football League (NFL) used to be adamantly against sports gambling. They wanted to protect the integrity of the game. They felt that sports gambling might lead to match fixing, point shaving, and other forms of corruption that would undermine the legitimacy of competition in the game. Match fixing occurs when players, referees, coaches, or others conspire to manipulate the outcome of the game. Point shaving involves manipulating the score—commonly by a player intentionally underperforming (even if it does not lead to “throwing the game” and causing the team to lose). Match fixing and point shaving can occur for a variety of reasons, but a common reason is to ensure that particular gamblers win their bet. Another reason the NFL was against sports betting was that, prior to the U.S. Supreme Court overturning the Professional and Amateur Sports Protection Act (PASPA) in 2018, there were legal restrictions on gambling that prevented sports betting in many parts of the country.

However, after the legalization of sports betting following the overturning of PASPA in 2018, the NFL has changed their tune and are now strongly in favor of sports betting. The rise of legal sports betting has created massive revenue streams for the NFL through partnerships, sponsorships, and advertising deals with sportsbooks.

There is substantial overconfidence (in particular overestimation) in sports betting/gambling. A study (archived at https://perma.cc/X2AW-SUBZ) of frequent sports bettors found that they tended to predict that they would gain 0.3 cents for every dollar wagered, but in fact lost 7.5 cents for every dollar wagered (Brown et al., 2025). Overconfidence was greatest among those who frequently wagered multi-leg bets (parlays), losing ~25 cents for every dollar wagered (Brown et al., 2025).

24.8.1 Reinforcement

Reinforcement involves a (typically appetitive) stimulus that increases the frequency of a behavior. For instance, reinforcers could include things like food, praise, attention, money, etc. Thus, sports gambling can provide reinforcement through the receipt of money.

24.8.1.1 Continuous Reinforcement

In general, when you want to train a new behavior, continuous reinforcement is the fastest way to do so. Continuous reinforcement means rewarding the behavior each time that it occurs. For instance, a parent might use continuous reinforcement to train their child to put their toys away—the parent may praise them or give them physical affection each time they put their toys away. However, continuous reinforcement is susceptible to extinction. Extinction means cessation of the target behavior (putting the toys away, in this example). That is, for a previously continusouly reinforced behavior, if the behavior stops being rewarded, the person is less likely to continue the behavior. Thus, to make a behavior less susceptible to extinction, intermittent reinforcement may be used.

24.8.1.2 Intermittent Reinforcement

Intermittent reinforcement (also called partial reinforcement) means rewarding the behavior sometimes but not everytime. There are various approaches to intermittent reinforcement, called schedules of reinforcement. The four primary reinforcement schedules are: fixed interval, fixed ratio, variable interval, adn variable ratio. The reinforcement schedules are depicted in Figure 24.8, as adapted from Spielman et al. (2020).

Reinforcement Schedules. The four reinforcement schedules yield different response patterns. The variable ratio schedule is unpredictable and yields high and steady response rates, with little if any pause after reinforcement (e.g., gambler who plays the slot machines). A fixed ratio schedule is predictable and produces a high response rate, with a short pause after reinforcement (e.g., salesperson paid on commission). The variable interval schedule is unpredictable and produces a moderate, steady response rate (e.g., restaurant manager who is paid a bonus if the food inspector, who comes at unpredictable times, gives the restaurant a good rating). The fixed interval schedule yields a scallop-shaped response pattern, reflecting a significant pause after reinforcement (e.g., surgery patient who can receive a painkiller after pressing a button, but only up to once per hour). (Figure and caption adapted from Spielman, Lovett, & Jenkins (2020); https://openstax.org/books/psychology-2e/pages/6-3-operant-conditioning; Spielman, R. M., Jenkins, W. J., & Lovett, M. D. (2000). *Psychology*. OpenStax. https://openstax.org/details/books/psychology-2e; archived at https://perma.cc/9DZ8-34X2)
Figure 24.8: Reinforcement Schedules. The four reinforcement schedules yield different response patterns. The variable ratio schedule is unpredictable and yields high and steady response rates, with little if any pause after reinforcement (e.g., gambler who plays the slot machines). A fixed ratio schedule is predictable and produces a high response rate, with a short pause after reinforcement (e.g., salesperson paid on commission). The variable interval schedule is unpredictable and produces a moderate, steady response rate (e.g., restaurant manager who is paid a bonus if the food inspector, who comes at unpredictable times, gives the restaurant a good rating). The fixed interval schedule yields a scallop-shaped response pattern, reflecting a significant pause after reinforcement (e.g., surgery patient who can receive a painkiller after pressing a button, but only up to once per hour). (Figure and caption adapted from Spielman et al. (2020); https://openstax.org/books/psychology-2e/pages/6-3-operant-conditioning; Spielman, R. M., Jenkins, W. J., & Lovett, M. D. (2020). Psychology. OpenStax. https://openstax.org/details/books/psychology-2e; archived at https://perma.cc/9DZ8-34X2).

A fixed interval reinforcement schedule occurs when the person is rewarded after a set amount of time. For instance, if the reward becomes available every 30 minutes, the person receives the reward the first time they engage in the behavior after the reward becomes available. As an example, consider a surgery patient who can receive a painkiller after pressing a button, but only up to once per hour.

A variable interval reinforcement schedule occurs when the person is rewarded after a varying (and unpredictable) amount of time. As an example, consider a restaurant manager who is paid a bonus if the food inspector, who comes at unpredictable times, gives the restaurant a good rating.

A fixed ratio reinforcement schedule occurs when the person is rewarded after a set number of responses. As an example, consider a coffee shop that offers a free coffee after every 10 purchases that a person makes.

A variable ratio reinforcement schedule occurs when the person is rewarded after a varying (and unpredictable) number of responses. As an example, consider a slot machine that provides a large monetary reward (paired with the machine lighting up and bells going off) after a varying number of attempts. Many forms of gambling involve variable ratio reinforcement.

The variable ratio reinforcement schedule is the least susceptible to extinction. People can go without reinforcement many times and may still continue to gamble out of the hope for a large reward on the next attempt. The low susceptibility of variable ratio reinforcement to extinction is, in part, why gambling (which often involves variable ratio reinforcement) can be highly addictive despite people’s tendency to lose money gambling.

24.8.2 Is Fantasy Football a Game of Luck or Skill?

The question of whether fantasy football is a game of chance or skill is important because such considerations help determine whether betting on one’s performance is considered gambling, which might be illegal or regulated in many jurisdictions. Fantasy football (and sports betting, more generally) is not 100% luck. Skill can be involved in identifying undervalued assets—whether in terms of stocks in the stock market or professional football players, where inside information can be especially valuable (and thus illegal to profit from). To evaluate the percentage of variability in fantasy football that is attributable to luck versus skill, Getty et al. (2018) evaluated the extent to which one’s performance in the first half of the season was correlated with one’s performance in the second half of the season, under the assumption that underlying skill would lead to persistence of performance across time. The authors examined FanDuel daily fantasy sports (DFS) leagues, which allow the user to set an entirely new lineup each week from all players, based on salary constraints (so a user cannot just select the best player at every position). The authors estimated that ~55% of the variability in fantasy football performance across time was due to skill and 45% was due to luck (Getty et al., 2018). Thus, performance in fantasy football is around half and half luck versus skill.

Because fantasy football is not 100% luck, it is not the same as a slot machine. However, there are still unpredictable elements (e.g., injuries), and there is considerable luck involved. As a result, sports betting and gambling on fantasy football still involve components of variable ratio reinforcement that lend them to being potentially highly addictive.

24.9 Suggestions

Based on the above discussion, here are some suggestions for decision making in the context of uncertainty:

  • Seek advice from diverse perspectives and incorporate it into your decision making.
  • Try to get opinions from others before you state your perspective and before the various sources of advice discuss with each other, to ensure independence of advice.
    • As noted by Kahneman (2011), “before an issue is discussed, all members of the committee should be asked to write a very brief summary of their position… The standard practice of open discussion gives too much weight to the opinions of those who speak early and assertively, causing others to line up behind them.” (p. 85).
  • If some sources of advice (or projections) are clearly more skilled and accurate than others, you can average this “select” crowd of projections or give them greater weight in a weighted average.
  • If it is unclear whether or which sources are reliably more accurate than others, using a simple average across all sources (i.e., crowd-averaged projections) can be a useful approach that is as accurate as—if not more accurate than—the average individual forecaster.
  • Incorporate at least 5–10 sources of projections. Use a weighted or robust average to account for outlier projections.
  • Do not bet on fantasy football (or anything else for that matter) unless you are willing to lose the money. Sports bettors tend to be overconfident; on average, they lose 7.5 cents for every dollar wagered (Brown et al., 2025). If you are going to bet, only bet a small portion of your money and never more than you are willing to lose. And do not make multi-leg bets (parlays); people lose on average 25 cents for every dollar wagered on parlays (Brown et al., 2025).

24.10 Conclusion

24.11 Session Info

Code
sessionInfo()
R version 4.5.0 (2025-04-11)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 24.04.2 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so;  LAPACK version 3.12.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

time zone: UTC
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] lubridate_1.9.4   forcats_1.0.0     stringr_1.5.1     dplyr_1.1.4      
 [5] purrr_1.0.4       readr_2.1.5       tidyr_1.3.1       tibble_3.2.1     
 [9] ggplot2_3.5.2     tidyverse_2.0.0   petersenlab_1.1.5

loaded via a namespace (and not attached):
 [1] gtable_0.3.6       xfun_0.52          htmlwidgets_1.6.4  psych_2.5.3       
 [5] lattice_0.22-6     tzdb_0.5.0         quadprog_1.5-8     vctrs_0.6.5       
 [9] tools_4.5.0        generics_0.1.4     stats4_4.5.0       parallel_4.5.0    
[13] cluster_2.1.8.1    pkgconfig_2.0.3    data.table_1.17.2  checkmate_2.3.2   
[17] RColorBrewer_1.1-3 lifecycle_1.0.4    compiler_4.5.0     farver_2.1.2      
[21] mnormt_2.1.1       mitools_2.4        htmltools_0.5.8.1  yaml_2.3.10       
[25] htmlTable_2.4.3    Formula_1.2-5      pillar_1.10.2      Hmisc_5.2-3       
[29] rpart_4.1.24       nlme_3.1-168       lavaan_0.6-19      tidyselect_1.2.1  
[33] digest_0.6.37      mvtnorm_1.3-3      stringi_1.8.7      reshape2_1.4.4    
[37] labeling_0.4.3     fastmap_1.2.0      grid_4.5.0         colorspace_2.1-1  
[41] cli_3.6.5          magrittr_2.0.3     base64enc_0.1-3    pbivnorm_0.6.0    
[45] foreign_0.8-90     withr_3.0.2        scales_1.4.0       backports_1.5.0   
[49] timechange_0.3.0   rmarkdown_2.29     nnet_7.3-20        gridExtra_2.3     
[53] hms_1.1.3          evaluate_1.0.3     knitr_1.50         mix_1.0-13        
[57] viridisLite_0.4.2  rlang_1.1.6        Rcpp_1.0.14        xtable_1.8-4      
[61] glue_1.8.0         DBI_1.2.3          rstudioapi_0.17.1  jsonlite_2.0.0    
[65] R6_2.6.1           plyr_1.8.9

Feedback

Please consider providing feedback about this textbook, so that I can make it as helpful as possible. You can provide feedback at the following link: https://forms.gle/LsnVKwqmS1VuxWD18

Email Notification

The online version of this book will remain open access. If you want to know when the print version of the book is for sale, enter your email below so I can let you know.