Here we can say, with some confidence that FiveThirtyEight predictions are not reliable probabilities. However, they masquerade as one, being between 0 and 1 and all. This is Taleb’s primary argument; FiveThirtyEight’s predictions do not behave like probabilities that incorporate all uncertainty and should not be passed off as them.
I do not want to suggest that FiveThirtyEight is bad at their craft. They are, likely, the best poll aggregator in the business. If we only look at the last reported probabilistic forecast and use the public’s decision boundary, they are more successful than any other source attempting the same task. However, positioning yourself to appear correct regardless of the outcome, making users infer their own decision boundaries, over-reporting of predictions, and ignoring epistemic uncertainty should not be overlooked. How goes FiveThirtyEight’s reputation, so goes much of the data community’s reputation.
Be clear on your suggested decision boundary, probabilistic statements, assumptions about uncertainty and you’ll be less likely to misguide