Goldman Sachs Might Be Dumber Than Your Friends
Goldman Sachs’ World Cup predictions were pretty terrible.
On the one hand, they correctly predicted three of the four semi-finalists. On the other hand, that’s about the only measure by which they weren’t completely awful. And it doesn’t take a cadre of geniuses to say, “I think the winningest team in the history of the competition, who also happens to be this year’s hosts, are a good bet to make the semis.”
Pick three of any of the other frontrunners—Germany (three titles and 2010 semi-finalist), Spain (reigning champion), Argentina (playing almost at home and has some guy named Messi), and maybe Italy (four titles) or the Netherlands (the other 2010 finalists)—and you’re a lock to get the final four at least half right.
But Goldman actually bothered to put scores to all 48 of its first round predictions. And just eyeballing it, it doesn’t even look like they built a model. It’s like someone watched a handful of Stoke City games, figured soccer was low-scoring, and decided that almost every World Cup match would end in a 1-1 draw. Because that’s what Goldman pretty much predicted: that most games would end in a 1-1 draw. Then they spiked Brazil with some 4- and 5-goal performances because, ZOMG, Neymar! Or something.
Of the 48 group stage games Goldman predicted 32 of them would end in 1-1 draws. Turned out there were two: Russia’s 1-1 draw with Korea and Russia’s 1-1 draw with Algeria. In fact if not for Fabio Capello’s cautious, low-scoring ways, Goldman would have been beyond terrible. The Russian draws account for two of the three games Goldman correctly predicted.
Three. They got three of 48 correct. That’s 6.25%. If someone were in, say, the wealth management business and they didn’t lose all of their clients’ money only about 6% of the time, they probably wouldn’t last long in that business.
Okay, but getting the exact score is hard. That’s kind of a high bar. What about just the outcome (wether it was a win, loss, or draw)?
A) For fun I built a short script that randomly picked pairs of numbers. More 0s, 1s, and 2s; fewer 3s; a couple of 4s, and 5s. Doing that, I got four games correct (and it would have been about 10 if I half-cheated and flipped the order I listed the teams).
B) I was in a Yahoo World Cup pick-em group with 18 people. Of the top 10 finishers, only one person picked as few as four games correctly and all ten averaged seven correct (and that’s the outcome with the score). And we’re not analyzing data and doing any kind of modeling. We were literally making up scores based on little more than the fact that we’re not slack-jawed troglodytes.
C) Even counting outcomes Goldman was kinda shit.
If you ignore the scores and just look at whether Goldman assigned the correct result—that is whether a team won, lost or drew, without regard to the precise score—then they got 19 of 48. That’s less than 40%. That’s also not good. Back to our pick-em, that would put Goldman with all of their Harvard, MIT, Stanford, etc. brilliance in 15th place. It would probably have been worse but one guy in the league didn’t even finish making all of his picks.
There you go, Goldman beat someone who couldn’t even be bothered to finish clicking buttons on a wesbite. Good job. Good effort.
By almost any measure, Goldman were awful. Total goals? They undershot it by a little over 15% (18 total goals or over six games worth of scoring). Goal difference? They were off by a cumulative 60. And that’s just using the simplistic approach of totaling up the goal-difference totals without accounting for who actually won.
In other words, Goldman predicted Spain to beat Chile 2-1. So Spain should be +1 goal. In the real world Chile won 2-0, making Spain -2. So the total swing is 3 goals. Doing that for all 48 games seemed too much time to spend to go into more excruciating detail of just how pitiful Goldman was (hey, I can be charitable, even if it’s really in the guise of being lazy). Instead, I just totaled up the raw differences.
So yeah, for Brazil, Goldman Sachs was absolutely pitiful at taking data, using it to build a model, then making predictions. But it’s cool, it’s just soccer. Goldman’s real business probably doesn’t depend in any way on them using data to make predictions about what’s going to happen in the future.