Reviewing PECOTA

So, yesterday I didn't post, and to be honest, it just kind of slipped my mind. And then it almost did again today. You would think after doing this for five years, it would sort of be second nature, and for the most part it is, but occasionally, I just kind of forget.

Fortunately, the editor of MNGameday.com, Tom Genrich, provided links to a couple of other awfully well done stories. At the top of the page, he linked to Nick Nelson's solid spring training preview, which I appreciated because I just hadn't taken the time to think about all that stuff. I'd love to give my two cents on a bunch of that stuff, and hope to soon.

And then over at SBG's site, Ubelmann pointed out that Baseball Prospectus' PECOTA standings (subscription required) came out and had the Twins winning 74 games and finishing 4th in the AL Central. Ubelmann thinks that sounds about right. To me, it seems a bit low, but I haven't even done any back-of-the-napkin figuring on that, and we'll need to get to that later. Hey, a that's my second future column idea.

PECOTA, in case you're wondering, stands for Player Empirical Comparison and Optimization Test Algorithm. For those of you who made it through that acronym and are still awake, it is what it says it is - it's a formula for comparing players to previous players based on their performance. The hope is that by looking at how similar players developed, one can predict how current players develop. And then, when you see how close it came to the results, it allows the owner to try and optimize the algorithm for the next year with new data.

You see it referenced more and more, and I'll admit I've become increasingly uncomfortable with the weight it's given. (3rd column idea!) It would seem to be especially difficult to trust it when used for a purpose for which it isn't really intended - predicting the results of teams instead of individual players.

After all, even if it nailed exactly the rates at which each player would perform, the results are still dependent on playing time. And then you need to convert those runs scored and run against to wins and losses. Every layer adds a level of uncertainty that makes the process seem increasingly masturbatory - kinda fun but mostly kind of pathetic. Kinda like blogging.

Anyhoo, in true PECOTA fashion, it seems like there should be a nice empirical test that we can fun to get a sense if we should pay any attention to it, and that is to compare last year's results to what PECOTA predicated. How did it do?

Well, for all my doubts, it didn't do a half bad job. It got three of the four playoff teams in the AL (just missed the Twins) and about three of the four (more or less) in the NL, too. Overall, on the average, it's usually off by a handful of 3.5 to 5 games. It was fairly accurate at predicting the general finish of a team.

On the other hand, it was off by seven games or more for 1/3 of the teams in the league, and seven games is plenty significant when the worst and best team in a league are 30 games apart. The Twins need to hope that PECOTA pegs them as poorly as it did last year - only in the other direction.