Putting UZR to the Test

First off, can I start by saying that chase utley is the second-best position player in the NL? This should not be a surprise, but apparently utley's talent came as a great shock to numerous national broadcasters and sportswriters this week. Last week, I'm not sure you could have gotten 30 percent of them to vote for him as the most valuable position player just on the phillies.

My aim this week is to start a series of posts about defensive metrics. I have a few simple thought experiments to look at their effectiveness. UZR (as the primary tool available to us) is probably not as well understood as it might be. I know that many posters are skeptical of UZR, others want to take it with a HUGE grain of salt, still others probably buy into it as a matter of faith rather than rigorous evaluation of the existing work on it. I confess that I often fall into the latter category. 

In a humble moment I would like say I often learn just enough to understand the very basics of a theory. From that point. I trust that people (ones I know are much smarter than I am) are unlikely to lead me astray. The scientific method it's not.

since I'm not the kind of expert saberologist that has turned his second computer into a SQL machine dedicated to running sabr analysis, the only thing I have to offer is that maybe I can - by working this stuff through for myself - bring along others who are on the same level as I am. FYI - my sabr MO ends at the point of downloading some stats in excel off of fangraphs. I have no idea if anybody has done this or something like it before, so I apologize in advance if it's redundant.

I want to take a couple team stats and compare them today: team E-F and team UZR. UZR you are probably familiar with, even if you don't understand it well. E-F is ERA minus FIP. ERA you know. FIP you probably know too; basically Fielding Independent Pitching is an index of those pitching outcomes totally in the control of the pitcher: walks, strikeouts, and home runs. The index is scaled to the ERA scale.

If FIP is truly "fielding independent," then earned runs minus what is fielding independent should leave you . . . well, fielding. And some luck. 

So, first let's try the old standby. Let's look at the teams with the lowest E-F and their corresponding fielding percentage.

This looks completely random. There's no correlation in any direction, even just by an eyeball test, between E-F and fielding percentage. Out of the best nine teams by E-F, two are in the top ten teams in fielding percentage, three are in the bottom ten, and four are in the middle ten.

Adding UZR to this table, on the other hand, shows something completely different -- the top nine teams in E-F include 6 of the top 7 teams in team UZR (the Rays, 2nd in the majors in UZR, were 14th in E-F). Three outliers - with less than top tier UZR - in the top nine are the Cubs, the Dodgers, and the Cardinals. I'm not the type to give a statistical estimate on the degree of correlation. Still, I'm inclined to eyeball this table and say this is good evidence that UZR means something.

One of the exceptional teams that jumps out though is the home team. The cards have a pretty ordinary team UZR but a good E-F. Could be it's just luck. On the other hand, the cards' staff finished with the best GB percentage in the majors. Consider that UZR is a measure of defensive runs above average.  So, if the balls the cards' staff put in play have a higher likelihood of turning into outs even with an average defender in the field, the staff might yet outperform the FIP without having above average defense. It could also just be luck; the Cubs and Dodgers finished with unimpressive groundball rates. I have no explanation other than luck for the Cubs' and Dodgers' staff E-F numbers. Even for the cards, my explanation is just a suggestion, a hypothesis, rather than an assertion that GB% explains the home team's appearance in the table.

E-F and UZR don't tell the whole story: some pitching skills not reflected in FIP like groundball rate may distort the results, so will dumb luck. But it's remarkable in some ways to combine a conventional run-counting stat like ERA with an advanced stat like FIP derived just from strikeouts, walks, and homers, and find a strong correspondence with a fielding stat that's measured by looking at ball trajectories on the field. The source material for all three stats are very different.

A last note, for anybody who wonders -- I used team UZR and team E-F just because they're extremely easy to work with. Team stats are a very blunt instrument; they will not give you much specific information, but they're good for finding broad trends. Trying to break down individual pitchers' E-Fs with particular defensive configurations behind them, etc. would require several guys much smarter than me and the secret 800 terabyte SABR Sam Adams-proof server farm Bill James is building under the bleachers at Fenway (I predict it will all end in tears when Papelbon sneaks into the computer room, accidentally puts a virus onto the servers when trying to download a solitaire app with pictures of naked women on the cards, and the Red Sox servers become the world's most powerful spambot, crashing internet service up and down the Eastern Seaboard). This little thought experiment has nothing to do with whether UZR properly apportions credit for good defense on the small scale, as between the SS and the 3B on the same team. The point is simply that UZR has something at its core, some real defensive truth that fielding percentage doesn't.