Reliever upside and an intro into how to build a bullpen

Two days ago, the Cardinals hired a new pitching coach in Mike Maddux. When he let Derek Lilliquist go, John Mozeliak shared some of the reasons for why they were looking for something different. According to Derrick Goold, they want to “rethink the strategy of pitching use and have a pitching coach that is open to the data available and some modern views of how pitchers should be deployed.”

That sounds great. Bullpens certainly feature prominently in that statement, and they are becoming more and more important. Teams are more and more hip to the fact that starters often struggle the third time through the order. Here’s total bullpen innings pitched in the majors by year over the last 20 years.

There’s at least a couple of things to read into here. One could observe that bullpen innings have increased every year from 2012 on. Did Tony LaRussa’s aggressive bullpen use in the 2011 playoffs start the reliever revolution? He was already a big part of building the modern bullpen, but was he still influencing bullpen use in his final year as a manager?

Maybe, but maybe not. Compared to prior years, 2012 through 2014 blends in well. The following years may have only been increases because 2011 was so low. Much more apparent is the spike forward from 2015 forward. 2016 set the record, but was topped in 2017. That’s since 1960, not just the last 20 years. 2015 comes in fourth.

In all likelihood, the most recent reliever revolution was ignited by the Royals’ World Series appearance in 2014. An inseparable part of their run was the three-headed monster of Wade Davis, Greg Holland, and Kelvin Herrera, who essentially gave the opposing team just six innings to score per game.

Having three strong pitchers is important, because that’s about how many you need to cover the highest leverage situations that can arise in the late innings, according to leverage index. Leverage Index is essentially a measure of the importance of a situation. A plate appearance that takes place with runners on first and third with one out in the bottom of the ninth of a one run game has a higher likelihood of affecting the result than one with no one on with two outs in the sixth of a three run game.

We can weigh the score, inning, and base-out state and boil it all down to a single number, scaled so that 1.0 is the average leverage, with higher and lower than that representing higher or lower leverage respectively. Well, we don’t have to, because Fangraphs already did, and they feature average leverage index on their leaderboards. Using that, I found the average leverage index of the pitcher who pitched with the highest leverage for each team. I also did this for each team’s second most leveraged reliever, and so on. This was for each team from 2012 to 2016. Here’s the results (originally shared here):

The league average FIP- and xFIP- is 100, that’s just how the stat works. Lower is better, so that a 98 means being 2% better than average. That’s the average reliever xFIP-. Despite as a whole being worse pitchers than starters, relievers maintain better rate stats than starters thanks to only seeing a hitter once, and expending all their effort over a much shorter duration, usually not more than an inning. The average FIP- for relievers over this time frame is about 96, as relievers also are able to avoid homers better than their starting counterparts.

On average, the team’s highest leveraged reliever pitches in situations that are almost twice as important as average. Looking at this year’s most leveraged relievers, sometimes it’s a closer, sometimes a late-inning pre-ninth inning fireman.

On average, a team’s second most leveraged reliever pitches with 50% more leverage than average. A team’s second-best reliever is thus expected to cover a lot of important situations. Same goes for a team’s third most leverage reliever, who is used on average with 28% more than average.

It drops off from there, but a team’s fourth most leveraged reliever is also expected to pitch in higher than average leverage. Fifth sits just below average, and hey, those situations aren’t exactly unimportant.

So generally, a team would like at least one really great reliever, to handle the highest leverage situations, and there’s enough high leverage situations to go around to have three strong relievers. There’s also enough average or higher situations that need to be handled that there’s room for five relievers you can trust. After that, you’re getting into low leverage situations, and the returns are discounted by the fact that they come in less meaningful innings.

So, a team should just collect one bullpen ace, two other really good relievers, and a couple more above averages arms. Oh, and it would be nice if they had an appropriate level of diversity in terms of handedness. Then a team’s set?

Well no, not really. That’s because relief pitchers have volatile results from season to season. Most of that can be explained by the fact that they only pitch 60 innings a year, one-third that of a starter. Hot and cold streaks are less likely to average out over that small of a sample. They’re also pitchers, which means you’re never surprised when one gets hurt, which is pretty often.

Whatever the explanation, it’s something a team has to take into consideration. The very best relievers often don’t stay the very best for very long, save for a few extraordinary arms. Each year there’s multiple ordinary relievers that suddenly become great. Sometimes they stay great, sometimes they don’t. At least that’s what it feels like.

That leads to the rule of thumb of not spending big on relievers. The idea instead, as finely articulated by fellow VEB writer Ben Godar, is to bet on several, less expensive arms instead of one or two pricey ones. The return on investment is most likely better that way. Even the Dodgers, with seemingly unlimited resources, seem to think that. Outside of the heavy investment in Kenly Jansen last winter, they’ve built a fantastic bullpen on the cheap.

I wanted a more concrete answer than that. So what I did was this: I grabbed every qualified reliever season from 2007 to 2016. That was a total 1381 seasons. I ranked them by xFIP-, because it’s more predictive of future FIP- than FIP- itself:

It’s not a huge increase, but xFIP- tells us more, so we’ll use that. Neither tells us all that much, at least over a single qualified reliever season. That speaks the to the volatility of reliever performance.

Anyway, after sorting by xFIP-, I split the seasons as equally as possible into five different groups, or quintiles. If the reliever had a qualified season the following year, I matched it up with that season’s FIP-. After I did that, I calculated some things. Here’s the results:

I found the difference between each pitcher’s xFIP- in year one and their FIP- in the following season, if he recorded a qualified season. I ranked all the players within their group by that difference, and found the top 20% for each group, including those that didn’t post qualifying seasons. I then found the average difference among that 20%, and is represented in “avg change top 20%”. I did the same for 33%, which is in the following column.

The next four columns call back to the first table in this article. 79 was the average FIP- of the average most-leveraged reliever on a team. “79 or under” then refers to the chances that a reliever from that quintile will be at least that good in the following season. The same goes for “85 or under”, which represents being at least as good as the average second most-leverage reliever on a team. That pattern holds until “93 or under”, where I jumped to the fifth most-leveraged reliever. The final column refers to the chances that a member of that group had a qualifying season the following year.

In other words, we’re trying to find out what type of upside different levels of relievers have. Take the top fifth of relievers in any given year, and in the following year they have on average 44.4% chance to pitch as well as the average team’s best reliever. The second fifth is just half as likely.

Still though, even the worst 5th has a 1 out of 20 chance of being a reliever ace the following season. That’s pretty neat. They have nearly 1 out of 10 chance of being as good as the average second most-leveraged reliever. This puts numbers to what we all observe: relievers are coming out of the woodwork all the time.

Perhaps we can drill down closer. We could look at what type of traits are common amon previously ordinary pitchers that suddenly turn into closers and set-up men. That’s something I’d love to do as a part 2. For now, we’ll analyze the Cards’ bullpen options (including impending free agents) with this data.

To start with, let’s look at some recent Cards relievers and their expected average top 20% and 33% performance. To do that, we’ll assign each one to their correct quintile, and then apply the average difference of that group to his 2016 xFIP-. Here’s the results:

OK, we played fast and loose with the innings pitched requirement. That’s why Ryan Sherriff of just 14 ¹⁄₃ innings pitched is at the top of the leaderboard. But hey, they were good innings, and he basically serves as a placeholder for a hypothetical top fifth reliever that the team may try to acquire over the winter. Notice that even among the top 5th of reliever performances, only 75% of them manage to have a qualifying season the following year. A big portion of the variance of relievers comes simplify from the fact that it’s hard to stay on the mound, even if you’re really good. If you sign a great reliever to a four year deal, you can basically expect to get 3 qualified seasons.

Tyler Lyons, Juan Nicasio, and Brett Cecil all placed in the second group. Because they each were so close to each other in xFIP-, it’s easy to consider them all at once. According to the data used here, they each have about a 1 out of 5 chance of being among the best relievers next year. In one sense, that’s what retaining Nicasio means: another 1 out of 5 chances of having a reliever ace.

Sam Tuivailala, Matt Bowman, and Sandy Alcantara all fall in the fourth group. They’re all assigned the same upside, but that’s a limit to this exercise: Tuivailala and Alcantara are both perceived as having higher ceilings (and lower floors) than Bowman. Ignoring that though, hitting just their top 33% projection means a quality reliever for late-inning situations.

We’ll also break it down the Cardinals relievers by the other method, which may be more or less intuitive or useful:

Again, a 79 is the average FIP- of a team’s most-leveraged reliever. Again, Ryan Sherriff shouldn’t really be at the top of this leaderboard, but whatever. Relievers that actually do put up qualified seasons in the top 5th have nearly 45% chance of being as good as a relief ace in the following season. The Cardinals have no one really that good in-house.

They’ll have to settle for the second tier. Cecil and Lyons both have a little better than a 1/5 chance of being at least as good as the average most-leveraged reliever. Signing Nicasio gives them a third shot at it. Tui, Bowman, and Alcantara each break 10% in terms of emerging as a reliever ace.

Taking the seven relievers featured above that are still in the organization right now, and shifting Sherriff to the 4th group with Alcantara and company, the Cards have a collective 63% of producing a qualified season at a 79 FIP- or lower. Adding Nicasio bumps that up to 73%. Add a top fifth reliever instead and it would bump it up to 80%. Both Nicasio and a top 5th reliever puts it at 84%. Nicasio and another reliever from the second quintile would improve the team to 78%.

There’s a lot of info here, and some limitations to all of this. It’s hard to draw any sweeping conclusions, but there’s stuff here to build on. At the least, we have a shorthand way to calculate a reliever’s chances of turning in a somewhat dominating performance, and thus a way a to calculate a bullpen as a whole’s chance of producing one. We have a frame of mind for how to think about building bullpens in terms of reliever upside. Otherwise, let me know what you think about all this.