Top 100 Prospect Lists are out. How well do they do at ranking?

MLB recently released their 2024 Top 100 Prospects, and that got me to thinking about how accurate the rankings are at predicting future MLB success and how valid the resulting "system" rankings turn out. Given that it takes 7-10 years to know how a draft turns out, we are a long way from knowing what the 2024 Top 100 Prospects will amount to.

However, we can go back 7-10 years and look at those Prospect rankings and begin to see how they turned out. To that end, I did an analysis of the 2017 Fangraphs Top 100 list. Kind of a "where are they now" assessment.

For the 2017 Top 100 Prospects list, we are dealing with players signed/drafted 2016 or earlier, meaning they are generally 26-30 years old now, so we know what they are…mostly. A Top 100 list looks across each team’s entire minor league system, not just last year’s draft class. I’d imagine they look only at each team’s Top 30 prospects, for a pool of 900 players who have not lost their rookie status, of which they select the "Top 100", or the top ~10%.

Something to keep in mind, almost all good players make their MLB debut within 4-5 years of signing. See the histogram below. For all intents and purposes, a 2017 prospect ranking would be looking at players signed 2012-2016. Given a historical average of each draft producing ~35 notable players/year, I’d expect this pool would produce about 100 notables. I know, 35*4 = 170, but many in the 2012-2016 pool will have debuted before 2017 (ex: Alex Bregman).

For the record, 70 of the players in the 2017 group (ranked or not) have gone on to have notable careers, already having accumulated more than 10 WAR (or > 5 WAR if a reliever). Another 30 reside between 5.7 WAR and 10 WAR and some of the 30 will undoubtedly achieve notable status in the next year or three. I call these the potentially notable. For Cardinal fans, Dylan Carlson would be in this list. With that we have the "actual" Top 100 prospects, given 20-20 hindsight to compare with the 2017 Top Prospects list.

So, how did the prospect writers do back in 2017?

Looking at the actuals (remember: hindsight is 20/20):

· 40 of the 70 players who have performed at the notable status in this group were not ranked in the Fangraphs Top 100. Oops.

· 23 of the 30 potential notable players were also not ranked in the Top 100 prospects.

· The net result: Total of 63 out of 100 best players were unranked in 2017, although they were all in the pool to be evaluated.

Some anecdotes on the group of notables. Matt Chapman (31 WAR) and Matt Olson (29 WAR) are ranked #2 and #3 in actual performance but were unranked by the 2017 Top 100 list. They had Juan Soto ranked #95 (FV of 50, average everyday player) and Ronald Acuna ranked at #35.

· Fangraphs ranks by Future Value (FV). They will openly say a 60 FV player (60 FV is considered All-Star caliber) at #7 really isn’t that far from a 60 FV player ranked at #20. If you slice their rankings this way, they ranked fourteen 60 FV players. Seven of these players have accumulated notable WAR. The other seven of them don’t have 10 WAR combined. Only 3 (Willy Adames, Cody Bellinger, Ozzie Albies) have achieved All-Star status. That leaves a 50% hit rate on 60 FV players. Results decline from there.

There appear to be some biases that lead them to a higher miss rate

· Recency - 18 players were signed/drafted during the 2016 season that were ranked in the Top 100 prior to the start of 2017 season. Of the 18, none (yes, zero) have gone on to notable careers. Probably a better way of saying it is that scouts haven’t seen enough performance at higher levels to know (yet) that their FV values are inflated. For Cardinal fans, think Delvin Perez.

· International FA’s - 34 of the 100 players on the top 100 list are IFAs. This should be a red flag all by itself, since only about 15% of notable player come from the international space. Of the 34 IFAs on the Top 100 list, 15 have gone on to notable careers (in line with that 15% historical average). Interestingly, half of the 34 have not accumulated any WAR at all. I wonder if the same issue applies … not enough exposure for evaluators to know (yet) that their FV values are inflated.

· Draft Position/Bonus Hype – It’s hard to overlook that pretty much every single Top 100 ranked prospect was a very high draft choice or a large bonus FA. Granted, the players who get these bonuses have graded out the highest in scouting, which is why they got the big bonus. Scouts don’t come off these grades easily, even as performance fails to match promise. This bias would also tend to favor more positive evaluations for players drafted by poor drafting teams, who tend to draft higher, but ignores that those same teams tend to produce worse results.

· The value of player development - Across multiple Prospect rankings and multiple years (Law, Fangraphs, MLB, etc.) it appears that the evaluators tend to chronically under-value the better development systems (Devil Rays, Cardinals, Astros among them, but not the Dodgers, peculiarly). See more detailed discussion below on organization rankings. It’s like they weight solely on raw talent and don’t attenuate for the system they come into. For example, of the 11 Cardinal farmhands that were in the pool that formed the 2017 Top Prospect ranking, only Sandy Alcantara, Delvin Perez, Carson Kelly, and Alex Reyes got ranked. So, they got 1 out of 4 right. They also missed on Gallen, Bader, Flaherty, Arozarena, Edman, and Dejong, who all made the Top 100 achieved WAR list even while not being ranked. Carlson lurks as an 11th player who may achieve notable status while not being considered top 100. Overall, they missed on 10 of 11 Cardinals. Without going into much detail, it's the exact opposite with a team like the Pirates, not noted for their player development system.

Impact on Organization System Rankings

MLB uses their Top 100 Prospect list to formulate a strawman ranking of overall systems. They describe this as a useful, but "…albeit imprecise" tool for doing so. They assigned points to ranked prospects, inverted from their position. So rank#1 gets 100 points, rank#100 gets 1 point. They add them up by team and formulate an overall system ranking. "…albeit, imprecise". It was this statement that actually got me intrigued enough to do this analysis. Let’s look at what "imprecise" means.

If we look back at the 2017 Top 100 Prospect list, and apply this methodology, they would have ranked the Houston Astros minor league system as 25th overall (as in, 6th worst). This just 18 months after the Astros had one of the best drafts ever (Tucker, Valdez, Bregman, Sandoval, Straw). In the backcast, the Astros should have been ranked #3, not 25th. Remember this when you see system rankings come out soon.

On the flip side, in 2017 the Rangers system was ranked #5 overall. The #5 ranking was on the backs of Jorge Alfaro, Luis Ortiz, Lewis Brinson, Yohander Mendez and Leodys Taveras. You are forgiven if you don’t remember all of these guys. Taveras accounts for 4.6 WAR, the rest are cumulatively negative WAR. In reality, their system produced WAR that should have placed them 26th.

Another example of an organization not noted for player development is the Prates. They rode the likes of Tyler Glasnow, Mitch Keller, Kevin Newman, Josh Bell, ke-Bryan Hayes and Austin Meadows to a #5 system ranking in 2017. How’d that work out? Hayes has the only notable career, although a few others may achieve it down the road. The Dodgers certainly hope Glasnow does. Actual performance of these players to-date would place them 25th on the list.

Overall, the 2017 rankings significantly under-rate the Astros, Cardinals, Rays, Athletics and Orioles. In that group lies some of the best draft-and-develop organizations. I gather they don’t weight this when they assess talent.

Applying some lessons to the 2024 Top 100 prospect list

First and foremost, note the large # of 1st year players in the 2024 list. 23. This is based on an evaluator’s assessment that 2023 was an unusually deep draft. Will have to watch, but I’d suspect recency bias is more the cause than depth of draft. Hard to say that it will be a complete shutout as it was in 2017, but …. For Cardinal fans, Chase Davis’ high ranking spot should be considered through this lens.

Next, note the relatively low# of 60 FV and higher players in the 2024 list. This creates a ranking problem because there are lots of guys who are 55 and 50 FV who get left off because there are only 100 spots. The Cardinals have 7 players with FV of 50 or higher. So, the notion that the Cardinal’s "only" have 3 top 100 prospects should be seen through the lens that once the ranking systems get below 60 FV, their accuracy in ranking is well below 50-50.

While there is not the over-abundance of IFA’s in this list that occurred in 2017, 20 is still 33% over the historical overage of notable players from the IFA space. Many of the IFA players that are so highly rated are in organizations that rarely graduate IFAs to the major leagues, much less notable ones.

One wrinkle that is hard to assess. The 2020 draft was much smaller than normal. How does this impact this class? It’s hard to figure how this impacts rankings and results over the next few years. Based on historical standards, this should be the peak year for 2020 draft kids to enter the MLB scene, but there really aren’t that many 2020 draft kids to begin with. Only 14 appear in the rankings (and, curiously, 3 of them are Cardinals).

Historically, 1 in 6 drafted players make a MLB appearance. Will that ratio hold for the 2020 class, or will this group have a higher percentage? Also, 1 in 6 of those players that appear in the MLB will accrue enough WAR to be "notable". Will this class be an outlier? High or low?

In closing, based on the data assessed, it seems likely that around 7-10 years from now, we will look back on the 2024 Top 100 Prospect Ranking and realize that about 67% of these 2024 Top 100 prospects didn’t amount to much, but someone else not on the list did.