Talent Evaluation in Pro Sports, the Critique Of Ability
Have teams' ability to evaluate talent in the four major U.S. sports improved over the last forty years? I take a look at the last 77,000 draft picks to find out.
Shiny App|GitHub
The critique of teams' ability to draft talented players in professional sports is far from a new endeavor. For as long as the games have existed, pundits and fans alike have nitpicked and needled the draft-day decisions of General Managers and their scouting departments, wondering how they could have chosen whomever they decided upon, while another obviously superior talent remained available to be swiftly scooped up by rival squads.
There’s also no shortage of research in determining best tactics for player evaluation or player selection in any of the four major U.S. sports leagues. And yet, little effort has been made to compare evaluation and selection abilities across sports, over an extended period of time - despite an intriguing influx of cross-sport hirings. These Front Office decisions indicate there may be an overlap, possibly a significant one, in knowledge required to evaluate and draft players across various sports.
Evaluating Talent
In its simplest form, when on the clock, the goal for every team is to pick the best player available, with each pick available, each season. Accomplishing this requires weighing many factors simultaneously, in order to successfully mitigate risk while maximizing opportunity cost: the expected value of each given pick.1 But at the end of that process, only one thing matters for teams and fans alike: Was that player any good?
Bearing all of this in mind, let’s attempt to:
- Research and employ best practices for valuing draft pick outcomes across sports
- Gather the requisite data needed to determine pick outcome values (based on step one)
- Rescale the pick outcome values to allow for cross-sport analysis
- Adjust for different draft lengths (different numbers of rounds for each sport)
- Explore and evaluate the results
Creating a Cross-Sport Analysis
With assistance from previous research in cross-sport analysis, I decided on four pick outcome value metrics (simply labelled value within the app), one for each sport, that would be most appropriate for evaluating the long-term success of each individual player2. This information would be gathered in addition to general pick information, such as Draft Year, Draft Round, Pick Number, and Team:
-
For NBA (Basketball) Players,
Value Over Replacement Player -
For NFL (Football) Players,
Career Approximate Value -
For NHL (Hockey) Players,
Point Share -
For MLB (Baseball) Players,
Wins Above Replacement3
In order to gather the data, I employed Python and the Scrapy package to create four unique scripts (well, five, as two NBA scrapers were needed. Code for all five scrapers is available here.) that gathered Draft Pick and Player Outcome Value information for the past forty years of drafts (1979-2018) via four sites within the SportsReference.com family. In total, I gathered information for nearly 78,000 player picks from:
- Basketball-Reference.com (for NBA data)
- Pro-Football-Reference.com (for NFL data)
- Hockey-Reference.com (for NHL data)
- Baseball-Reference.com (for MLB data)
The next step was rescaling each of the four value metrics (see above) so that the highest rated player in each sport had a value of 1 and the lowest rated player had a value 0. I used minmax scaling for value calculations.4
Finally, I converted each Draft Pick to a Draft Pick Percentile, so that we can (again) more effectively evaluate across sports which have wildly different draft lengths.5
The Data
There’s a ton of information to unpack here. After scraping, cleaning, processing and scaling the data, I combined the information into an easy-to-navigate R-Shiny App to assist myself and others in performing exploratory data analysis.
Multidraft Exlporer
Initial Research Questions
Now that we have the necessary data, we can begin to answer questions about the general state of player evaluation across the four major sports. Some questions this research can begin to answer:
- Are General Managers and Talent Scouts generally good at their job?
- Has the ability to evaluate talent improved over time?
- Are certain sports ahead or behind others in talent evaluation?
- How should franchises best use early-round draft capital?
- Are there particular positions that appear harder or easier to evaluate?
- Are any franchises particularly good or bad at evaluating talent?
Let's dig into the data and see what we discover for each of these questions.
Are General Managers and Talent Scouts generally good at their job?
We'l first take a look at player value compared to the percentile in which the draft pick occurred, grouped by sport. Note actual Pick Numbers corresponding to the Draft Pick Percentiles for each sport are denoted in the subtitle of the plot.
The NHL sees the steepest curve in pick outcome value from through the first 25% of its draft, followed by the NFL, NBA, and MLB. Compared to other sports, MLB (baseball) player value is completely flat from ~70th Percentile Pick to the End of the draft.
Now we can take a look at only the last decade to see if there's any major discrepancies. Not much improvement at first glance:
Another way we can visualize this is by looking at average pick value by draft year, again grouped by sport:
Even taking into account the fact that not all current players who will eventually become superstars have yet to do-so, one could make an argument that, with the exception of the NHL, player outcome values on a per-pick basis are falling, not rising, over the years.
Do things look any better when we assess cumulative values for each draft? Not really.
Are certain sports ahead or behind others in talent evaluation?
Let's zoom-in on the cumulative values, grouped by sport, in only the top 25% of draft picks.
Fascinatingly, NFL teams get roughly the same value from the first 64 picks of the draft than the first 300 MLB picks. That's 10 full rounds of MLB drafting equaling two rounds of NFL drafts in terms of pick value.
To check on this further, let's peruse only the Top-60 picks. As the NBA has just sixty picks per year. For this graph we use the actual pick number as opposed to the draft pick percentile, in the x-axis:
Per these visuals, it's starting to appear that evaluating pro baseball talent might be the most difficult of the four major sports, particularly when attempting to delineate first-rounders from say, fifth-rounders. In general, there's very little separating the outcomes of those picks, despite their multi-round difference in price.
Are particular positions harder or easier to evaluate?
For this analysis, players whose positions performed similar in-game functions were grouped into position categories (i.e. NFL Safeties and Cornerbacks were grouped into a singular Defensive Backs (DB) category) to determine if any of positions had vastly different outcomes than others:
There appears to be some value in picking a Center (C) around the 50th Percentile of drafts, as they have carried more value per-pick than the the 75th Percentile. Still, the best chance of finding a superstar at any position appears to be in the top 25% of drafts.
The Tight-End (TE) position appears to have a much flatter curve than the other positions, indicating that an early round TE pick might carry extra inherent risk.
There appears to be fluctuations in the value of Wings (W) from about the 75th to 50th Percentile, indicating there may be some value in picking that position in the middle of the draft if you miss-out on selecting one at the very top of the draft.
Baseball players show a nearly equal per-pick value after the first 15% of drafts. We can barely see the positional curves, despite this graph already being zoomed-in compared to the other three sports. Let's look at the top 10% just to see if we can find any trends at all:
Let's continue looking at player values, this time in the aggregate, along with the underlying value distributions:
Amongst all the sports, the NHL Defenseman appears to have the most expected value of any position. The NFL Tight End (TE) is on-par with MLB picks (the riskiest picks on a per-pick level) in terms of value.
We can see that certain positions carry significantly more value on average than others when selected in the top-10% of drafts:
NFL Quarterbacks (QB) jump Offensive Lineman (OL) in terms of value in the top-10% of drafts, which begins to explain why many Quarterback-needy NFL teams trade-up to acquire one. In the MLB, the Catcher (C) position differentiates itself as the highest-upside pick early in drafts.
Are any franchises particularly good or bad at evaluating talent?
While one could devote hundreds of thousands of words to fully answering this equation, we can begin to evaluate the differences in teams ability to draft certain positions exploring the shiny app.
Below is a comparison of the NFL's AFC North Division teams' pick outcome values, in the top 25% of drafts, grouped by position:
We can see major differences in how well these teams have drafted Quarterbacks (QB) and Running Backs (RB) early in drafts.
A similar analysis of the NBA's Southeast Division:
The Magic have been excellent at selecting Centers (C) while the Hornets have been more successful than others in selecting Guards (G)
And now the MLB's American League East Division:
Despite razor-thin value differences (notice the x-axis maxes-out at a value of 0.04 out of 1), it appears the Red Sox and Yankees have a noticeable advantage in drafting Infielders (INF) compared to their division rivals.
Conclusion
While significantly more research can be done regarding each of the above mentioned questions, there are two general conclusions I am quite comfortable making:
- Evaluating talent in pro sports is difficult. It hasn't noticeably improved over the last forty years, and just because a player pick is earlier doesn't necessarily mean they're even close to a guarantee to be more valuable.
- The best way to acquire more value than your opponent is to have more picks than your opponent. That means: trade and acquire more total picks.
In addition, exploratory data analysis revealed:
- MLB picks are worth significantly less, per-pick, than the other major sports, but the amount of picks per draft (40 per team) makes up for this difference. The margin for error is razor-thin despite so many chances to find talent each year.
- The Tight End (NFL) and Center (NBA & NHL) are the riskiest early picks.
- Quarterback (NFL), Catcher (MLB), Defensemen (NHL), and Guard (NBA) seem to be the safest early picks.
- It appears that one should wait to select skill-position Players (running back, tight end, and wide receiver) in the NFL.
- NHL and NFL have most variable “hit rates” per year.
Further Research
- I'd love to expand on previous research to create draft curves to determine the expected value of each pick of each round in each sport
- Additionally, it would be rewarding to create a machine learning model to predict future draft pick outcome values
Explore for Yourself
There is much left to discover from this data. Feel free to explore at the app below:
Multidraft Explorer
Footnotes
1. The full list of these factors, again for each sport, would be quite exhaustive, and determining expected values and “draft curves” for different sports may be the subject of future work.
2. Michael Lopez, the Director of Data and Analytics at the NFL and author of the previously linked article, has posted a slew of cross-sport analyses that were highly influential on this article's author.
3. Each of these metrics were selected because they are considered optimal metrics for inter-positional analysis within each sport. For more information on the derivation of each these value metrics, please visit the following links: (NBA) Value Over Replacement Player, (NFL) Approximate Value, (NHL) Point Share, (MLB) Wins Over Replacement
4. Note that in some sports (NBA, NFL, and MLB) a player’s value can dip into the negative if they are measured to be “below replacement,” but players who never play receive a value of 0 prior to rescaling - technically a greater value than the aforementioned below-replacement players. In effect, teams were rewarded for choosing not to play negative-value players in this analysis.
5. There was one additional hurdle: The baseball eligibility problem. MLB players can be (and often are) drafted more than one time. This analysis, as previous research has done, automatically sets all baseball players' pick outcome values to 0, prior to rescaling, if they were drafted again.
Shiny App|GitHub
About Author
