Comparing Players’ RAPM to their BPM - Assessing Who Gets Advantaged by Box Metrics

[ceiling raiser]

These are all fair points. But I don’t think any of this indicates that looking at who does better or worse than other players in BPM relative to RAPM isn’t informative, at least with regards to the specific questions of (1) whether claims people make on these forums about impact-correlated box metrics like BPM underrating certain players compared to impact are actually directionally right; and (2) whether we think certain player archetypes are underrated or overrated by BPM, such that we might look at BPM for pre-1997 players and bump up or down our mental approximation of what we think that tends to suggest about their impact. Regarding #1, this directly tests that premise, though I grant that scaling being different and BPM not being inherently multi-season makes the analysis not completely precise (certainly I agree that the specific numerical difference between a player’s RAPM and BPM isn’t actually indicating BPM is overrating or underrating impact per 100 possessions by that exact amount). I don’t see good reason why it wouldn’t be generally directionally right about what players are underrated or overrated by BPM compared to RAPM, even if we might not take the specific numerical differences in a player’s RAPM and BPM numbers as being particularly meaningful.

The way BPM was developed was Daniel Myers DSMok1 (a poster on APBRmetrics) ran a regression analysis on a 15-year dataset of RAPM, with minutes, blowout, and home court adjustments (00-01 through 14-15, I believe). Here's the thread on his blog: https://godismyjudgeok.com/DStats/box-plusminus/

There are two issues with this:

1. You are comparing two orthogonal datasets. Even if there are coefficients weighting differently, it is still a statistic based on a dataset without any new information. As such, you are not determining overachievers vs underachievers. Rather, you are evaluating box score vs non box score contributions. This is very different. If your goal is to compare box score vs non box score contributions, that's fine, but this is different.
2. From a validation perspective, you are comparing years in the training set to years outside of the training set. 00-01 through 14-15 comprises a big chunk of the samples there. It's much more useful predicting outside of that set, as one of the core tenets of model validation (accuracy vs reliability vs secondary metrics) is that any analysis performed on the training set is overfit and warped.

Regarding #2, I don’t think there’s really any other way to try to determine what players might be overrated or underrated by pre-1997 BPM than by trying to look at what types of players are overrated or underrated compared to RAPM in post-1997 BPM.

Playing devil's advocate, if we assume the goal is what I mentioned above at the end of my reply to point (1), there are several better ways to determine:

1. Squared2020's partial RAPM from 1985-1996
(1a. On-Off from the Pollack Guides, though these are more limited in size and as such utility)
2. WOWY or WOWYr, which we have with reliability going back to the beginning of the league (or if you care about weighting by possessions, until at least 1973-74, when we first have turnovers...though I actually have team turnovers and I believe offensive rebounds from the Pollack guides for 69-70 through 72-73 somewhere.)

As for comparing “pure RAPM” with “box-score informed RAPM,” that’s another interesting avenue to look at a similar thing. I’d imagine that if the RAPM was being informed by BPM then it’d end up to the benefit of a very similar set of players that seem to benefit from BPM in this analysis, though I take your point about weighted-averages and whatnot, so it’s not guaranteed to just correlate exactly in terms of who it helps the most/least.

One problem with comparing “pure RAPM” and “box-score informed RAPM” is that there’s a whole bunch of other relevant methodological decisions that go into RAPM. For instance, let’s say Player A does better in “pure RAPM” than “box-score informed RAPM” but the “pure RAPM” also has a rubberband adjustment and the “box-score informed RAPM” doesn’t. We wouldn’t really be able to tell which change is causing the difference. Ideally, the best way to test it would be to have “pure RAPM” and “box-score informed RAPM” where the only methodological difference is the existence of a box prior, but I don’t think we actually have any natural experiment like that. If we do, then I’d certainly be interested in seeing it, though!

Suppose you want pure, unadjusted RAPM. You can either:

(a) Calculate your own RAPM. It's not super difficult once you've went through the process once (there are guides on APBRmetrics). The matchupfiles are 95% of the work. I don't like the idea of gatekeeping though, so there is also option...
(b) Request datasets on the APBRmetrics board or ask a creator on Twitter. Most would be happy to oblige.

If you want to make your own adjustments for HCA, or garbage time/blowouts, or FT%/3p%, etc., then yeah, you'll need to edit the matchupfiles, which is very difficult. But is likely unnecessary for this test.

---

As an aside, there are two other things to consider:

1. You excluded Kobe, but some of the players (in particular Jordan) don't have proper 5 year samples. Comparing 2 vs 5 years is noisier.
2. Keep in mind that BPM is measuring *on-court* performance. RAPM is finding the difference between on-court and off-court (roughly speaking), and finding a good fit to the constraints.

I am a big Curry and KG fan, who likely would perform very well in a proper comparison, so I don't take issue with the concept. This just isn't measuring what it intends to measure.

[lessthanjake]

These are all fair points. But I don’t think any of this indicates that looking at who does better or worse than other players in BPM relative to RAPM isn’t informative, at least with regards to the specific questions of (1) whether claims people make on these forums about impact-correlated box metrics like BPM underrating certain players compared to impact are actually directionally right; and (2) whether we think certain player archetypes are underrated or overrated by BPM, such that we might look at BPM for pre-1997 players and bump up or down our mental approximation of what we think that tends to suggest about their impact. Regarding #1, this directly tests that premise, though I grant that scaling being different and BPM not being inherently multi-season makes the analysis not completely precise (certainly I agree that the specific numerical difference between a player’s RAPM and BPM isn’t actually indicating BPM is overrating or underrating impact per 100 possessions by that exact amount). I don’t see good reason why it wouldn’t be generally directionally right about what players are underrated or overrated by BPM compared to RAPM, even if we might not take the specific numerical differences in a player’s RAPM and BPM numbers as being particularly meaningful.

The way BPM was developed was Daniel Myers DSMok1 (a poster on APBRmetrics) ran a regression analysis on a 15-year dataset of RAPM, with minutes, blowout, and home court adjustments (00-01 through 14-15, I believe). Here's the thread on his blog: https://godismyjudgeok.com/DStats/box-plusminus/

There are two issues with this:

1. You are comparing two orthogonal datasets. Even if there are coefficients weighting differently, it is still a statistic based on a dataset without any new information. As such, you are not determining overachievers vs underachievers. Rather, you are evaluating box score vs non box score contributions. This is very different. If your goal is to compare box score vs non box score contributions, that's fine, but this is different.

I’m not really aiming at “compar[ing] box score vs non box score contributions,” to be honest, and unfortunately I don’t know that this analysis can reliably do that. I’m assessing how BPM compares to RAPM for various players. An important distinction here is that, if a player seems to do relatively “better” in RAPM than in BPM compared to other star players, that doesn’t necessarily mean that the player has more “non-box score contributions” than those other star players. That’s certainly a possibility. But BPM isn’t the sole arbiter of what someone’s “box score contributions” are. Players do not do equally well in all box measures, so a comparison between their box-metrics and RAPM won’t be the same across all box-metrics. In other words, since there’s no way to truly say for sure what someone’s “box score contributions” are, there’s no way to really use analysis like this to truly try to isolate out non-box-score contributions.

For that reason, I’m not really being so ambitious as to try to come to that sort of conclusion with this data. I’m more just wanting to assess how BPM compares to RAPM for players. As you mention, BPM is of course designed to correlate with RAPM. This makes it “accurate” in the aggregate, and presumably perform well in the aggregate when tested out of sample. But because it does not correlate perfectly with RAPM, it is objectively going to underrate or overrate some players relative to others. That is, two players with the same RAPM might well have notably different BPMs. And if we assume that long-term RAPM is the underlying truth (which definitely is an assumption that may not be right, to be fair, but it is an assumption underlying this discussion), then it stands to reason that in that particular comparison BPM is underrating one person and overrating the other, at least relative to each other. The creator of BPM alluded to this sort of thing in his write-up of the measure, when noting that the metric doesn’t fully capture Steve Nash’s offense, because his ORAPM way overshoots his OBPM. This is the sort of thing I’m getting at. No matter how “accurate” it is overall, any impact-correlated box measure is definitely going to underrate or overrate some players relative to others. There’s no way around that unless the box measure correlated 100% with RAPM (which is of course impossible). The analysis in this thread seems like a straightforward way of getting a good idea of who is relatively more advantaged by the metric than others (and I imagine the opportunity to do this sort of analysis is precisely why those charts were included in the About Box Plus Minus page).

The other point I’m making here was to push back on people here who have been saying that, as compared to other players, certain players are categorically underrated or overrated by box measures compared to RAPM. If that’s not the case for BPM, then the *categorical* claim must be wrong. That does not mean that the claim couldn’t be true regarding some other box measure. After all, there’s a host of box measures, and even if they all test similarly well against RAPM, they will inevitably often come to materially different results about an individual player—which means we’d generally expect that some would be underrating the player relative to other players and some would be overrating them.

I know I’m talking past you a bit with a lot of this, but figured I’d explicate my thought process here a bit more.

2. From a validation perspective, you are comparing years in the training set to years outside of the training set. 00-01 through 14-15 comprises a big chunk of the samples there. It's much more useful predicting outside of that set, as one of the core tenets of model validation (accuracy vs reliability vs secondary metrics) is that any analysis performed on the training set is overfit and warped.

That’s helpful information to know (I did not know what years were in the training set here). I’m not sure how much it matters here though. I certainly agree that data produced on the training set is probably overfit, and that for purposes of deciding whether BPM is a good measure it’d be critically important for it to perform well out of sample. But I’m not trying to critique BPM as a metric. I’m just talking about its output for specific players. BPM gets reported out for that 2001-2015 time period, so it is part of the stat’s output. And even in those years that are part of the training set, it doesn’t correlate perfectly with RAPM. And since it never correlates perfectly with RAPM, that inherently means that it is overrating or underrating players relative to each other (at least if we take long-term RAPM as the underlying truth).

Regarding #2, I don’t think there’s really any other way to try to determine what players might be overrated or underrated by pre-1997 BPM than by trying to look at what types of players are overrated or underrated compared to RAPM in post-1997 BPM.

Playing devil's advocate, if we assume the goal is what I mentioned above at the end of my reply to point (1), there are several better ways to determine:

1. Squared2020's partial RAPM from 1985-1996
(1a. On-Off from the Pollack Guides, though these are more limited in size and as such utility)
2. WOWY or WOWYr, which we have with reliability going back to the beginning of the league (or if you care about weighting by possessions, until at least 1973-74, when we first have turnovers...though I actually have team turnovers and I believe offensive rebounds from the Pollack guides for 69-70 through 72-73 somewhere.)

These are certainly other ways to get at impact in those pre-1997 years. And I’ve posted extensively about all of those things you mention. But those are all flawed in various ways. Squared’s RAPM is only partial. Pollack’s stuff only includes a few years and just games against the Sixers prior to that, so it doesn’t cover a lot, and it’s also just on-off rather than RAPM. WOWY and WOWYR are nice but we don’t always have good samples, and they can be quite noisy.

So yeah, all of that stuff can help us get an idea of what someone’s impact was prior to 1997. But it doesn’t really give us a very clear picture, which is why people often lean on box metrics a lot for pre-1997 players. If a box metric tends to undersell certain types of players compared to others, then I think it’s helpful to know that, in order to account for it when trying to use that box metric to help triangulate one’s own view/estimate of a pre-1997 player’s impact. I certainly wouldn’t say that that is an airtight methodology. After all, it’s definitely possible that a specific player isn’t underrated by a box metric even if seemingly similar types of players are. But there’s a dearth of information from prior to 1997, so even an informed inference can be at least somewhat helpful IMO.

As for comparing “pure RAPM” with “box-score informed RAPM,” that’s another interesting avenue to look at a similar thing. I’d imagine that if the RAPM was being informed by BPM then it’d end up to the benefit of a very similar set of players that seem to benefit from BPM in this analysis, though I take your point about weighted-averages and whatnot, so it’s not guaranteed to just correlate exactly in terms of who it helps the most/least.

One problem with comparing “pure RAPM” and “box-score informed RAPM” is that there’s a whole bunch of other relevant methodological decisions that go into RAPM. For instance, let’s say Player A does better in “pure RAPM” than “box-score informed RAPM” but the “pure RAPM” also has a rubberband adjustment and the “box-score informed RAPM” doesn’t. We wouldn’t really be able to tell which change is causing the difference. Ideally, the best way to test it would be to have “pure RAPM” and “box-score informed RAPM” where the only methodological difference is the existence of a box prior, but I don’t think we actually have any natural experiment like that. If we do, then I’d certainly be interested in seeing it, though!

Suppose you want pure, unadjusted RAPM. You can either:

(a) Calculate your own RAPM. It's not super difficult once you've went through the process once (there are guides on APBRmetrics). The matchupfiles are 95% of the work. I don't like the idea of gatekeeping though, so there is also option...
(b) Request datasets on the APBRmetrics board or ask a creator on Twitter. Most would be happy to oblige.

If you want to make your own adjustments for HCA, or garbage time/blowouts, or FT%/3p%, etc., then yeah, you'll need to edit the matchupfiles, which is very difficult. But is likely unnecessary for this test.

I’ll consider it, though unfortunately probably isn’t something I personally will have time to do anytime in the near future (and I’ve got a baby on the way any day now, so don’t know that that’ll be changing for a while! ).

As an aside, there are two other things to consider:

1. You excluded Kobe, but some of the players (in particular Jordan) don't have proper 5 year samples. Comparing 2 vs 5 years is noisier.

I didn’t actually exclude Kobe. I just had to exclude him for the second list in my OP because he had a negative RAPM value in one of the time periods, which made it impossible to run a geometric mean for him. I did include him in a subsequent post in which I listed the average of the simple difference between RAPM and BPM. But yeah, I agree that some of the time periods for players that are shorter are noisier and therefore less conclusive for these purposes than they might otherwise be.

2. Keep in mind that BPM is measuring *on-court* performance. RAPM is finding the difference between on-court and off-court (roughly speaking), and finding a good fit to the constraints.

Yeah, but BPM is meant to be measuring on-court box performance as a way of estimating impact. It is intended to get at the same thing as RAPM (i.e. impact per 100 possessions), just using different data.

[homecourtloss]

Spoiler:

These are all fair points. But I don’t think any of this indicates that looking at who does better or worse than other players in BPM relative to RAPM isn’t informative, at least with regards to the specific questions of (1) whether claims people make on these forums about impact-correlated box metrics like BPM underrating certain players compared to impact are actually directionally right; and (2) whether we think certain player archetypes are underrated or overrated by BPM, such that we might look at BPM for pre-1997 players and bump up or down our mental approximation of what we think that tends to suggest about their impact. Regarding #1, this directly tests that premise, though I grant that scaling being different and BPM not being inherently multi-season makes the analysis not completely precise (certainly I agree that the specific numerical difference between a player’s RAPM and BPM isn’t actually indicating BPM is overrating or underrating impact per 100 possessions by that exact amount). I don’t see good reason why it wouldn’t be generally directionally right about what players are underrated or overrated by BPM compared to RAPM, even if we might not take the specific numerical differences in a player’s RAPM and BPM numbers as being particularly meaningful

.

The way BPM was developed was Daniel Myers DSMok1 (a poster on APBRmetrics) ran a regression analysis on a 15-year dataset of RAPM, with minutes, blowout, and home court adjustments (00-01 through 14-15, I believe). Here's the thread on his blog: https://godismyjudgeok.com/DStats/box-plusminus/

There are two issues with this:

1. You are comparing two orthogonal datasets. Even if there are coefficients weighting differently, it is still a statistic based on a dataset without any new information. As such, you are not determining overachievers vs underachievers. Rather, you are evaluating box score vs non box score contributions. This is very different. If your goal is to compare box score vs non box score contributions, that's fine, but this is different.
2. From a validation perspective, you are comparing years in the training set to years outside of the training set. 00-01 through 14-15 comprises a big chunk of the samples there. It's much more useful predicting outside of that set, as one of the core tenets of model validation (accuracy vs reliability vs secondary metrics) is that any analysis performed on the training set is overfit and warped.

Spoiler:

Regarding #2, I don’t think there’s really any other way to try to determine what players might be overrated or underrated by pre-1997 BPM than by trying to look at what types of players are overrated or underrated compared to RAPM in post-1997 BPM.

Playing devil's advocate, if we assume the goal is what I mentioned above at the end of my reply to point (1), there are several better ways to determine:

1. Squared2020's partial RAPM from 1985-1996
(1a. On-Off from the Pollack Guides, though these are more limited in size and as such utility)
2. WOWY or WOWYr, which we have with reliability going back to the beginning of the league (or if you care about weighting by possessions, until at least 1973-74, when we first have turnovers...though I actually have team turnovers and I believe offensive rebounds from the Pollack guides for 69-70 through 72-73 somewhere.)

Spoiler:

As for comparing “pure RAPM” with “box-score informed RAPM,” that’s another interesting avenue to look at a similar thing. I’d imagine that if the RAPM was being informed by BPM then it’d end up to the benefit of a very similar set of players that seem to benefit from BPM in this analysis, though I take your point about weighted-averages and whatnot, so it’s not guaranteed to just correlate exactly in terms of who it helps the most/least.

One problem with comparing “pure RAPM” and “box-score informed RAPM” is that there’s a whole bunch of other relevant methodological decisions that go into RAPM. For instance, let’s say Player A does better in “pure RAPM” than “box-score informed RAPM” but the “pure RAPM” also has a rubberband adjustment and the “box-score informed RAPM” doesn’t. We wouldn’t really be able to tell which change is causing the difference. Ideally, the best way to test it would be to have “pure RAPM” and “box-score informed RAPM” where the only methodological difference is the existence of a box prior, but I don’t think we actually have any natural experiment like that. If we do, then I’d certainly be interested in seeing it, though!

Suppose you want pure, unadjusted RAPM. You can either:

(a) Calculate your own RAPM. It's not super difficult once you've went through the process once (there are guides on APBRmetrics). The matchupfiles are 95% of the work. I don't like the idea of gatekeeping though, so there is also option...
(b) Request datasets on the APBRmetrics board or ask a creator on Twitter. Most would be happy to oblige.

If you want to make your own adjustments for HCA, or garbage time/blowouts, or FT%/3p%, etc., then yeah, you'll need to edit the matchupfiles, which is very difficult. But is likely unnecessary for this test.

As an aside, there are two other things to consider:

1. You excluded Kobe, but some of the players (in particular Jordan) don't have proper 5 year samples. Comparing 2 vs 5 years is noisier.
2. Keep in mind that BPM is measuring *on-court* performance. RAPM is finding the difference between on-court and off-court (roughly speaking), and finding a good fit to the constraints.

I am a big Curry and KG fan, who likely would perform very well in a proper comparison, so I don't take issue with the concept. This just isn't measuring what it intends to measure.

I see what the OP is trying to do, but the OP is confused/unaware of how BPM he’s using and/or different RAPM sets are calculated, and you capture that here.

You are comparing two orthogonal datasets. Even if there are coefficients weighting differently, it is still a statistic based on a dataset without any new information. As such, you are not determining overachievers vs underachievers. Rather, you are evaluating box score vs non box score contributions. This is very different. If your goal is to compare box score vs non box score contributions, that's fine, but this is different.

By the way, DSMok1 posts here or was posting some really interesting things. Hope he can chime in,

Here is every player in the 1997-2023 3 years stint dataset, minimum 5000 minutes in the stint, that had a 3 season stint above +3 in this prior-informed RAPM. This is number of 3 year stints above each RAPM threshold.

This should help contextualize some of these lower-level candidates.

This is sorted by my overall ranking based on this data, which generally translates to a "value over +2" metric.

Code: Select all

1997-2023, 5000 minutes min.: 3 Season RAPM Stints above Threshold Pts/100 Poss.
Name                        +3  +4  +5  +6  +7  +8  +9 +10 +11 +12
LeBron James                19  18  17  16  16  13  10  6   3   1
Tim Duncan                  18  17  16  13  11  10  8   3   1   0
Kevin Garnett               16  15  15  13  10  6   5   4   2   1
Chris Paul                  17  15  14  14  8   5   0   0   0   0
Dirk Nowitzki               15  15  12  12  9   7   0   0   0   0
Stephen Curry               12  10  10  9   6   4   4   1   0   0
Shaquille O'Neal            10  9   8   8   7   7   2   0   0   0
Kevin Durant                11  10  9   8   5   3   0   0   0   0
Kobe Bryant                 13  11  8   5   3   3   1   0   0   0
Dwight Howard               8   8   7   4   3   3   3   1   0   0
James Harden                10  8   7   6   4   0   0   0   0   0
Jason Kidd                  11  9   7   4   2   1   0   0   0   0
Paul George                 11  9   7   4   2   1   0   0   0   0
Steve Nash                  8   8   8   5   3   2   0   0   0   0
Kawhi Leonard               7   7   7   5   4   2   2   0   0   0
Dwyane Wade                 9   8   7   4   3   2   0   0   0   0
Giannis Antetokounmpo       6   5   5   5   5   3   3   0   0   0
Paul Pierce                 14  9   4   3   0   0   0   0   0   0
Rudy Gobert                 7   7   6   4   4   2   0   0   0   0
Nikola Jokic                6   6   6   5   2   2   1   1   0   0
Vince Carter                14  8   3   2   0   0   0   0   0   0
Manu Ginóbili               12  8   5   2   0   0   0   0   0   0
Joel Embiid                 5   5   5   5   4   1   1   0   0   0
Rasheed Wallace             10  8   6   0   0   0   0   0   0   0
Draymond Green              8   6   3   3   3   1   0   0   0   0
Ray Allen                   11  8   4   0   0   0   0   0   0   0
Kyle Lowry                  9   6   3   3   2   0   0   0   0   0
Jimmy Butler                8   7   5   3   0   0   0   0   0   0
Tracy McGrady               8   7   4   2   1   0   0   0   0   0
Russell Westbrook           8   6   4   2   1   0   0   0   0   0
Damian Lillard              8   6   3   3   1   0   0   0   0   0
Paul Millsap                12  7   1   0   0   0   0   0   0   0
Ben Wallace                 7   5   5   3   0   0   0   0   0   0
Metta World Peace           7   6   5   1   1   0   0   0   0   0
LaMarcus Aldridge           11  6   2   0   0   0   0   0   0   0
Anthony Davis               9   6   3   0   0   0   0   0   0   0
Jayson Tatum                5   4   4   2   2   1   0   0   0   0
Baron Davis                 7   6   4   0   0   0   0   0   0   0
Jrue Holiday                8   4   3   1   0   0   0   0   0   0
Dikembe Mutombo             5   5   4   2   0   0   0   0   0   0
Andrei Kirilenko            6   3   3   2   1   0   0   0   0   0
John Stockton               5   5   4   1   0   0   0   0   0   0
Alonzo Mourning             4   3   3   3   1   1   0   0   0   0
Chris Bosh                  7   4   2   1   0   0   0   0   0   0
Gary Payton                 5   4   2   1   1   0   0   0   0   0
Karl-Anthony Towns          7   5   0   0   0   0   0   0   0   0
Andre Iguodala              5   4   2   1   0   0   0   0   0   0
Yao Ming                    6   4   1   0   0   0   0   0   0   0
David Robinson              5   5   1   0   0   0   0   0   0   0
Blake Griffin               7   2   1   0   0   0   0   0   0   0
Pau Gasol                   9   0   0   0   0   0   0   0   0   0
Mike Conley                 7   2   0   0   0   0   0   0   0   0
Shane Battier               5   3   1   0   0   0   0   0   0   0
Chauncey Billups            4   3   2   0   0   0   0   0   0   0
Kemba Walker                4   3   2   0   0   0   0   0   0   0
Pascal Siakam               4   3   2   0   0   0   0   0   0   0
Karl Malone                 3   3   2   1   0   0   0   0   0   0
Allen Iverson               7   1   0   0   0   0   0   0   0   0
Eddie Jones                 6   2   0   0   0   0   0   0   0   0
Marc Gasol                  6   2   0   0   0   0   0   0   0   0
John Wall                   5   2   1   0   0   0   0   0   0   0
Tony Parker                 5   2   1   0   0   0   0   0   0   0
Klay Thompson               4   3   1   0   0   0   0   0   0   0
Nenê                        4   3   1   0   0   0   0   0   0   0
Michael Jordan              3   1   1   1   1   1   0   0   0   0
Kevin Love                  6   1   0   0   0   0   0   0   0   0
Rashard Lewis               5   1   1   0   0   0   0   0   0   0
Bo Outlaw                   5   2   0   0   0   0   0   0   0   0
Chris Webber                5   2   0   0   0   0   0   0   0   0
Brad Miller                 4   2   1   0   0   0   0   0   0   0
DeAndre Jordan              4   2   1   0   0   0   0   0   0   0
Elton Brand                 4   2   1   0   0   0   0   0   0   0
Ricky Rubio                 4   2   1   0   0   0   0   0   0   0
Steve Francis               4   2   1   0   0   0   0   0   0   0
Lamar Odom                  4   3   0   0   0   0   0   0   0   0
Shawn Marion                4   3   0   0   0   0   0   0   0   0
Derrick Rose                2   2   1   1   1   0   0   0   0   0
Vlade Divac                 5   1   0   0   0   0   0   0   0   0
DeMarcus Cousins            3   2   1   0   0   0   0   0   0   0
Joe Johnson                 5   0   0   0   0   0   0   0   0   0
Luol Deng                   5   0   0   0   0   0   0   0   0   0
Danny Green                 4   1   0   0   0   0   0   0   0   0
Jermaine O'Neal             4   1   0   0   0   0   0   0   0   0
Kyrie Irving                4   1   0   0   0   0   0   0   0   0
Al Horford                  3   2   0   0   0   0   0   0   0   0
Amir Johnson                3   2   0   0   0   0   0   0   0   0
Clint Capela                3   2   0   0   0   0   0   0   0   0
Jason Terry                 3   2   0   0   0   0   0   0   0   0
Grant Hill                  2   2   1   0   0   0   0   0   0   0
Glenn Robinson              3   1   0   0   0   0   0   0   0   0
Josh Smith                  3   1   0   0   0   0   0   0   0   0
Luka Doncic                 3   1   0   0   0   0   0   0   0   0
Robert Covington            3   1   0   0   0   0   0   0   0   0
Andrew Bogut                2   1   1   0   0   0   0   0   0   0
Kyle Korver                 2   2   0   0   0   0   0   0   0   0
Carmelo Anthony             3   0   0   0   0   0   0   0   0   0
Eric Gordon                 3   0   0   0   0   0   0   0   0   0
Arvydas Sabonis             2   1   0   0   0   0   0   0   0   0
Bam Adebayo                 2   1   0   0   0   0   0   0   0   0
Brandon Roy                 2   1   0   0   0   0   0   0   0   0
Devin Booker                2   1   0   0   0   0   0   0   0   0
Khris Middleton             2   1   0   0   0   0   0   0   0   0
Patrick Patterson           2   1   0   0   0   0   0   0   0   0
Victor Oladipo              2   1   0   0   0   0   0   0   0   0
Hakeem Olajuwon             1   1   1   0   0   0   0   0   0   0
Mookie Blaylock             1   1   1   0   0   0   0   0   0   0
Tim Hardaway                1   1   1   0   0   0   0   0   0   0
Anderson Varejão            2   0   0   0   0   0   0   0   0   0
Anfernee Hardaway           2   0   0   0   0   0   0   0   0   0
Doug Christie               2   0   0   0   0   0   0   0   0   0
Gerald Wallace              2   0   0   0   0   0   0   0   0   0
Gordon Hayward              2   0   0   0   0   0   0   0   0   0
Jae Crowder                 2   0   0   0   0   0   0   0   0   0
Mikal Bridges               2   0   0   0   0   0   0   0   0   0
Monta Ellis                 2   0   0   0   0   0   0   0   0   0
Nick Collison               2   0   0   0   0   0   0   0   0   0
Nikola Mirotic              2   0   0   0   0   0   0   0   0   0
Peja Stojakovic             2   0   0   0   0   0   0   0   0   0
Serge Ibaka                 2   0   0   0   0   0   0   0   0   0
Steven Adams                2   0   0   0   0   0   0   0   0   0
Tayshaun Prince             2   0   0   0   0   0   0   0   0   0
Tiago Splitter              2   0   0   0   0   0   0   0   0   0
Tyson Chandler              2   0   0   0   0   0   0   0   0   0
Andre Miller                1   1   0   0   0   0   0   0   0   0
Deron Williams              1   1   0   0   0   0   0   0   0   0
Derrick White               1   1   0   0   0   0   0   0   0   0
Devin Harris                1   1   0   0   0   0   0   0   0   0
Gilbert Arenas              1   1   0   0   0   0   0   0   0   0
Joakim Noah                 1   1   0   0   0   0   0   0   0   0
Patrick Ewing               1   1   0   0   0   0   0   0   0   0
Sam Cassell                 1   1   0   0   0   0   0   0   0   0
Scottie Pippen              1   1   0   0   0   0   0   0   0   0
Aaron Gordon                1   0   0   0   0   0   0   0   0   0
Al-Farouq Aminu             1   0   0   0   0   0   0   0   0   0
Antawn Jamison              1   0   0   0   0   0   0   0   0   0
Ben Simmons                 1   0   0   0   0   0   0   0   0   0
Bradley Beal                1   0   0   0   0   0   0   0   0   0
Brevin Knight               1   0   0   0   0   0   0   0   0   0
Brook Lopez                 1   0   0   0   0   0   0   0   0   0
Carlos Boozer               1   0   0   0   0   0   0   0   0   0
Charles Barkley             1   0   0   0   0   0   0   0   0   0
CJ McCollum                 1   0   0   0   0   0   0   0   0   0
Cody Zeller                 1   0   0   0   0   0   0   0   0   0
Dale Davis                  1   0   0   0   0   0   0   0   0   0
Darius Garland              1   0   0   0   0   0   0   0   0   0
David Lee                   1   0   0   0   0   0   0   0   0   0
David West                  1   0   0   0   0   0   0   0   0   0
DeMarre Carroll             1   0   0   0   0   0   0   0   0   0
Derek Fisher                1   0   0   0   0   0   0   0   0   0
Domantas Sabonis            1   0   0   0   0   0   0   0   0   0
Donovan Mitchell            1   0   0   0   0   0   0   0   0   0
Earl Watson                 1   0   0   0   0   0   0   0   0   0
Evan Mobley                 1   0   0   0   0   0   0   0   0   0
Franz Wagner                1   0   0   0   0   0   0   0   0   0
Fred VanVleet               1   0   0   0   0   0   0   0   0   0
Goran Dragic                1   0   0   0   0   0   0   0   0   0
Hedo Türkoglu               1   0   0   0   0   0   0   0   0   0
Iman Shumpert               1   0   0   0   0   0   0   0   0   0
Immanuel Quickley           1   0   0   0   0   0   0   0   0   0
Jamal Murray                1   0   0   0   0   0   0   0   0   0
Jarrett Allen               1   0   0   0   0   0   0   0   0   0
Jeff Foster                 1   0   0   0   0   0   0   0   0   0
Josh Howard                 1   0   0   0   0   0   0   0   0   0
Julius Randle               1   0   0   0   0   0   0   0   0   0
Kristaps Porzingis          1   0   0   0   0   0   0   0   0   0
Lindsey Hunter              1   0   0   0   0   0   0   0   0   0
Marcin Gortat               1   0   0   0   0   0   0   0   0   0
Marcus Camby                1   0   0   0   0   0   0   0   0   0
Mike Miller                 1   0   0   0   0   0   0   0   0   0
Nemanja Bjelica             1   0   0   0   0   0   0   0   0   0
Nikola Vucevic              1   0   0   0   0   0   0   0   0   0
Otto Porter Jr.             1   0   0   0   0   0   0   0   0   0
P.J. Brown                  1   0   0   0   0   0   0   0   0   0
Patrick Beverley            1   0   0   0   0   0   0   0   0   0
Rajon Rondo                 1   0   0   0   0   0   0   0   0   0
Robert Horry                1   0   0   0   0   0   0   0   0   0
Scot Pollard                1   0   0   0   0   0   0   0   0   0
Shai Gilgeous-Alexander     1   0   0   0   0   0   0   0   0   0
Terry Porter                1   0   0   0   0   0   0   0   0   0
Thaddeus Young              1   0   0   0   0   0   0   0   0   0
Trae Young                  1   0   0   0   0   0   0   0   0   0
Vladimir Radmanovic         1   0   0   0   0   0   0   0   0   0
Zach Randolph               1   0   0   0   0   0   0   0   0   0


[lessthanjake]

I see what the OP is trying to do, but the OP is confused/unaware of how BPM he’s using and/or different RAPM sets are calculated, and you capture that here.

I’m certainly aware of how BPM and RAPM sets are calculated. I think one issue here is that the assumptions underlying my analysis might not be being appreciated (which may be a result of me not being entirely clear). BPM was designed to correlate with a particular measure of long-term RAPM. If that RAPM is wrong about a player’s impact, then how BPM compares to RAPM can’t really tell us how a player’s BPM compares to his actual impact. But, as I’ve noted, I’m assuming for purposes of the discussion that the long-term RAPM measure used is actually correct regarding a player’s actual impact. In that case, BPM is just a measure that correlates with actual impact. But it cannot possibly correlate 100%. And if it doesn’t correlate 100%, then essentially definitionally it is overrating some players’ actual impact and underrating other players’ actual impact. Thus, comparing how different players’ BPM compares to their long-term RAPM is a straightforward way of seeing whose BPM overrates or underrates their actual impact compared to other players.

Of course, there’s some complications and caveats here.

- The BPM numbers used are a weighted average of individual years, rather than BPM calculated over a 5-year span. That makes it a little different than the RAPM, which was calculated over the longer timespan. So it’s not *precisely* comparing apples to apples, because a longer-term BPM measure might end up scaling a bit different. But a weighted average of BPM of individual years is surely very similar to what it’d be if BPM was run over a 5-year span. I don’t really see this as a materially significant issue here, though it is one reason to not get too caught up in the specific numbers or small differences.

- While they’re both meant to be estimating impact per 100 possessions, BPM and RAPM are naturally not scaled exactly the same. This certainly gives good reason to not look at the simple difference between the two numbers and conclude that BPM underrates/overrates a player’s impact per 100 possessions by that exact amount. I’m not proposing to use this analysis for that purpose though (and, indeed, this is part of why my OP did not actually analyze based on the simple difference). Rather, I’m comparing how BPM and RAPM compare for different star players. If most star players’ RAPM is a bit above their BPM, then that might not actually tell us that BPM is underrating all of them, but that doesn’t mean we can’t tell which of those players is being underrated or overrated compared to each other. So I don’t see this as a significant issue, as long as the analysis is being used in the way I’m using it (i.e. not putting concrete meaning on the actual numerical values, but rather simply using it to compare the relationship between BPM and RAPM for different players).

- The biggest issue IMO is with the underlying assumption that the long-term RAPM measure in question is always right for every player. As noted above, if the RAPM measure isn’t actually telling us what a player’s actual impact is, then comparing RAPM to BPM doesn’t really tell us how different players’ BPM compares to their actual impact. However, while five-year RAPM is a very good way of measuring actual impact, even five-year RAPM is not perfect and isn’t actually exactly right as to any/every player. And there are various ways of doing it (including lots of different methodological decisions that must be made, with no clear answer as to what’s best) and those different methodologies will come to at least somewhat different results. While the RAPM measure used here was certainly developed by a credible person and likely was done well, there’s really no way of knowing for sure what methodological decisions are “right” (i.e. which ones actually get us closest to actual impact). I made this point early in this thread. If we did this same analysis based on a different RAPM set, the results would likely be similar but at least a little different. And for some players it might actually be significantly different. Basically, RAPM itself isn’t perfect, and itself is merely a regression that will overrate or underrate players compared to the actual objective truth, with the extent of that depending in part on various methodological choices. So that’s a huge caveat here. Therefore, in a sense, this thread is just getting at whether BPM underrates or overrates players as compared to a measure that may itself be overrating or underrating those players as compared to actual impact. I think one could reasonably conclude from that that the analysis isn’t very useful. And it certainly might make sense to sense-check these results with multiple other RAPM sets, to see if it’s clear that this particular RAPM measure’s methodological choices are driving things for particular players or whether things look very similar across multiple RAPM sets. Even that, though, wouldn’t quite get us past the fact that RAPM simply isn’t perfect. All that said, I do think the analysis is at least informative about the underlying question, because I think the long-term RAPM used here probably is at least fairly close to the truth, such that we can use it to get a general directional sense of who BPM is relatively underrating or overrating compared to actual impact. And I think that rejecting that premise would effectively require us to basically reject five-year RAPM as a measure, which I would not be in favor of doing since I think it’s perhaps the best data we have (even if it is still flawed and imperfect).

[lessthanjake]

An interesting related thing I just found:

Before creating BPM, the creator of it did a similar type of comparison as I’ve done here, in order to try to determine who is most underrated or overrated by PER and WS/48 as compared to RAPM. See here:

https://godismyjudgeok.com/DStats/2012/nba-stats/underrated-and-overrated-via-rapm/

http://godismyjudgeok.com/DStats/2012/nba-stats/overrated-and-underrated-via-rapm-part-2/

This is from 2000-2012 (so obviously isn’t based on any data from outside that timeframe), and the Tableau charts don’t work anymore (at least for me), but the write-up does tell us who are the most underrated and overrated players by PER and WS/48 under his analysis. And it doesn’t seem that different in results from what we found here with BPM. For instance, Kevin Durant and old Karl Malone are overrated, and Kevin Garnett is underrated. The write-up also specifically calls out Nash and Ray Allen being very underrated offensively. There’s also some similarities with players that didn’t make the cut to include in the 30 players I used in this post but that I recall from looking at the data. For instance, Rasheed Wallace, Luol Deng, and Metta World Peace are listed as underrated in PER and WS/48, and I did notice them looking significantly underrated in BPM as well. I haven’t checked all the names to see if it’s all consistent (and I’m sure there’s at least some differences), but at first glance it definitely seems to me like underrated/overrated players in the various box score measures are probably pretty similar overall.

[ceiling raiser]

An interesting related thing I just found:

Before creating BPM, the creator of it did a similar type of comparison as I’ve done here, in order to try to determine who is most underrated or overrated by PER and WS/48 as compared to RAPM. See here:

https://godismyjudgeok.com/DStats/2012/nba-stats/underrated-and-overrated-via-rapm/

http://godismyjudgeok.com/DStats/2012/nba-stats/overrated-and-underrated-via-rapm-part-2/

This is from 2012 (so obviously isn’t based on any data from after that), and the Tableau charts don’t work anymore (at least for me), but the write-up does tell us who are the most underrated and overrated players by PER and WS/48 under his analysis. And it doesn’t seem that different in results from what we found here with BPM. For instance, Kevin Durant and old Karl Malone are overrated, and Kevin Garnett is underrated. The write-up also specifically calls out Nash and Ray Allen being very underrated offensively. There’s also some similarities with players that didn’t make the cut to include in the 30 players I used in this post but that I recall from looking at the data. For instance, Rasheed Wallace, Luol Deng, and Metta World Peace are listed as underrated in PER and WS/48, and I did notice them looking significantly underrated in BPM as well. I haven’t checked all the names to see if it’s all consistent (and I’m sure there’s at least some differences), but at first glance it definitely seems to me like underrated/overrated players in the various box score measures are probably pretty similar overall.

I take issue with his word choice, but he is making a different claim -- that certain players are underrated or overrated vis a vis RAPM compared to box metrics, vs that *the impact of* certain players is underrated or overrated vis a vis RAPM compared to box metrics. Former can mean anything. Could mean impact. Could mean scoring. Could mean TV marketability. Etc. More vague of a claim.

[lessthanjake]

An interesting related thing I just found:

Before creating BPM, the creator of it did a similar type of comparison as I’ve done here, in order to try to determine who is most underrated or overrated by PER and WS/48 as compared to RAPM. See here:

https://godismyjudgeok.com/DStats/2012/nba-stats/underrated-and-overrated-via-rapm/

http://godismyjudgeok.com/DStats/2012/nba-stats/overrated-and-underrated-via-rapm-part-2/

This is from 2012 (so obviously isn’t based on any data from after that), and the Tableau charts don’t work anymore (at least for me), but the write-up does tell us who are the most underrated and overrated players by PER and WS/48 under his analysis. And it doesn’t seem that different in results from what we found here with BPM. For instance, Kevin Durant and old Karl Malone are overrated, and Kevin Garnett is underrated. The write-up also specifically calls out Nash and Ray Allen being very underrated offensively. There’s also some similarities with players that didn’t make the cut to include in the 30 players I used in this post but that I recall from looking at the data. For instance, Rasheed Wallace, Luol Deng, and Metta World Peace are listed as underrated in PER and WS/48, and I did notice them looking significantly underrated in BPM as well. I haven’t checked all the names to see if it’s all consistent (and I’m sure there’s at least some differences), but at first glance it definitely seems to me like underrated/overrated players in the various box score measures are probably pretty similar overall.

I take issue with his word choice, but he is making a different claim -- that certain players are underrated or overrated vis a vis RAPM compared to box metrics, vs that *the impact of* certain players is underrated or overrated vis a vis RAPM compared to box metrics. Former can mean anything. Could mean impact. Could mean scoring. Could mean TV marketability. Etc. More vague of a claim.

I think you may be misunderstanding the point I’m making, because I don’t think I’m making a different claim at all. In both cases, the point is to take long-term RAPM as a presumptively accurate assessment of a players’ impact and then compare how players do in a box metric compared to RAPM to determine who the box metric overrates or underrates in terms of impact. I don’t think either analysis is concerned with the non-impact things you’re talking about.