Engelmann Playoff only RAPM (1997-2024)

[parsnips33]

I think a lot of what Doc is getting at is maybe the fundamental limitations of trying to isolate individual impact in a game that is so necessarily dynamic and conjunctive. I'm not sure that the whole sabermetrics movement that was adopted (and adapted) from the world baseball is really such a natural fit in the world of basketball. There are good reasons to try to isolate impact to individuals - if you are in the front office of a team trying to figure out which players to target or how much to offer. However, I'm not sure if it's always, or even broadly, appropriate when trying to expand our knowledge of the game. I'm actually more interested in how Duncan and Ginobili play off of each other and their teammates than exactly how many percentage points of impact should be assigned to each.

[Peregrine01]

The problem with RAPM and on/off data is that it provides virtually no context:
- Who did the guy play with when he was on the floor?
- Who did the guy play against when he was on the floor?
- Was the game competitive when he was on or off the floor or already decided?

And on it goes.

It's a bit ridiculous to criticize Jokic for having a poor on/off in the 2022 playoffs when he was going up against Steph and Draymond and his back-up Boogie was going up against Poole and Bjelic and two out of five games were complete Warrior blowouts before the 3rd quarter ended. Or how bad the Nuggets' on/off numbers look with Jokic/without Murray when the bulk of that sample came when Murray was injured and the Nuggets were a clearly inferior team going up against championship contenders.

[Doctor MJ]

The problem with RAPM and on/off data is that it provides virtually no context:
- Who did the guy play with when he was on the floor?
- Who did the guy play against when he was on the floor?
- Was the game competitive when he was on or off the floor or already decided?

And on it goes.

It's a bit ridiculous to criticize Jokic for having a poor on/off in the 2022 playoffs when he was going up against Steph and Draymond and his back-up Boogie was going up against Poole and Bjelic and two out of five games were complete Warrior blowouts before the 3rd quarter ended. Or how bad the Nuggets' on/off numbers look with Jokic/without Murray when the bulk of that sample came when Murray was injured and the Nuggets were a clearly inferior team going up against championship contenders.

So, I really love your post even if I will push back against it some, as I would argue that on-off & RAPM are specifically ways in which context is applied to give a more meaningful value that pure +/- while still allowing us to represent all these with about the same level of complication.

But of course, you can't understand the player by simply memorizing how impactful a metric says he is. These numbers are about either a starting point, or a middle point, but should not be the end of an analysis that asserts conclusions - and I'd argue the same for any stat. Whether it's 50 PPG or 6 rings or +20 on-off, if that's all ya got, you shouldn't be "done".

I'm glad you gave a specific example with misleading playoff on/off.

In my experience studying what I call OnWins (positive +/-) in losing causes, it's a thing that is really hard to do if you're playing big minutes.

If I do a query of the guys with most these OnWin-losses in the playoffs, here's what I get:

1. Derek Fisher 31
2. Manu Ginobili 23
(tie). James Jones 23
4. Kyle Korver 20
(tie) Kendrick Perkins 20

I should note that Tim Duncan is next on the list (tied with Patty Mills & Davis Bertans), so shout out to Duncan for being the first guy known for really high minutes on the list, but clearly the trend is for guys who played a bit less.

If I put a threshold in there of at least 36 MPG, we get a star-oriented list:

1. Tim Duncan 16
(tie) LeBron James 16
3. Jason Kidd 14
4. Russell Westbrook 12
(tie) James Harden 12

Props to every one of them, but when considering the meaning of the players +/- in a loss, and the expectation of achieving OnWin, consider what fraction of their playoff losses this actually was:

Duncan 16 OnWins in 63 big-minute playoff losses.
LeBron 16 OnWins in 102 big-minute playoff losses.
Kidd 14 OnWins in 63 big-minute playoff losses.
Westbrook 12 OnWins in 47 big-minute playoff losses.
Harden 12 OnWins in 56 big-minute playoff losses.

So in these situations, stars should be expected to have negative +/- when they lose series, and from there's the distinct possibility of rubberband effects to go along with the noise of extremely small off samples, quite easy for the on-off to go negative.

And even if it doesn't, to get a huge RAPM from a negative +/- you basically have to see off numbers low enough that I'd be cautious about taking them seriously in small sample.

This then to say, I really haven't decided how much I should trust a playoff RAPM. I have concerns on top of my questions about method. But I really do appreciate being able to see such data!

[Peregrine01]

The problem with RAPM and on/off data is that it provides virtually no context:
- Who did the guy play with when he was on the floor?
- Who did the guy play against when he was on the floor?
- Was the game competitive when he was on or off the floor or already decided?

And on it goes.

It's a bit ridiculous to criticize Jokic for having a poor on/off in the 2022 playoffs when he was going up against Steph and Draymond and his back-up Boogie was going up against Poole and Bjelic and two out of five games were complete Warrior blowouts before the 3rd quarter ended. Or how bad the Nuggets' on/off numbers look with Jokic/without Murray when the bulk of that sample came when Murray was injured and the Nuggets were a clearly inferior team going up against championship contenders.

So, I really love your post even if I will push back against it some, as I would argue that on-off & RAPM are specifically ways in which context is applied to give a more meaningful value that pure +/- while still allowing us to represent all these with about the same level of complication.

But of course, you can't understand the player by simply memorizing how impactful a metric says he is. These numbers are about either a starting point, or a middle point, but should not be the end of an analysis that asserts conclusions - and I'd argue the same for any stat. Whether it's 50 PPG or 6 rings or +20 on-off, if that's all ya got, you shouldn't be "done".

I'm glad you gave a specific example with misleading playoff on/off.

In my experience studying what I call OnWins (positive +/-) in losing causes, it's a thing that is really hard to do if you're playing big minutes.

If I do a query of the guys with most these OnWin-losses in the playoffs, here's what I get:

1. Derek Fisher 31
2. Manu Ginobili 23
(tie). James Jones 23
4. Kyle Korver 20
(tie) Kendrick Perkins 20

I should note that Tim Duncan is next on the list (tied with Patty Mills & Davis Bertans), so shout out to Duncan for being the first guy known for really high minutes on the list, but clearly the trend is for guys who played a bit less.

If I put a threshold in there of at least 36 MPG, we get a star-oriented list:

1. Tim Duncan 16
(tie) LeBron James 16
3. Jason Kidd 14
4. Russell Westbrook 12
(tie) James Harden 12

Props to every one of them, but when considering the meaning of the players +/- in a loss, and the expectation of achieving OnWin, consider what fraction of their playoff losses this actually was:

Duncan 16 OnWins in 63 big-minute playoff losses.
LeBron 16 OnWins in 102 big-minute playoff losses.
Kidd 14 OnWins in 63 big-minute playoff losses.
Westbrook 12 OnWins in 47 big-minute playoff losses.
Harden 12 OnWins in 56 big-minute playoff losses.

So in these situations, stars should be expected to have negative +/- when they lose series, and from there's the distinct possibility of rubberband effects to go along with the noise of extremely small off samples, quite easy for the on-off to go negative.

And even if it doesn't, to get a huge RAPM from a negative +/- you basically have to see off numbers low enough that I'd be cautious about taking them seriously in small sample.

This then to say, I really haven't decided how much I should trust a playoff RAPM. I have concerns on top of my questions about method. But I really do appreciate being able to see such data!

I really like the way you contextualize the data to tell a story that any layman can understand. I’m not a huge fan of these one number catch-alls (even for large regular season sample sizes). I still think the eye-test (particularly from those who understand the game at a high level) is pretty good at capturing the value of an offensive superstar (which is where most NBA superstars get their value from) and the analytics community deserves some opprobrium for trying to distill everything down to neat little numbers.

[Doctor MJ]

The problem with RAPM and on/off data is that it provides virtually no context:
- Who did the guy play with when he was on the floor?
- Who did the guy play against when he was on the floor?
- Was the game competitive when he was on or off the floor or already decided?

And on it goes.

It's a bit ridiculous to criticize Jokic for having a poor on/off in the 2022 playoffs when he was going up against Steph and Draymond and his back-up Boogie was going up against Poole and Bjelic and two out of five games were complete Warrior blowouts before the 3rd quarter ended. Or how bad the Nuggets' on/off numbers look with Jokic/without Murray when the bulk of that sample came when Murray was injured and the Nuggets were a clearly inferior team going up against championship contenders.

So, I really love your post even if I will push back against it some, as I would argue that on-off & RAPM are specifically ways in which context is applied to give a more meaningful value that pure +/- while still allowing us to represent all these with about the same level of complication.

But of course, you can't understand the player by simply memorizing how impactful a metric says he is. These numbers are about either a starting point, or a middle point, but should not be the end of an analysis that asserts conclusions - and I'd argue the same for any stat. Whether it's 50 PPG or 6 rings or +20 on-off, if that's all ya got, you shouldn't be "done".

I'm glad you gave a specific example with misleading playoff on/off.

In my experience studying what I call OnWins (positive +/-) in losing causes, it's a thing that is really hard to do if you're playing big minutes.

If I do a query of the guys with most these OnWin-losses in the playoffs, here's what I get:

1. Derek Fisher 31
2. Manu Ginobili 23
(tie). James Jones 23
4. Kyle Korver 20
(tie) Kendrick Perkins 20

I should note that Tim Duncan is next on the list (tied with Patty Mills & Davis Bertans), so shout out to Duncan for being the first guy known for really high minutes on the list, but clearly the trend is for guys who played a bit less.

If I put a threshold in there of at least 36 MPG, we get a star-oriented list:

1. Tim Duncan 16
(tie) LeBron James 16
3. Jason Kidd 14
4. Russell Westbrook 12
(tie) James Harden 12

Props to every one of them, but when considering the meaning of the players +/- in a loss, and the expectation of achieving OnWin, consider what fraction of their playoff losses this actually was:

Duncan 16 OnWins in 63 big-minute playoff losses.
LeBron 16 OnWins in 102 big-minute playoff losses.
Kidd 14 OnWins in 63 big-minute playoff losses.
Westbrook 12 OnWins in 47 big-minute playoff losses.
Harden 12 OnWins in 56 big-minute playoff losses.

So in these situations, stars should be expected to have negative +/- when they lose series, and from there's the distinct possibility of rubberband effects to go along with the noise of extremely small off samples, quite easy for the on-off to go negative.

And even if it doesn't, to get a huge RAPM from a negative +/- you basically have to see off numbers low enough that I'd be cautious about taking them seriously in small sample.

This then to say, I really haven't decided how much I should trust a playoff RAPM. I have concerns on top of my questions about method. But I really do appreciate being able to see such data!

I really like the way you contextualize the data to tell a story that any layman can understand. I’m not a huge fan of these one number catch-alls (even for large regular season sample sizes). I still think the eye-test (particularly from those who understand the game at a high level) is pretty good at capturing the value of an offensive superstar (which is where most NBA superstars get their value from) and the analytics community deserves some opprobrium for trying to distill everything down to neat little numbers.

Well I thank you for the kind words Peregrine, even as I'm not necessarily satisfied with my ability to speak to the layman. While that's precisely what I'm trying to do, I think someone better at it wouldn't have the same tendency toward really long posts. I'm an oddball and I know it.

[Djoker]

I don't know why it's so hard for some people to understand that playoff RAPM is extremely noisy. First of all, for most players, the sample is at most 1-2 seasons worth of games for their entire careers. This includes non-prime years too. Then there's the issue of biased samples with overrepresentation of particular opponents and issues with a large proportion of off minutes being garbage time. Single season RAPM typically has such large error bars that the best player is typically not better, statistically speaking than the average NBA player. Given the similar sample size plus these compounding factors, the variances with playoffs RAPM must be absolutely huge.

[DraymondGold]

It's seriously impressive that a Draymond-reliant Warriors could perform well in the first round without Curry.

it's not just impressive, it was necessary. steph has only played 7 healthy playoffs in his career, way less than most any other superstar. without draymond wrecking 1st round teams, steph missed out on a finals in 2016 and a championship in 2018. a lot different resume without that.

Indeed, it was necessary! If they hadn't won in the early rounds without Curry (largely on the back of Draymond's impact), the Warriors don't make the the finals in 2016 and don't win the championship in 2018, although they still win in 2015, 2017, make a finals run in 2019, and win in 2022.

But I don't think this is compelling evidence to think Draymond was more valuable to the playoff Warriors than Curry was over the course of their careers. I think the sample is just being biased here by the easier first-round opponents and the easier schedule that Draymond faced, which is exactly the kind of thing a playoff-only RAPM would miss if it ignores all regular season data.

why isn't it compelling evidence? after all, RAPM either works or it doesn't, right? do i personally believe he was "more" valuable? probably not. though it seems unlikely that there is much difference. but like i said below, i don't weight RAPM as much as others. for those that do, this is exactly what the data is telling us. after all, isn't RAPM supposed to be able to account for opponent strength and adjust for it? so a first round opponent being bludgeoned or a close win over a great team should look similar. we can't just pick the samples where draymond only looks equal instead of better.

Sorry to miss this! It got lost in the shuffle with the oddly acerbic accusations of bias.

This was my point about playoff-only RAPM. If Draymond looked more valuable in full-season RAPM, or playoff-only RAPM with a regular season prior, or full-season RAPM with playoff games heavily weighted, that would be one thing. But this is playoff-only RAPM without any of those corrections... and so the conditions aren't great to appropriately "account for opponent strength and adjust for it" as you say.

Let's take that 2016 first round battle vs Houston (since I suspect you're familiar with that Houston team) as an example. Curry played 2 games in short minutes (1/2 of game 1 which the Warriors won handily, 1/2 of game 4 which the Warriors won handily). The Warriors won the series in 5 games.

How well would a playoff-only RAPM estimate the value of these players, to appropriately correct for the strength of schedule in the Draymond-only minutes? Let's sort the Houston players by minute, and look at their playoff basketball reference BPM (an imperfect stat, but at least somewhat stable in single-series samples, and a good enough first pass for this exercise).

-Harden played the most minutes, and had a very respectable BPM of +8.8. Harden has played 167 playoff games until 2024 (roughly 2 seasons of regular season data), in both conferences and including many multi-series runs so against a reasonable number of opponents at least for playoff RAPM. So his value estimation is caveated by the usual limitations of this exercise ('only' 2 season worth of data; assuming he's the same player from 2010 to 2024 which is obviously false), but otherwise the method shouldn't be too limited in evaluating him.

-Trevor Ariza played the next most minutes. He's played 106 playoff games (~1.3 seasons worth of data) over 15 years, so still a reasonable amount of data to not be noise, but starting to decrease. He's also played against 16 opposing teams in the playoffs -- so not bad, but clearly less than an equivalent length regular season sample. Again, this lack of opponent diversity makes it harder for the RAPM to accurately adjust. His series BPM was -6.6, his 2nd worst playoffs ever, and a far cry from his playoff career average of +1.1.
Do I think playoff-only RAPM accurately estimated the value of Trevor Ariza in the 5-game sample, which is needed to help scale the Curry-less Draymond minutes? It probably does OK, but lack of opponent diversity is starting to limit things, and the small-sample poor performance of Ariza is likely not captured -- so presumably playoff-only RAPM overrates 2016 playoff Trevor Ariza at least slightly.

-Dwight Howard played the next most minutes. He's played 125 playoff games (~1.5 seasons) over 15 years, had a series BPM of -0.4 compared to a career value of +2.9. He's played 15 opposing teams in the playoffs.
Like Trevor Ariza, Dwight has a non-negligible sample size, but has not played as many opponents as we would hope for an RAPM, had a below-average series for his standard, and so there's the possibility that playoff-only RAPM slightly overrates 2016 Dwight Howard.

-Patrick Beverly. 71 playoff games (~0.9 seasons) over over 12 years against 10 opposing teams. Again below average BPM in 2016 -- which no doubt may be correlated with the Warriors winning in the Draymond-run minutes due to Draymond's contributions, but also again may indicate that Beverly had a below average series and may be overrated by lifetime playoff RAPM relative to his career average. Now we only have 10 opposing teams to accurately evaluate the player by...

We'd expect as we go further down the Houston Roster, we'd get to worse players, who thus have fewer minutes and playoff games, and thus the RAPM (in playoff-only samples, where the opponent numbers are limited and the sample size is reduced) would have a harder time evaluating them. For example:
-Donatas Motiejūnas, 6th in Houston minutes in the series, has played 11 playoff games total (an unusably small sample for accurate RAPM) against 3 opposing teams total (he didn't even play against multiple teams like in regular season...)
-Michael Beasley, 7th in Houston minutes in the series, has played 25 playoff games total over 9 years against 7 opponents.

Already we're seeing a trend here. Harden aside, we either have players where there's basically no sample to accurately calculate RAPM, or players had below-average series relative to their career average and would thus likely be overrated by playoff-only RAPM. Perhaps some of their poor performance can be attributed to Draymond. But If those players were overrated, that would overrate the Rockets, thus overrate Draymond in the Draymond-only minutes, making it more likely for him to be wrongly rated over Curry.

And again this is somewhat unique to playoff-only data here.
-The small samples for players here are different from Engelmann's previous full-season and regular-season RAPM, albeit like the historical RAPM we have (though the off samples in this playoff-only RAPM are likely smaller for many players we care about than for the historical RAPM, like jake said above.)
-The small samples are spread out over many years and unevenly sampled making accurate evaluation of the mean value of the player difficult, different from Engelmann's previous full-season and regular season RAPM, especially different from the age-adjusted data Englemann posted, albeit like the historical rapm we have
-However, unlike any other lifetime RAPM (including the historical RAPM we have!!), these playoff games are not randomly selected against different opponents, but are instead limited to the very small sample of opponents the players happened to play in their career... which for many players is just a few first-round opponents before first-round exits, or exclusively opponents in their conference. This makes the RAPM adjustment less accurate!
-And again, unlike the historical RAPM we have, this playoff data does not have uncertainty bars posted (and indeed, the historical RAPM we have includes the team record relative to the full-season data, which allows us to estimate whether the missing data is biased up or down)

Since the Draymond-only and Curry-only minutes are so few, slight changes in the evaluation of players in the Draymond-only minutes can skew things. If the Draymond-only opponents would be slightly overrated in a playoff-only exercise like this (which I've just shown, at least for the 2016 Rockets series), then that would likewise cause Draymond to be overrated in these minutes.

It's not about picking samples where one player looks better. It's about the pros and cons and uncertainties of different metrics. Again, if they had used a regular season prior, or full-season data with postseason games optionally up-weighted and Draymond still came out over Curry, that would be different. But there really are genuine limitations with playoff-only data, that are different from full-season data, different from Squared2020's historical data, just different from any other lifetime RAPM we have.

It's not Engelmann's fault. I'm glad he posted this! There's some genuine broad trends to learn here! Like you said, Harden looks great here -- perhaps reason to reconsider the magnitude of his reputed playoff decline over his career. It's just that there's limitations beyond Engelmann's control. And as I have always argued (regardless of the accusations of a few oddly upset posters), it's important to remember the uncertainties and inherent limitations in any metric before interpreting. And alas, playoff-only data has some limitations that are unique to playoff-only data, and different from full-season data, regular season data, or the historical data we have.

[lessthanjake]

Reading through DraymondGold’s post made me think about some things further. So, to piggyback on some stuff DraymondGold said:

Honestly, amongst other issues, I actually think the biggest problem with playoff-only RAPM is that the actual adjustments for the other people on the floor are often based on very little data. The reason RAPM is better than on-off is because it adjusts for the quality of everyone else on the court. But, in order for that to actually be helpful, the model needs to actually be able to accurately estimate the quality of those other players. And, with playoff-only RAPM, it is inevitably going to be adjusting for a whole lot of players who really haven’t played a lot of playoff games (not to mention that those games are likely against a small number of opponents). This makes those adjustments super noisy. In which case it’s not entirely clear to me that it’s actually better than simply looking at playoff on-off (or rather, playoff on-off while keeping the “on” value in mind, because of diminishing returns). The big flaw of on-off is that it doesn’t adjust for who is on the floor, but a super noisy adjustment isn’t necessarily better than no adjustment.

Of course, if a player simply hasn’t played a lot of playoff games themselves, then the whole thing kind of just fails at an earlier hurdle, because even that player’s raw playoff on-off will be really noisy due to low sample size. So the above is more about players who have played lots of playoff games. Some guys have played enough playoff games that we wouldn’t expect their raw playoff on-off to be *super* noisy. Which means that playoff impact data could potentially be meaningful for that player. The question is just whether the RAPM adjustment will actually make things more accurate for those players. And, for the playoffs, it’s not clear to me that the answer is yes. Maybe it is, but I’m not really sure.

I will note that I think the same issue exists for other RAPM, notably including Squared’s partial RAPM. Squared’s RAPM has a good number of games for most every top player, but there’s a lot of other players with small samples (in many cases because their teams’ games weren’t sampled much). So it’s not entirely clear to me that the RAPM adjustment is better than just looking at on-off for those years. Of course, aside from some specific data on guys like Jordan and Magic, we don’t actually have access to the underlying on-off data that the Squared RAPM is based on. Which naturally means that the RAPM is typically going to be the best piece of data we have, even if it isn’t necessarily better than on-off would be. This is part of why I think looking at the on-off data for guys like Jordan and Magic is particularly interesting. Normally, on-off data is just clearly worse than RAPM, but if RAPM is largely adjusting for players with really small samples, then on-off data isn’t necessarily worse-quality data.

[Doctor MJ]

I don't know why it's so hard for some people to understand that playoff RAPM is extremely noisy. First of all, for most players, the sample is at most 1-2 seasons worth of games for their entire careers. This includes non-prime years too. Then there's the issue of biased samples with overrepresentation of particular opponents and issues with a large proportion of off minutes being garbage time. Single season RAPM typically has such large error bars that the best player is typically not better, statistically speaking than the average NBA player. Given the similar sample size plus these compounding factors, the variances with playoffs RAPM must be absolutely huge.

I think where people find it challenging is that the term "noise" gets used to mean so many things as if they all have the same level of noise.

With Playoff RAPM, the fact that basically no matter how much sample we get, the structure of the playoff makes things different from the regular season, makes it really hard to ever really trust it.

By contrast with something like Squared's '80s RAPMs, the noise concern is something that will largely just disappear as his sample gets more complete.

I won't both sets of data, but I don't quite have the same concerns about them.

[DraymondGold]

Reading through DraymondGold’s post made me think about some things further. So, to piggyback on some stuff DraymondGold said:

Honestly, amongst other issues, I actually think the biggest problem with playoff-only RAPM is that the actual adjustments for the other people on the floor are often based on very little data. The reason RAPM is better than on-off is because it adjusts for the quality of everyone else on the court. But, in order for that to actually be helpful, the model needs to actually be able to accurately estimate the quality of those other players. And, with playoff-only RAPM, it is inevitably going to be adjusting for a whole lot of players who really haven’t played a lot of playoff games (not to mention that those games are likely against a small number of opponents). This makes those adjustments super noisy. In which case it’s not entirely clear to me that it’s actually better than simply looking at playoff on-off (or rather, playoff on-off while keeping the “on” value in mind, because of diminishing returns). The big flaw of on-off is that it doesn’t adjust for who is on the floor, but a super noisy adjustment isn’t necessarily better than no adjustment.

Of course, if a player simply hasn’t played a lot of playoff games themselves, then the whole thing kind of just fails at an earlier hurdle, because even that player’s raw playoff on-off will be really noisy due to low sample size. So the above is more about players who have played lots of playoff games. Some guys have played enough playoff games that we wouldn’t expect their raw playoff on-off to be *super* noisy. Which means that playoff impact data could potentially be meaningful for that player. The question is just whether the RAPM adjustment will actually make things more accurate for those players. And, for the playoffs, it’s not clear to me that the answer is yes. Maybe it is, but I’m not really sure.

I will note that I think the same issue exists for other RAPM, notably including Squared’s partial RAPM. Squared’s RAPM has a good number of games for most every top player, but there’s a lot of other players with small samples (in many cases because their teams’ games weren’t sampled much). So it’s not entirely clear to me that the RAPM adjustment is better than just looking at on-off for those years. Of course, aside from some specific data on guys like Jordan and Magic, we don’t actually have access to the underlying on-off data that the Squared RAPM is based on. Which naturally means that the RAPM is typically going to be the best piece of data we have, even if it isn’t necessarily better than on-off would be. This is part of why I think looking at the on-off data for guys like Jordan and Magic is particularly interesting. Normally, on-off data is just clearly worse than RAPM, but if RAPM is largely adjusting for players with really small samples, then on-off data isn’t necessarily worse-quality data.

Hey jake! A few quick things:

1. Re: on-off while keeping the "on" in mind
The Augmented Plus Minus formula does a fit to several year RAPM, and does a pretty impressive job at getting this roughly right given the noise of the data IMO. In that formula, On-off is about 2x as informative as On in terms of predicting several-year RAPM (in the regular season with complete data where we don't have any of the limitations listed in the prior posts). The exact weighting would differ if we took out the box-score components or got rid of the partial-adjustment of AuPM, but I do think it's a helpful ballpark for the relative usefulness/informativeness of ON-off vs on.... about 2 to 1.

2. Re: Usefulness of historical RAPM vs historical On-off
I guess it’s about the convergence/stability of the two metrics as one goes from a partial sample size to a complete sample-size, in the face of randomly measured samples and non-randomly measured samples. The historical data favors tracking certain teams over others, but is fairly random in the selection of which games to track for those favored teams, while also including games from teams that aren’t favored. The modern playoff data is relatively not randomly measured, given the extreme bias for teams to face other teams only in the same conference, and often face the similar matchups in neighboring years.

I’m genuinely not sure which converges faster — on/off or rapm. It could be on/off, but I’ve never found any discussion on what the uncertainty of on/off is versus sample size, so I’m just not sure. They could also converge at a similar rate, in which case it's not clear on/off would be more useful.

When we have partial sample, I think the key is remembering the uncertainty. The uncertainty range would include the possibility that some opposing player is mis-calibrated, and the effect that would have on every other player (e.g. Jordan and Magic like you say). Since Squared kindly posts the uncertainty ranges, that’s at least one area where the rapm has clear advantage over on/off for the small-sample guys.

And as you and others have said, as we continue to get a larger sample due to the fantastic efforts of Squared, we start to trust the multi-year RAPM more and more.

Personally, I've always been most interested in some sort of rolling several year RAPM over lifetime RAPM. With lifetime RAPM, we start to get concerns that the player is really a different player at the start vs middle vs end of their careers. Over a several year time period, there can be small changes and occasional drastic changes, but most of the time the player is roughly at a similar level of effectiveness. So something like:
-5 year historical RAPM, looking at 85-89, 86-90, 87-91, etc. (taking 5 years just because it's a longer sample when we have partial data, but still short enough that player evolution is minimized), or
-time-decay historical RAPM (with the appropriate decay time set) for each season
This would allow for some sense of both the peak and career curve, unlike lifetime RAPM. That said, all this is basically reliant on the voluntary work of people doing this because it's interesting or fun (which it definitely is!), so I try my best not to be greedy with what I ask for

[iggymcfrack]

It's seriously impressive that a Draymond-reliant Warriors could perform well in the first round without Curry.

it's not just impressive, it was necessary. steph has only played 7 healthy playoffs in his career, way less than most any other superstar. without draymond wrecking 1st round teams, steph missed out on a finals in 2016 and a championship in 2018. a lot different resume without that.

Indeed, it was necessary! If they hadn't won in the early rounds without Curry (largely on the back of Draymond's impact), the Warriors don't make the the finals in 2016 and don't win the championship in 2018, although they still win in 2015, 2017, make a finals run in 2019, and win in 2022.

But I don't think this is compelling evidence to think Draymond was more valuable to the playoff Warriors than Curry was over the course of their careers. I think the sample is just being biased here by the easier first-round opponents and the easier schedule that Draymond faced, which is exactly the kind of thing a playoff-only RAPM would miss if it ignores all regular season data.

why isn't it compelling evidence? after all, RAPM either works or it doesn't, right? do i personally believe he was "more" valuable? probably not. though it seems unlikely that there is much difference. but like i said below, i don't weight RAPM as much as others. for those that do, this is exactly what the data is telling us. after all, isn't RAPM supposed to be able to account for opponent strength and adjust for it? so a first round opponent being bludgeoned or a close win over a great team should look similar. we can't just pick the samples where draymond only looks equal instead of better.

Sorry to miss this! It got lost in the shuffle with the oddly acerbic accusations of bias.

This was my point about playoff-only RAPM. If Draymond looked more valuable in full-season RAPM, or playoff-only RAPM with a regular season prior, or full-season RAPM with playoff games heavily weighted, that would be one thing. But this is playoff-only RAPM without any of those corrections... and so the conditions aren't great to appropriately "account for opponent strength and adjust for it" as you say.

Let's take that 2016 first round battle vs Houston (since I suspect you're familiar with that Houston team) as an example. Curry played 2 games in short minutes (1/2 of game 1 which the Warriors won handily, 1/2 of game 4 which the Warriors won handily). The Warriors won the series in 5 games.

How well would a playoff-only RAPM estimate the value of these players, to appropriately correct for the strength of schedule in the Draymond-only minutes? Let's sort the Houston players by minute, and look at their playoff basketball reference BPM (an imperfect stat, but at least somewhat stable in single-series samples, and a good enough first pass for this exercise).

-Harden played the most minutes, and had a very respectable BPM of +8.8. Harden has played 167 playoff games until 2024 (roughly 2 seasons of regular season data), in both conferences and including many multi-series runs so against a reasonable number of opponents at least for playoff RAPM. So his value estimation is caveated by the usual limitations of this exercise ('only' 2 season worth of data; assuming he's the same player from 2010 to 2024 which is obviously false), but otherwise the method shouldn't be too limited in evaluating him.

-Trevor Ariza played the next most minutes. He's played 106 playoff games (~1.3 seasons worth of data) over 15 years, so still a reasonable amount of data to not be noise, but starting to decrease. He's also played against 16 opposing teams in the playoffs -- so not bad, but clearly less than an equivalent length regular season sample. Again, this lack of opponent diversity makes it harder for the RAPM to accurately adjust. His series BPM was -6.6, his 2nd worst playoffs ever, and a far cry from his playoff career average of +1.1.
Do I think playoff-only RAPM accurately estimated the value of Trevor Ariza in the 5-game sample, which is needed to help scale the Curry-less Draymond minutes? It probably does OK, but lack of opponent diversity is starting to limit things, and the small-sample poor performance of Ariza is likely not captured -- so presumably playoff-only RAPM overrates 2016 playoff Trevor Ariza at least slightly.

-Dwight Howard played the next most minutes. He's played 125 playoff games (~1.5 seasons) over 15 years, had a series BPM of -0.4 compared to a career value of +2.9. He's played 15 opposing teams in the playoffs.
Like Trevor Ariza, Dwight has a non-negligible sample size, but has not played as many opponents as we would hope for an RAPM, had a below-average series for his standard, and so there's the possibility that playoff-only RAPM slightly overrates 2016 Dwight Howard.

-Patrick Beverly. 71 playoff games (~0.9 seasons) over over 12 years against 10 opposing teams. Again below average BPM in 2016 -- which no doubt may be correlated with the Warriors winning in the Draymond-run minutes due to Draymond's contributions, but also again may indicate that Beverly had a below average series and may be overrated by lifetime playoff RAPM relative to his career average. Now we only have 10 opposing teams to accurately evaluate the player by...

We'd expect as we go further down the Houston Roster, we'd get to worse players, who thus have fewer minutes and playoff games, and thus the RAPM (in playoff-only samples, where the opponent numbers are limited and the sample size is reduced) would have a harder time evaluating them. For example:
-Donatas Motiejūnas, 6th in Houston minutes in the series, has played 11 playoff games total (an unusably small sample for accurate RAPM) against 3 opposing teams total (he didn't even play against multiple teams like in regular season...)
-Michael Beasley, 7th in Houston minutes in the series, has played 25 playoff games total over 9 years against 7 opponents.

Already we're seeing a trend here. Harden aside, we either have players where there's basically no sample to accurately calculate RAPM, or players had below-average series relative to their career average and would thus likely be overrated by playoff-only RAPM. Perhaps some of their poor performance can be attributed to Draymond. But If those players were overrated, that would overrate the Rockets, thus overrate Draymond in the Draymond-only minutes, making it more likely for him to be wrongly rated over Curry.

And again this is somewhat unique to playoff-only data here.
-The small samples for players here are different from Engelmann's previous full-season and regular-season RAPM, albeit like the historical RAPM we have (though the off samples in this playoff-only RAPM are likely smaller for many players we care about than for the historical RAPM, like jake said above.)
-The small samples are spread out over many years and unevenly sampled making accurate evaluation of the mean value of the player difficult, different from Engelmann's previous full-season and regular season RAPM, especially different from the age-adjusted data Englemann posted, albeit like the historical rapm we have
-However, unlike any other lifetime RAPM (including the historical RAPM we have!!), these playoff games are not randomly selected against different opponents, but are instead limited to the very small sample of opponents the players happened to play in their career... which for many players is just a few first-round opponents before first-round exits, or exclusively opponents in their conference. This makes the RAPM adjustment less accurate!
-And again, unlike the historical RAPM we have, this playoff data does not have uncertainty bars posted (and indeed, the historical RAPM we have includes the team record relative to the full-season data, which allows us to estimate whether the missing data is biased up or down)

Since the Draymond-only and Curry-only minutes are so few, slight changes in the evaluation of players in the Draymond-only minutes can skew things. If the Draymond-only opponents would be slightly overrated in a playoff-only exercise like this (which I've just shown, at least for the 2016 Rockets series), then that would likewise cause Draymond to be overrated in these minutes.

It's not about picking samples where one player looks better. It's about the pros and cons and uncertainties of different metrics. Again, if they had used a regular season prior, or full-season data with postseason games optionally up-weighted and Draymond still came out over Curry, that would be different. But there really are genuine limitations with playoff-only data, that are different from full-season data, different from Squared2020's historical data, just different from any other lifetime RAPM we have.

It's not Engelmann's fault. I'm glad he posted this! There's some genuine broad trends to learn here! Like you said, Harden looks great here -- perhaps reason to reconsider the magnitude of his reputed playoff decline over his career. It's just that there's limitations beyond Engelmann's control. And as I have always argued (regardless of the accusations of a few oddly upset posters), it's important to remember the uncertainties and inherent limitations in any metric before interpreting. And alas, playoff-only data has some limitations that are unique to playoff-only data, and different from full-season data, regular season data, or the historical data we have.

I get the points that you're making and surely it's an issue that there's no age adjustment for people at wildly different points in their career. However, I also think you're overselling it. For instance, you bring up Ariza having one of the worst series of his career. Well surely that's in large part due to Draymond and the rest of the Warriors' excellent defense right? Ariza had a career BPM of 0.6 in the regular season and an on/off of +0.9. In 2016, he had a BPM of 0.4 and an on/off of +2.4. He was age 30 right in the middle of his career and the numbers were likely quite representative of his overall talent. His performing well below that was likely due to the Warriors' defense first and the same random variance you'd see in any sample second.

Beverley in the regular season had a 0.2 BPM and a +7.5 on/off compared to a career 0.7 BPM and +2.9 on/off. I feel like this stuff tends to even out in the long run more than you realize. Also, while the time Draymond played without Steph in 2016 is certainly a huge data point, it's not really THAT much of the sample. For instance, in the 2016 playoffs, Draymond played 715 total possessions without Steph. In 2017 with both players completely healthy through the playoffs and the Warriors playing 7 fewer games, Draymond played 265 possessions without Steph.

Now I'm certainly not saying to take all this data at face value and assume it has no more noise than a typical one season RAPM where everyone's at the same age, and I would never say for instance that Draymond's actually more valuable in the playoffs than Steph. I do think individual opponent adjustments aren't going to be a big issue for guys like Draymond with 150+ career playoff games though. I think where we really have to take the data with a grain of salt is guys like KG who played 2/3 of their playoff games from age 31-37 or someone like Jokic who got most of his off minutes in seasons where either he hadn't become a superstar yet or his entire starting lineup was hurt.

I think our conclusions aren't that dissimilar here in that I've decided Harden was less of a playoff choker than I thought and moved him up my list signficantly (although still not a huge amount) and more than anything, it really has shown me that Draymond was at least close to as crucial to the Warriors as Steph was. He's made the biggest jump for me, moving all the way to #30 for me from somewhere in the 40s. Other than those two guys though, I haven't adjusted anyone more than 2 or 3 spots on my all-time list and most guys didn't move at all.

[DraymondGold]

it's not just impressive, it was necessary. steph has only played 7 healthy playoffs in his career, way less than most any other superstar. without draymond wrecking 1st round teams, steph missed out on a finals in 2016 and a championship in 2018. a lot different resume without that.

Indeed, it was necessary! If they hadn't won in the early rounds without Curry (largely on the back of Draymond's impact), the Warriors don't make the the finals in 2016 and don't win the championship in 2018, although they still win in 2015, 2017, make a finals run in 2019, and win in 2022.

why isn't it compelling evidence? after all, RAPM either works or it doesn't, right? do i personally believe he was "more" valuable? probably not. though it seems unlikely that there is much difference. but like i said below, i don't weight RAPM as much as others. for those that do, this is exactly what the data is telling us. after all, isn't RAPM supposed to be able to account for opponent strength and adjust for it? so a first round opponent being bludgeoned or a close win over a great team should look similar. we can't just pick the samples where draymond only looks equal instead of better.

Sorry to miss this! It got lost in the shuffle with the oddly acerbic accusations of bias.

This was my point about playoff-only RAPM. If Draymond looked more valuable in full-season RAPM, or playoff-only RAPM with a regular season prior, or full-season RAPM with playoff games heavily weighted, that would be one thing. But this is playoff-only RAPM without any of those corrections... and so the conditions aren't great to appropriately "account for opponent strength and adjust for it" as you say.

Let's take that 2016 first round battle vs Houston (since I suspect you're familiar with that Houston team) as an example. Curry played 2 games in short minutes (1/2 of game 1 which the Warriors won handily, 1/2 of game 4 which the Warriors won handily). The Warriors won the series in 5 games.

How well would a playoff-only RAPM estimate the value of these players, to appropriately correct for the strength of schedule in the Draymond-only minutes? Let's sort the Houston players by minute, and look at their playoff basketball reference BPM (an imperfect stat, but at least somewhat stable in single-series samples, and a good enough first pass for this exercise).

-Harden played the most minutes, and had a very respectable BPM of +8.8. Harden has played 167 playoff games until 2024 (roughly 2 seasons of regular season data), in both conferences and including many multi-series runs so against a reasonable number of opponents at least for playoff RAPM. So his value estimation is caveated by the usual limitations of this exercise ('only' 2 season worth of data; assuming he's the same player from 2010 to 2024 which is obviously false), but otherwise the method shouldn't be too limited in evaluating him.

-Trevor Ariza played the next most minutes. He's played 106 playoff games (~1.3 seasons worth of data) over 15 years, so still a reasonable amount of data to not be noise, but starting to decrease. He's also played against 16 opposing teams in the playoffs -- so not bad, but clearly less than an equivalent length regular season sample. Again, this lack of opponent diversity makes it harder for the RAPM to accurately adjust. His series BPM was -6.6, his 2nd worst playoffs ever, and a far cry from his playoff career average of +1.1.
Do I think playoff-only RAPM accurately estimated the value of Trevor Ariza in the 5-game sample, which is needed to help scale the Curry-less Draymond minutes? It probably does OK, but lack of opponent diversity is starting to limit things, and the small-sample poor performance of Ariza is likely not captured -- so presumably playoff-only RAPM overrates 2016 playoff Trevor Ariza at least slightly.

-Dwight Howard played the next most minutes. He's played 125 playoff games (~1.5 seasons) over 15 years, had a series BPM of -0.4 compared to a career value of +2.9. He's played 15 opposing teams in the playoffs.
Like Trevor Ariza, Dwight has a non-negligible sample size, but has not played as many opponents as we would hope for an RAPM, had a below-average series for his standard, and so there's the possibility that playoff-only RAPM slightly overrates 2016 Dwight Howard.

-Patrick Beverly. 71 playoff games (~0.9 seasons) over over 12 years against 10 opposing teams. Again below average BPM in 2016 -- which no doubt may be correlated with the Warriors winning in the Draymond-run minutes due to Draymond's contributions, but also again may indicate that Beverly had a below average series and may be overrated by lifetime playoff RAPM relative to his career average. Now we only have 10 opposing teams to accurately evaluate the player by...

We'd expect as we go further down the Houston Roster, we'd get to worse players, who thus have fewer minutes and playoff games, and thus the RAPM (in playoff-only samples, where the opponent numbers are limited and the sample size is reduced) would have a harder time evaluating them. For example:
-Donatas Motiejūnas, 6th in Houston minutes in the series, has played 11 playoff games total (an unusably small sample for accurate RAPM) against 3 opposing teams total (he didn't even play against multiple teams like in regular season...)
-Michael Beasley, 7th in Houston minutes in the series, has played 25 playoff games total over 9 years against 7 opponents.

Already we're seeing a trend here. Harden aside, we either have players where there's basically no sample to accurately calculate RAPM, or players had below-average series relative to their career average and would thus likely be overrated by playoff-only RAPM. Perhaps some of their poor performance can be attributed to Draymond. But If those players were overrated, that would overrate the Rockets, thus overrate Draymond in the Draymond-only minutes, making it more likely for him to be wrongly rated over Curry.

And again this is somewhat unique to playoff-only data here.
-The small samples for players here are different from Engelmann's previous full-season and regular-season RAPM, albeit like the historical RAPM we have (though the off samples in this playoff-only RAPM are likely smaller for many players we care about than for the historical RAPM, like jake said above.)
-The small samples are spread out over many years and unevenly sampled making accurate evaluation of the mean value of the player difficult, different from Engelmann's previous full-season and regular season RAPM, especially different from the age-adjusted data Englemann posted, albeit like the historical rapm we have
-However, unlike any other lifetime RAPM (including the historical RAPM we have!!), these playoff games are not randomly selected against different opponents, but are instead limited to the very small sample of opponents the players happened to play in their career... which for many players is just a few first-round opponents before first-round exits, or exclusively opponents in their conference. This makes the RAPM adjustment less accurate!
-And again, unlike the historical RAPM we have, this playoff data does not have uncertainty bars posted (and indeed, the historical RAPM we have includes the team record relative to the full-season data, which allows us to estimate whether the missing data is biased up or down)

Since the Draymond-only and Curry-only minutes are so few, slight changes in the evaluation of players in the Draymond-only minutes can skew things. If the Draymond-only opponents would be slightly overrated in a playoff-only exercise like this (which I've just shown, at least for the 2016 Rockets series), then that would likewise cause Draymond to be overrated in these minutes.

It's not about picking samples where one player looks better. It's about the pros and cons and uncertainties of different metrics. Again, if they had used a regular season prior, or full-season data with postseason games optionally up-weighted and Draymond still came out over Curry, that would be different. But there really are genuine limitations with playoff-only data, that are different from full-season data, different from Squared2020's historical data, just different from any other lifetime RAPM we have.

It's not Engelmann's fault. I'm glad he posted this! There's some genuine broad trends to learn here! Like you said, Harden looks great here -- perhaps reason to reconsider the magnitude of his reputed playoff decline over his career. It's just that there's limitations beyond Engelmann's control. And as I have always argued (regardless of the accusations of a few oddly upset posters), it's important to remember the uncertainties and inherent limitations in any metric before interpreting. And alas, playoff-only data has some limitations that are unique to playoff-only data, and different from full-season data, regular season data, or the historical data we have.

I get the points that you're making and surely it's an issue that there's no age adjustment for people at wildly different points in their career. However, I also think you're overselling it. For instance, you bring up Ariza having one of the worst series of his career. Well surely that's in large part due to Draymond and the rest of the Warriors' excellent defense right? Ariza had a career BPM of 0.6 in the regular season and an on/off of +0.9. In 2016, he had a BPM of 0.4 and an on/off of +2.4. He was age 30 right in the middle of his career and the numbers were likely quite representative of his overall talent. His performing well below that was likely due to the Warriors' defense first and the same random variance you'd see in any sample second.

Beverley in the regular season had a 0.2 BPM and a +7.5 on/off compared to a career 0.7 BPM and +2.9 on/off. I feel like this stuff tends to even out in the long run more than you realize. Also, while the time Draymond played without Steph in 2016 is certainly a huge data point, it's not really THAT much of the sample. For instance, in the 2016 playoffs, Draymond played 715 total possessions without Steph. In 2017 with both players completely healthy through the playoffs and the Warriors playing 7 fewer games, Draymond played 265 possessions without Steph.

Now I'm certainly not saying to take all this data at face value and assume it has no more noise than a typical one season RAPM where everyone's at the same age, and I would never say for instance that Draymond's actually more valuable in the playoffs than Steph. I do think individual opponent adjustments aren't going to be a big issue for guys like Draymond with 150+ career playoff games though. I think where we really have to take the data with a grain of salt is guys like KG who played 2/3 of their playoff games from age 31-37 or someone like Jokic who got most of his off minutes in seasons where either he hadn't become a superstar yet or his entire starting lineup was hurt.

I think our conclusions aren't that dissimilar here in that I've decided Harden was less of a playoff choker than I thought and moved him up my list signficantly (although still not a huge amount) and more than anything, it really has shown me that Draymond was at least close to as crucial to the Warriors as Steph was. He's made the biggest jump for me, moving all the way to #30 for me from somewhere in the 40s. Other than those two guys though, I haven't adjusted anyone more than 2 or 3 spots on my all-time list and most guys didn't move at all.

I’m not opposed to this interpretation at all. Indeed, I’m not trying to pin all the difference between Curry and Draymond’s RAPM on this tiny sample size - of course not!

Moreso, I’m trying to walk through the flaws of playoff specific RAPM, that are more limiting to this playoff RAPM sample. Players have smaller samples than usual, and uniquely, players have significantly more unevenly sampled opponents than usual.

For Draymond vs Curry, I’m certainly not trying to pin all of the difference on this one series or anything silly like that. But when players share a lot of time together — when there risks being collinearity — the importance of small samples becomes magnified. According to pbpstats, 885/2392 = 37% of the total possessions in this sample where Draymond was on the court and Curry was off the court occurred when Curry was out for the full game due to an injury. This specific series which I walked through accounts for 297/2392 = 12% of the total time when Draymond was on and Curry was off. Not huge, but not as minuscule as you might think.

Like you say, Trevor Ariza had a 0.6 career BPM, 0.4 regular season BPM in 2016, and -6.6 BPM in this series. Of course that’s due to the Warriors’ defense (which Draymond was the biggest part of) and random variance! I’m not trying to say otherwise. This one series I walked through doesn’t explain everything. My point is that with Draymond and Curry (and indeed any partnership that overlaps significantly in minutes; Stockton and Malone come to mind in the historical RAPM), the mean estimation from the RAPM becomes more sensitive on small samples where one plays and the other doesn’t… which can be biased by a bit of context here and a player missing a few games there, such as (but not limited to) the context I walked through with the 2016 Houston series.

Again, Draymond was absolutely crucial to the Warriors’ success, and I’ve been arguing he’s underrated for years. But I don’t think people should start rating Draymond in the top 15 over Curry; and it's not inconsistent to still value RAPM, while still thinking playoff Draymond < playoff Curry. Playoff RAPM can have weird quirks — there’s a reason people first started making regular season RAPM when RAPM was popularized, despite the playoffs being considered more important. The unexpected ranking can be simply explained based on the limitations of the playoff-only data, limitations in the stat itself, and the inherent wide uncertainty range of RAPM. And the general strong performance of both players in this stat can be simply explained by the general strength of both players in the playoffs. That’s all.

Agreed about Jokic and KG. It’s such a shame that we saw so little of playoff KG in the years surrounding his peak. His casual reputation would probably be a lot different with a few more deeper runs during his peak.

[iggymcfrack]

Indeed, it was necessary! If they hadn't won in the early rounds without Curry (largely on the back of Draymond's impact), the Warriors don't make the the finals in 2016 and don't win the championship in 2018, although they still win in 2015, 2017, make a finals run in 2019, and win in 2022.

Sorry to miss this! It got lost in the shuffle with the oddly acerbic accusations of bias.

This was my point about playoff-only RAPM. If Draymond looked more valuable in full-season RAPM, or playoff-only RAPM with a regular season prior, or full-season RAPM with playoff games heavily weighted, that would be one thing. But this is playoff-only RAPM without any of those corrections... and so the conditions aren't great to appropriately "account for opponent strength and adjust for it" as you say.

Let's take that 2016 first round battle vs Houston (since I suspect you're familiar with that Houston team) as an example. Curry played 2 games in short minutes (1/2 of game 1 which the Warriors won handily, 1/2 of game 4 which the Warriors won handily). The Warriors won the series in 5 games.

How well would a playoff-only RAPM estimate the value of these players, to appropriately correct for the strength of schedule in the Draymond-only minutes? Let's sort the Houston players by minute, and look at their playoff basketball reference BPM (an imperfect stat, but at least somewhat stable in single-series samples, and a good enough first pass for this exercise).

-Harden played the most minutes, and had a very respectable BPM of +8.8. Harden has played 167 playoff games until 2024 (roughly 2 seasons of regular season data), in both conferences and including many multi-series runs so against a reasonable number of opponents at least for playoff RAPM. So his value estimation is caveated by the usual limitations of this exercise ('only' 2 season worth of data; assuming he's the same player from 2010 to 2024 which is obviously false), but otherwise the method shouldn't be too limited in evaluating him.

-Trevor Ariza played the next most minutes. He's played 106 playoff games (~1.3 seasons worth of data) over 15 years, so still a reasonable amount of data to not be noise, but starting to decrease. He's also played against 16 opposing teams in the playoffs -- so not bad, but clearly less than an equivalent length regular season sample. Again, this lack of opponent diversity makes it harder for the RAPM to accurately adjust. His series BPM was -6.6, his 2nd worst playoffs ever, and a far cry from his playoff career average of +1.1.
Do I think playoff-only RAPM accurately estimated the value of Trevor Ariza in the 5-game sample, which is needed to help scale the Curry-less Draymond minutes? It probably does OK, but lack of opponent diversity is starting to limit things, and the small-sample poor performance of Ariza is likely not captured -- so presumably playoff-only RAPM overrates 2016 playoff Trevor Ariza at least slightly.

-Dwight Howard played the next most minutes. He's played 125 playoff games (~1.5 seasons) over 15 years, had a series BPM of -0.4 compared to a career value of +2.9. He's played 15 opposing teams in the playoffs.
Like Trevor Ariza, Dwight has a non-negligible sample size, but has not played as many opponents as we would hope for an RAPM, had a below-average series for his standard, and so there's the possibility that playoff-only RAPM slightly overrates 2016 Dwight Howard.

-Patrick Beverly. 71 playoff games (~0.9 seasons) over over 12 years against 10 opposing teams. Again below average BPM in 2016 -- which no doubt may be correlated with the Warriors winning in the Draymond-run minutes due to Draymond's contributions, but also again may indicate that Beverly had a below average series and may be overrated by lifetime playoff RAPM relative to his career average. Now we only have 10 opposing teams to accurately evaluate the player by...

We'd expect as we go further down the Houston Roster, we'd get to worse players, who thus have fewer minutes and playoff games, and thus the RAPM (in playoff-only samples, where the opponent numbers are limited and the sample size is reduced) would have a harder time evaluating them. For example:
-Donatas Motiejūnas, 6th in Houston minutes in the series, has played 11 playoff games total (an unusably small sample for accurate RAPM) against 3 opposing teams total (he didn't even play against multiple teams like in regular season...)
-Michael Beasley, 7th in Houston minutes in the series, has played 25 playoff games total over 9 years against 7 opponents.

Already we're seeing a trend here. Harden aside, we either have players where there's basically no sample to accurately calculate RAPM, or players had below-average series relative to their career average and would thus likely be overrated by playoff-only RAPM. Perhaps some of their poor performance can be attributed to Draymond. But If those players were overrated, that would overrate the Rockets, thus overrate Draymond in the Draymond-only minutes, making it more likely for him to be wrongly rated over Curry.

And again this is somewhat unique to playoff-only data here.
-The small samples for players here are different from Engelmann's previous full-season and regular-season RAPM, albeit like the historical RAPM we have (though the off samples in this playoff-only RAPM are likely smaller for many players we care about than for the historical RAPM, like jake said above.)
-The small samples are spread out over many years and unevenly sampled making accurate evaluation of the mean value of the player difficult, different from Engelmann's previous full-season and regular season RAPM, especially different from the age-adjusted data Englemann posted, albeit like the historical rapm we have
-However, unlike any other lifetime RAPM (including the historical RAPM we have!!), these playoff games are not randomly selected against different opponents, but are instead limited to the very small sample of opponents the players happened to play in their career... which for many players is just a few first-round opponents before first-round exits, or exclusively opponents in their conference. This makes the RAPM adjustment less accurate!
-And again, unlike the historical RAPM we have, this playoff data does not have uncertainty bars posted (and indeed, the historical RAPM we have includes the team record relative to the full-season data, which allows us to estimate whether the missing data is biased up or down)

Since the Draymond-only and Curry-only minutes are so few, slight changes in the evaluation of players in the Draymond-only minutes can skew things. If the Draymond-only opponents would be slightly overrated in a playoff-only exercise like this (which I've just shown, at least for the 2016 Rockets series), then that would likewise cause Draymond to be overrated in these minutes.

It's not about picking samples where one player looks better. It's about the pros and cons and uncertainties of different metrics. Again, if they had used a regular season prior, or full-season data with postseason games optionally up-weighted and Draymond still came out over Curry, that would be different. But there really are genuine limitations with playoff-only data, that are different from full-season data, different from Squared2020's historical data, just different from any other lifetime RAPM we have.

It's not Engelmann's fault. I'm glad he posted this! There's some genuine broad trends to learn here! Like you said, Harden looks great here -- perhaps reason to reconsider the magnitude of his reputed playoff decline over his career. It's just that there's limitations beyond Engelmann's control. And as I have always argued (regardless of the accusations of a few oddly upset posters), it's important to remember the uncertainties and inherent limitations in any metric before interpreting. And alas, playoff-only data has some limitations that are unique to playoff-only data, and different from full-season data, regular season data, or the historical data we have.

I get the points that you're making and surely it's an issue that there's no age adjustment for people at wildly different points in their career. However, I also think you're overselling it. For instance, you bring up Ariza having one of the worst series of his career. Well surely that's in large part due to Draymond and the rest of the Warriors' excellent defense right? Ariza had a career BPM of 0.6 in the regular season and an on/off of +0.9. In 2016, he had a BPM of 0.4 and an on/off of +2.4. He was age 30 right in the middle of his career and the numbers were likely quite representative of his overall talent. His performing well below that was likely due to the Warriors' defense first and the same random variance you'd see in any sample second.

Beverley in the regular season had a 0.2 BPM and a +7.5 on/off compared to a career 0.7 BPM and +2.9 on/off. I feel like this stuff tends to even out in the long run more than you realize. Also, while the time Draymond played without Steph in 2016 is certainly a huge data point, it's not really THAT much of the sample. For instance, in the 2016 playoffs, Draymond played 715 total possessions without Steph. In 2017 with both players completely healthy through the playoffs and the Warriors playing 7 fewer games, Draymond played 265 possessions without Steph.

Now I'm certainly not saying to take all this data at face value and assume it has no more noise than a typical one season RAPM where everyone's at the same age, and I would never say for instance that Draymond's actually more valuable in the playoffs than Steph. I do think individual opponent adjustments aren't going to be a big issue for guys like Draymond with 150+ career playoff games though. I think where we really have to take the data with a grain of salt is guys like KG who played 2/3 of their playoff games from age 31-37 or someone like Jokic who got most of his off minutes in seasons where either he hadn't become a superstar yet or his entire starting lineup was hurt.

I think our conclusions aren't that dissimilar here in that I've decided Harden was less of a playoff choker than I thought and moved him up my list signficantly (although still not a huge amount) and more than anything, it really has shown me that Draymond was at least close to as crucial to the Warriors as Steph was. He's made the biggest jump for me, moving all the way to #30 for me from somewhere in the 40s. Other than those two guys though, I haven't adjusted anyone more than 2 or 3 spots on my all-time list and most guys didn't move at all.

I’m not opposed to this interpretation at all. Indeed, I’m not trying to pin all the difference between Curry and Draymond’s RAPM on this tiny sample size - of course not!

Moreso, I’m trying to walk through the flaws of playoff specific RAPM, that are more limiting to this playoff RAPM sample. Players have smaller samples than usual, and uniquely, players have significantly more unevenly sampled opponents than usual.

For Draymond vs Curry, I’m certainly not trying to pin all of the difference on this one series or anything silly like that. But when players share a lot of time together — when there risks being collinearity — the importance of small samples becomes magnified. According to pbpstats, 885/2392 = 37% of the total possessions in this sample where Draymond was on the court and Curry was off the court occurred when Curry was out for the full game due to an injury. This specific series which I walked through accounts for 297/2392 = 12% of the total time when Draymond was on and Curry was off. Not huge, but not as minuscule as you might think.

Like you say, Trevor Ariza had a 0.6 career BPM, 0.4 regular season BPM in 2016, and -6.6 BPM in this series. Of course that’s due to the Warriors’ defense (which Draymond was the biggest part of) and random variance! I’m not trying to say otherwise. This one series I walked through doesn’t explain everything. My point is that with Draymond and Curry (and indeed any partnership that overlaps significantly in minutes; Stockton and Malone come to mind in the historical RAPM), the mean estimation from the RAPM becomes more sensitive on small samples where one plays and the other doesn’t… which can be biased by a bit of context here and a player missing a few games there, such as (but not limited to) the context I walked through with the 2016 Houston series.

Again, Draymond was absolutely crucial to the Warriors’ success, and I’ve been arguing he’s underrated for years. But I don’t think people should start rating Draymond in the top 15 over Curry; and it's not inconsistent to still value RAPM, while still thinking playoff Draymond < playoff Curry. Playoff RAPM can have weird quirks — there’s a reason people first started making regular season RAPM when RAPM was popularized, despite the playoffs being considered more important. The unexpected ranking can be simply explained based on the limitations of the playoff-only data, limitations in the stat itself, and the inherent wide uncertainty range of RAPM. And the general strong performance of both players in this stat can be simply explained by the general strength of both players in the playoffs. That’s all.

Agreed about Jokic and KG. It’s such a shame that we saw so little of playoff KG in the years surrounding his peak. His casual reputation would probably be a lot different with a few more deeper runs during his peak.

Yeah, I feel like I was disagreeing with you on minor details while still largely drawing the same conclusions as you did overall. We seem to be on pretty similar pages. Now that you’ve clarified with more detail, I think our interpretations are even more harmonious than I thought before.