LukaTheGOAT wrote:Using just raw on/off is not a great way to evaluate impact, as unlike PIPM and RAPM, it does not try to adjust for the teammates you are playing for. And furthermore, if we just follow raw on/off, a past peak David Robinson is better than Shaq, Lebron, etc. It is notable but not the only thing.
A few thoughts about raw on/off for the playoffs. I agree with you that you should never look at a raw on/off (or really any stat) and say "player X is better than players Y, Z and Q". Stats always have to be tied to what they're measuring, and this stat is (to whatever degree noise allows) is trying to correlate a player's presence with the team's scoring margins. That's not the same thing as 'this player is better'.
With that said, I do think people way over-react (and under-appreciate) the information that can be found in playoffs on/off +/-. Yes, it's noisy. And yes, you have to pay attention to sample sizes and be more discerning than if you had a perfect dataset to work with. But as I've pointed out before, there is usable signal within that noise, and I think that signal is important BECAUSE there's such a lack of great postseason +/- information to work with. Even in the larger regular season samples, it is ABUNDANTLY clear that the box scores aren't equipped to accurately estimate overall impact. That's why there's a need for the impact stat approach in the first place. That need doesn't just go away for the playoffs, so relying purely on the box scores in the playoffs is just as lacking as it would be in the regular season...more-so, actually, because of the very small samples. Even box score stats can be gamed in smaller samples, so if we only have the box scores to go off, way too much of the real story is lost.
Anyway. You made a good point about raw on/off not adjusting sufficiently for teammates. It does offer a first-blush, basic correction for teammates that purely raw +/- doesn't, but not enough. So, one quick hack would be to compare an individual's playoffs on/off +/- with the on/off +/- of his teammates. There's several ways to do this, but for the below example I compared the highest playoffs on/off with the second-best on the team (among big-minute rotation players), and note the difference.
So, for this exercise, I use a few general rules to increase the likelihood of useful signal in the midst of the small sample/potential noise.
Rules:*Teams had to make at least the Conference Finals
*Noting only the highest on-off +/- scores on the team as the ones likely driving the positive margin
*Using difference b/w highest on/off score and second-highest as a means to separate strong individual impact from strong units
*I looked at every conference finalist from 1997 - 2020 for this data
*Setting thresholds (to sort the 96 represented seasons)
I took the average on/off +/- of the top player on each team (+16.4)
I then took the average difference b/w top and 2nd player on each team (+7.7)
Results summary of the "16/8" club31 player playoffs where team leader on/off > 16, difference b/w P1 & P2 > 8
Only 4 players (LeBron, Shaq, KG, Duncan) made list more than once (accounted for 12 of the 31 16/8 seasons)
Notable 1-offs include 1997 Jordan & 2006 D. Wade
Typically, the non-megastars fit into one of several categories not well captured by boxscores. Few non-intuitive 1-offs. Defense: '04 Ben Wallace, 02 Doug Christie, '15 Draymond Green, '11 Luol Deng, '13 Marc Gasol, '99 Robinson
Floor stretchers: '00 Reggie Miller, '01 Ray Allen, '10 Rashard Lewis
Floor generals: '02 Jason Kidd, '16 Kyle Lowry, '06 Steve Nash
Mega 6th men: '05 Manu GInobili, '12 James Harden
Past prime stars: '97 Clyde Drexler, '99 Arvydys Sabonis, (Jordan, Robinson & Reggie would fit here too)
('17 Kawhi Leonard makes list, w/ small asterisk since he only played 12 of the team's 16 games that postseason)
Anyway, just a simple exercise like this shows a lot of, IMO, useful info. For example, you mention that by raw playoffs on/off +/- David Robinson looks better than Shaq and LeBron. Well, here we see that '99 Robinson was the only version of late-prime DRob to make this list (by 2001, even though he had a great on/off, he had been surpassed by Duncan), while both Shaq and LeBron were on here multiple times (7 total, between them).
I should note, NOT making this list doesn't necessarily mean a player didn't have a great postseason impact in a given season. They could've had a great on/off in a great unit (e.g. 2011 Dirk met the on/off threshold, but not gap over 2nd because unit was strong). But, on that note...
Focus: you were talking about Steph, in a post that compares him with KD and also mentions Magic, MJ and LeBron's postseason peaks. While it may not be rigorous, it's at least worth noting, to me, that out of all the many times that Steph and KD have played on contending teams (and their entire careers were in the databall era), neither of them had even one postseason run make the above list. Of the others listed, LeBron has the most such seasons on record (4) and even though we only have the last 2 seasons of MJ's Bulls career, he reached the thresholds in '97 and really was barely short of the thresholds to have a second such postseason run in '98 (led Bulls w/ +14.6 postseason on/off, +6.3 higher than 2nd on team). When you factor in that most of the other players in Ben's peak project have seasons like that as well...like I said, I at least take note.
Following up, of the 2, Curry seems to have the more impressive postseason +/- record. As I noted in a previous post, Curry's raw on/off +/- in his prime-ish seasons was +12.6, while Durant's was +5.1. Curry's record looks more like what I described for 2011 Dirk, where he had high numbers but may have been parts of units such that we couldn't ascribe the impact just to him (e.g. the Draymond effect), whereas Durant really doesn't have a postseason record where he stands out even to that extent in the +/- data.
Anyway. All of this is obviously food for thought. But to me it is useful information that adds depth to postseason analysis that you just can't find if you only rely on team results and/or boxscore data.