What Are Your Personal Opinions On RAPM?

[The-Power]

People have been looking for the basketball version of baseballs WAR for years. Per winshares plus minus all flopped now it's RAPM

You don't think RAPM is a big step forward in the area of advanced stats and that its approach (!) is actually pretty much flawless?

[trex_8063]

I somewhat understand where the backlash/resistance to RAPM is coming from, but I really can't understand the vehemence with which some people are against it. I mean I do, but I don't. I myself tend to be resistant to change. Even necessary change I have to allow time for it to occur. We've all got our preconceived notions, preconceived "truths" (more on these "truths" below), and it's only human nature to be resistant to something that runs counter to those "truths". And the longer we've held on to them, the more resistant we are to change (or the more vehement and hostile we may be to a differing ideal).

That being said, from a practical standpoint, why cast information aside? RAPM clearly has some application, even if you're skeptical of the noise. Why would we not want to use all available information in player evaluations? I simply don't get that. And fwiw, this is from someone who debated Doc in the top 100 project on the use of the stat, and stated I thought he was putting too many eggs in this particular basket.

Yes, it's got noise based on line-ups. Yes, it's somewhat subject to player use/application (how coaching/management chooses to use a player). Yes, it appears to have the occasional anomalous result.

All of this just suggests it should not be used as the ONLY tool of evaluation, but rather as one more piece in a fairly broad puzzle. After all.....

...In many instances, it appears to confirm what is largely expected based on other metrics, eye-test, and narrative: hey, Lebron James was the best in the league (surprise surprise); prime/peak versions of Tim Duncan, Kevin Garnett, Shaquille O'Neal, Steve Nash all appear (surprise surprise) among the league's hyper-elite. DRob, Mutombo, Mourning, Ben Wallace all look remarkable defensively (surprise surprise); Carlos Boozer, Amar'e Stoudemire, James Harden (prior to this season), and David Lee all look consistently poor defensively (no big surprise there); for certain "glue guys" who, based on eye-test and narrative, appear to be better than their box stats would suggest, we often find support of their impact on RAPM (e.g. Iggy); or in some instances it lends credence to narratives/perceptions about players which aren't well-supported by other box/advanced metrics (e.g. the box and advanced metrics of prime/peak Jason Kidd did not suggest he was as good as the media and most of the rest of us thought he was; but RAPM kinda rounds out that perception); etc.

I'm not trying to suggest it's always right. But I am suggesting it appears to be more or less right (that is: re-enforcing what was already kinda obvious) often enough that it's proven some degree of reliability; and thus should (imo) be among the tools (NOT the only tool, and not necessarily even the primary tool) used for player evaluation.


On "truths"....
I've used it in quotation marks because we often get it wrong; but I want it noted that I suspect in some instances our preconceived truths were probably reasonably accurate. And this is where the databall era and our heavy reliance on the numbers----not just RAPM, although RAPM is part of the databall "revolution"----can sometimes give a potentially unflattering or unfair representation of certain old players (I suspect purely because we do not have the impact data to back up what was long held to be true).
What I see as a slight overemphasis on shooting efficiency is where this has the most potential to come into play. Not that it isn't important to score efficiently (unless you're the 1960's Celtics), but there's a tendency to conclude that anyone with mediocre (or poor) shooting efficiency must be an offensive negative (or neutral at best)......which isn't always the case.

I'll again site Jason Kidd as an example. You scrutinize Kidd's box and advanced stats and one thing which may jump out at you was that he was pretty consistently below (a couple times pretty significantly below) league average shooting efficiency during his prime, sometimes while taking more shot attempt than anyone else on the team.

Guy like that has got to be an offensive liability, right? And WS/48 (largely an offensive measure), for instance, consistently did NOT rate Kidd as the best on the Nets team in any of '02-'04, nor on the Suns of '00 or '01.

Yet all of this flies in the face of our perception of what was happening at the time. While he was clearly no Magic Johnson, I doubt many of us viewed him as an offensive liability. And in the years specified, there was little doubt he was the leader/best player on those teams. Indeed, and in spite of what those numbers say, I suspect most/many of us considered him a top 10 (if not occasionally top 5) player for multiple seasons in the heart of his prime. In the RealGM RPoY project he was 8th in '99 and '01, 6th in '02. In the MVP vote was in the top 9 FIVE times, twice in the top 5 (as high as #2 in '02).

In this instance, RAPM can perhaps shed a little light on things, indicating why our perceptions run counter to what many of the box/advanced metrics suggest. That he was an effective defender was pretty apparent, and despite his poor shooting efficiency and his usually just one step above pedestrian WS/48, his rank in non-scaled PI ORAPM (where available) by year was:
'99: 13th in league
'00: 14th in league
'02: tied for 22nd
'03: tied for 8th
and very respectable in '04.

So he appeared to, in some atypical fashion that is poorly captured by advanced metrics, exert a fairly positive impact on the offensive end.


How is this relevant to what I mentioned about certain older players getting under-repped by an increased emphasis on the numbers? Well, it comes into play with other high-ish volume/not so high-ish efficiency players who were lauded at the time, but are now falling from grace in the databall era. Guys who come to mind are Isiah Thomas and Bob Cousy. Both were considered fairly big-time players in their era (Cousy especially), and in particular dangerous and effective offensive players. Yet in this day, we appear to be down-grading their quality of play based largely on mediocre shooting efficiency (and the effect that has on WS/48, another favorite stat).

I've done a couple little studies to try and evaluate Cousy's impact, but obviously it's limited. I suspect (but obviously cannot prove) that, similar to Kidd, if we had RAPM for these guys, we would see an offensive impact which appears bigger than what the advanced metrics suggest. But we don't have that, and consequently these guys are getting *downgraded because the metrics which ARE available aren't as flattering as we'd like.

*I should note that a bit of a downgrade is justified imo, particularly for Isiah (who I think gets insanely overrated by many casual fans).

Anyway, I think I'm rambling at this point, so I'll stop.

[magicmerl]

^
That's such a bad example though. Why would a (terrific) player who specializes in taking inefficient jump shots at high volume be more portable than a DPOY caliber player whose salient quality on that end is his versatility?

The fact that that's what you think LaMarcus Aldridge is shows you don't see much of the Blazers.

We don't see much of them because they aren't a team going deep in the playoffs.

[magicmerl]

Coming back to the RAPM debate, I haven't actually used it in the basket ball realm, but I *have* used the technique when analysing the stock market. It's very useful but it is absolutely not gospel. There is an error component that can easily be larger than the difference between the coefficients (i.e. a +8.05 isn't necessarily better than +7.5 if the error range is 1).

The problem comes when some factors are non-randomly paired with each other, which happens in basketball games ALL THE TIME (like the starters playing most of the game with each other). This means that the few minutes where the unlikely combinations occur are given disproportianate weighting, and the random events of that tiny sample size overwhelm the other factors.

For this reason I have the suspicion that RAPM is a less valid application of covariate analysis than other areas (like stock prices).

[bondom34]

I somewhat understand where the backlash/resistance to RAPM is coming from, but I really can't understand the vehemence with which some people are against it. I mean I do, but I don't. I myself tend to be resistant to change. Even necessary change I have to allow time for it to occur. We've all got our preconceived notions, preconceived "truths" (more on these "truths" below), and it's only human nature to be resistant to something that runs counter to those "truths". And the longer you've held on to them, the more resistant we are to change (or the more vehement and hostile we may be to a differing ideal).

That being said, from a practical standpoint, why cast information aside? RAPM clearly has some application, even if you're skeptical of the noise. Why would we not want to use all available information in player evaluations? I simply don't get that. And fwiw, this is from someone who debated Doc in the top 100 project on the use of the stat, and stated I thought he was putting too many eggs in this particular basket.

Yes, it's got noise based on line-ups. Yes, it's somewhat subject to player use/application (how coaching/management chooses to use a player). Yes, it appears to have the occasional anomalous result.

All of this just suggests it should not be used as the ONLY tool of evaluation, but rather as one more piece in a fairly broad puzzle. .

This really sums up a lot of what I was trying to say as well. Again, I'm not a believer in it as gospel, or even the best stat available, but its a stat. It is relevant and may be considered. It isn't the be all end all player rank, and just because player A is better than B in RAPM doesn't really mean he's better in reality.

[ceiling raiser]

I'm a big proponent of RAPM and other approaches intended to isolate player “impact” (WOWY SRS studies, APM, net on/off, etc.). I feel I did spend a large chunk of my formative years watching the NBA and focussing on the wrong things. Even if one doesn't agree with the specific technique (there are tutorials, and datasets out there which are very accessible for those interested in calculating a vanilla APM or RAPM for himself/herself), I think the approach makes sense to most (analyzing all lineups for performance, adjusting them for quality of teammates and opponents, etc.).

My only concerns would be:

1) Lack of a consistent data set for each season. The three main providers (J.E., acrossthecourt, talkingpractice/gotbuckets) have all actually been pretty transparent with in describing their priors (and from speaking with mystic on APBR, he’s been very clear as well in responding to specific questions), but I think it would be difficult to replicate their results precisely, since I believe they do a lot of quality control to improve their results (adjusting for margin/removing garbage time, cleaning up bad data, removing players in limited minutes, etc.), that a simple vanilla APM/RAPM would not.

2) I wish more studies would list error estimates. I think a couple have (and they’ve been linked here), but that can improve context.

3) That people seem to be moving away from RAPM in favor of a hybrid approach. RPM/xRAPM either uses a box score SPM in a blend, or as a prior from my understanding, which certainly does improve predictive value.

4) Some posters (not so much on here) tend to cite RAPM, without proper context of player role/responsibility.

One thing that does frustrate me about a lot of the opposition to RAPM and other “impact” metrics, is that the alternative isn’t attractive to me. A big part of why I joined RealGM is because I enjoyed reading some of the older projects, with attempts to ascertain player “impact” (or "goodness", as opposed to "greatness"/"legacies").

There are plenty of ways to dissect the game without citing RAPM though. I really wish more posters would analyze and break down tape (I could definitely afford to do so, myself). Spaceman (among others…there are plenty in the Westbrook/LeBron/CP3/etc. threads who excel in this regard as well) is terrific at this, ElGee and Dipper 13 have actually generated their own data *from* tape which is incredibly cool, and some posters (MacGill, fatal9, lorak etc.) are great at scouting players from video. If there was more of this, and less citing of RAPM and other “impact” metrics in debates and in studies, I'd be all for it. If instead, we’re going to have more box score (and box score based) metrics, and tons of narrative/accolade laundry lists, I don’t think the discussion is as interesting or exciting for me. I definitely appreciate tons of depth, whether it is quantitative *or* qualitative (it's especially worthwhile for me when it incorporates both).

EDIT: This is just my perspective. I understand that some people are interested in different approaches, I just wanted to note that I prefer the "goodness" discussions rather than those concerning player "greatness".

[E-Balla]

^
That's such a bad example though. Why would a (terrific) player who specializes in taking inefficient jump shots at high volume be more portable than a DPOY caliber player whose salient quality on that end is his versatility?

The fact that that's what you think LaMarcus Aldridge is shows you don't see much of the Blazers.

We don't see much of them because they aren't a team going deep in the playoffs.

Ok that's cool. Just don't have a strong opinion on a player you don't watch. I haven't seen much of Brooklyn this year so if I said Deron lost a step I'd really just be talking out of my ass.

[Moonbeam]

I like the idea behind RAPM very much. My current hesitation in using it is that I'd like to see a little more under the hood in terms of the technical details.

It's important to note that there is an element of choice behind RAPM. It involves a penalty term which is meant to reduce the variance in the estimates by way of adding bias (toward 0). How big that penalty should be is a common problem in statistics, and standard techniques are subject to variation themselves. I suspect that is why there appear to be different sources for RAPM - the guys behind it may not have the same method of choosing this penalty. I'd be very interested in those details, and whether the penalty is the same for all years or whether it varies from year to year.

The prior-informed version requires choices as well. Basically, it requires for each player a distribution of the estimate as a starting point. So at the beginning, for example, you might say "I think LeBron is a +8 player so I'll put on his coefficient a prior with an average at 8 for which most of the distribution is between 6 and 10. I think Durant is also a +8 but I'm not as sure, so for his prior I'll put a distribution with an average of 8 but where most of the distribution is between 5 and 11". This is not a bad strategy because it helps to reduce the bias toward 0 (we'd most likely be interested in using this or any stat to highlight the best players after all).

But there are harder choices as well. I think most of the prior-informed RAPM stats use the previous season to supply the prior. What do you do with Derrick Rose? What prior will you use for Paul George next year? How about rookies?

Another thing I'd like to see are error estimates. With any method (non-prior informed or prior-informed), you should be able to get not just the coefficient for each player, but an estimate of the associated error. So I'd like to see it say "LeBron is estimated as a +8 player, and we're 95% sure he's somewhere between a 6.7 player and a 9.3 player". Ridge regression has a known formula for standard errors, and with a Bayesian estimation for prior-informed RAPM it is easy to get 95% credible intervals.

In short, I think more transparency would help me buy in a bit more. I have no doubt that the guys who produce RAPM estimates know what they are doing, but those details would be great to see!

I'm pretty big on playoff performance, and as far as I know, there is no playoff version available, or even a version that combines regular season and playoffs.

The usual caveats apply as well. There is more that goes into how good a player is than a stat like this (or any other) can measure. Role is one, and coaching is another. If a coach has a scheme that totally depends on a player, his impact may inflate his "goodness", while a coach with a more robust system might cause his players to have impact stats that measure closer to 0. If a player is being misused, his impact stats will undersell him.

[magicmerl]

The fact that that's what you think LaMarcus Aldridge is shows you don't see much of the Blazers.

We don't see much of them because they aren't a team going deep in the playoffs.

Ok that's cool. Just don't have a strong opinion on a player you don't watch. I haven't seen much of Brooklyn this year so if I said Deron lost a step I'd really just be talking out of my ass.

Oh, I'm fully aware that Portland's offense is built around LMA, and as such he shouldn't be unfairly penalised for taking shots the offense calls for him to take even though it makes him 'inefficient'. The exact same thing happens to Harden.

[Moonbeam]

Personally I like RPM better.

You know that they're essentially the same thing, created by the same statistician, with the only difference being that RPM is informed by the box score, yes? RPM is a biased version of RAPM.

All RAPM measures are biased, by design! I understand that incorporating box score stuff makes RPM less "pure" from the perspective of not caring about production, but those priors have to come from somewhere.

[E-Balla]

We don't see much of them because they aren't a team going deep in the playoffs.

Ok that's cool. Just don't have a strong opinion on a player you don't watch. I haven't seen much of Brooklyn this year so if I said Deron lost a step I'd really just be talking out of my ass.

Oh, I'm fully aware that Portland's offense is built around LMA, and as such he shouldn't be unfairly penalised for taking shots the offense calls for him to take even though it makes him 'inefficient'. The exact same thing happens to Harden.

Yeah I wasn't referring to you specifically just that point.

James Harden doesn't play like that because of his coach or his team's strategy. He's not a player that bent his game for a coach, the coach picked him because he was already the most extreme 3 + free throw + 3 player in the league by far. At Arizona State he was known as someone that only attempted layups, free throws, and threes and in OKC he was known for the same thing. His shot distribution in OKC and Houston are almost exactly the same. You can go to the original James Harden is a superstar thread and see a ton of people saying he isn't because he has too simple a game. On the other hand guys that really changed with their teams' systems or their own progression like LMA, Lebron, Ray Allen, KG, and Melo have had their shot distributions change drastically.

[Moonbeam]

Should have read the whole topic before replying!

trex, magicmerl and fpliii have excellent posts on this page. Excellent!

Perhaps I just haven't looked in the right places for the technical details.

[LA Bird]

I'm pretty big on playoff performance, and as far as I know, there is no playoff version available, or even a version that combines regular season and playoffs.

GotBuckets RAPM combines regular season and playoffs.
I don't think a playoffs only RAPM will be very reliable considering how small the sample size is.

[Moonbeam]

I'm pretty big on playoff performance, and as far as I know, there is no playoff version available, or even a version that combines regular season and playoffs.

GotBuckets RAPM combines regular season and playoffs.
I don't think a playoffs only RAPM will be very reliable considering how small the sample size is.

Maybe not, but you might be able to do something by getting regular season estimates as well as regular season + playoffs estimates to get something useful that is playoffs only.

[Jaivl]

I'm pretty big on playoff performance, and as far as I know, there is no playoff version available, or even a version that combines regular season and playoffs.

GotBuckets RAPM combines regular season and playoffs.
I don't think a playoffs only RAPM will be very reliable considering how small the sample size is.

Maybe not, but you might be able to do something by getting regular season estimates as well as regular season + playoffs estimates to get something useful that is playoffs only.

PO RAPM with RS RAPM as a heavy prior?

[G35]

I somewhat understand where the backlash/resistance to RAPM is coming from, but I really can't understand the vehemence with which some people are against it. I mean I do, but I don't. I myself tend to be resistant to change. Even necessary change I have to allow time for it to occur. We've all got our preconceived notions, preconceived "truths" (more on these "truths" below), and it's only human nature to be resistant to something that runs counter to those "truths". And the longer we've held on to them, the more resistant we are to change (or the more vehement and hostile we may be to a differing ideal).

That being said, from a practical standpoint, why cast information aside? RAPM clearly has some application, even if you're skeptical of the noise. Why would we not want to use all available information in player evaluations? I simply don't get that. And fwiw, this is from someone who debated Doc in the top 100 project on the use of the stat, and stated I thought he was putting too many eggs in this particular basket.

Yes, it's got noise based on line-ups. Yes, it's somewhat subject to player use/application (how coaching/management chooses to use a player). Yes, it appears to have the occasional anomalous result.

All of this just suggests it should not be used as the ONLY tool of evaluation, but rather as one more piece in a fairly broad puzzle. After all.....

This is a great response, very tactful and you seem as if you want to know how someone has some resistance to the stat. I will only speak for myself, but I have no resistance against information. I always welcome information. Now here is one of my favorite quotes to help explain my position,

"I am always willing to learn, I am not always willing to be taught."

When advanced stats are brought up it's not just brought up as information. It's brought up to provide a hammer and bludgeon everyone with "facts". It's all in how a person passes along the information. Typically, if you do not buy into the results RAPM (or any other advanced stat provides) then it immediately goes into questioning your intelligence, you don't understand, you only know how to analyze one way, etc etc. Very condescending and if the proponents want people to buy into it, it's really not the way to go about it. Think of it as getting more bees with honey approach.

Honestly, I would look at it the same way many people see Kobe and his shot selection. He is so sure his way is the best way and refuses to alter his approach but it's not the best way for everyone to get on board and help with the process. It's inefficient, but I think everyone knows it's hard to change an approach that has been successful.

I do recognize the information, and I do recognize that it does have value, and many times it does represent what it claims to do. But I am on penbeasts side in this argument,

I despise RAPM. It's got no melody, just a beat. The lyrics are frequently shallow and inane,even by the normal standards of pop music. And, so many of its icons are real, pretend, or wannabe criminals. Hopefully the next major movement in music will move back to a more intelligent format.

mmmm, yeah

RAPM is like current hip-hop/rap, it's a very shallow and hollow, and it doesn't speak to my soul lol.......

[Keller61]

You can't break down RAPM like you can with other stats. You can't tell me what a player's RAPM is after the all-star break, what his RAPM is in the 4th quarter, what his RAPM is against top 10 defenses, what his RAPM is when a certain teammate is on/off the floor, etc. There is no room for analysis and interpretation. You just have to accept the noisy, biased output for what it is.

[fuzzy_dunlop]

^
You sort of can break it down (into 4 factors on each end of the floor). Also you can indeed "break down" other stats in such a way, but often the results are meaningless due to sample size/selection bias.

[Krodis]

I'm really pro-RAPM, but it has huge limitations. As to why I like it, I think Dr. Spaceman covered most of it: it actually tries to measure what I think is the most important thing to measure on a basketball court in terms of player evaluation: impact. Now of course, impact isn't the ONLY way to evaluate a player and in some cases may be off base. For example, if a player is in a poor situation, or if they're a young developing player, you may want to look at other things than their current impact.

And, of course, on top of that, the stat itself is flawed. It's noisy even in the best of cases, and wildly slow on small sample sizes. Also, it's... unreliable in terms of when and where it comes out. RAPM, especially as of late, has been rather irregularly updated.

Now, despite all those issues, I still find it valuable, partially because of the first point, and partially because of the lack of good defensive impact stats other than +/- ones. While the box score can certainly be deceiving for offensive players, it's practically useless for defensive ones, and while there are promising things coming out from things like SportsVU, RAPM to me is still the only real meaningful defensive stat.

I mean, ultimately, the box score has flaws, and while ideally the solution would be to watch every game completely objectively and memorized every single second of it, that's obviously not possible. So I like RAPM, even if it's probably a bit too unreliable to prove any singular point.

[NyCeEvO]

While I don't have the time to fully understand all of the components used to compute RAPM, I get the gist of it and I like it. It agrees with the eye test for me for a lot of players. I use RAPM to evaluate past production, not to predict the future or evaluate current production. I think it's too noisy for that.

To reiterate what others have said, there's no single stat that truly captures a player's total impact and thus RAPM ought to be used along with other qualitative and quantitative data. If I was forced to choose one stat to use to rank players, I would choose RAPM because I think it does a best job of giving a numerical value representative of a player's impact on a team WITHIN their specific role.

Here is an analogy:
If I was only allowed to use one tool to build a house (i.e. evaluate player impact), I'd choose the "RAPM" tool because (I believe) in comparison with any other single tool (e.g. PER, BPM, RAPM), I think it helps me to complete my task better than any other tool.

I'm not saying I can build a perfect house with the RAPM tool. In fact, I'd even admit that just using the RAPM tool alone would leave you with a pretty crappy house. However, I believe that I'd have a less crappy house (or have a better evaluation of a player) using just the RAPM tool to build a house than just using the PER tool, BPM, VORP, etc.



I have very little to add to what the pro-RAPM crowd has already stated. However, there is one point I've seen the anti-RAPM crowd state but I don't think the pro-RAPM has addressed it succinctly and clearly enough.

(If I understand it correctly) RAPM is most accurate for players who have played in the same system and have had similar roles on a team for at least a few years prior to the season one is getting RAPM data for.

Also, RAPM is describing how much impact a particular player has within their defined role on the team.

When we see Chris Anderson with a higher total RAPM than Dwyane Wade in 2012-13, people think it's saying Chris Anderson is better than Dwyane Wade and it's not saying that at all. (http://stats-for-the-nba.appspot.com/ratings/2013.html)

What it's saying is that Chris Anderson provided tremendous defensive impact in the 14.9mpg he played in the backup C spot for a pace-and-space Heat team. If you extended his minutes and asked him to do more than what he's good at (e.g. shooting jumpers, passing the ball, etc.), his RAPM would definitely go down.
However, given Anderson's role as a near 15mpg backup defensive big man on an uptempo team, you'd be hard pressed to find other big men who'd be able to provide as much impact as Anderson did.