2017-18 RAPM/RPM/etc. Thread

[Dr Spaceman]

I think there's some weird stuff going on with RPM this year. It looks more constrained at the top and bottom (pulled towards 0) compared to previous seasons.

Anybody?

I don’t take RPM results seriously until we start getting really close to the playoffs. Well, to the extent that I ever take RPM seriously, which these days is not all that much.

It’s telling to me that JE will release RPM so early into the season but withholds actual RAPM until the last second. That, combined with certain results, leads me to the conclusion that what we’re getting right now is honestly just a box score metric with lineup data on the side.

[TheSuzerain]

I think there's some weird stuff going on with RPM this year. It looks more constrained at the top and bottom (pulled towards 0) compared to previous seasons.

Anybody?

I don’t take RPM results seriously until we start getting really close to the playoffs. Well, to the extent that I ever take RPM seriously, which these days is not all that much.

It’s telling to me that JE will release RPM so early into the season but withholds actual RAPM until the last second. That, combined with certain results, leads me to the conclusion that what we’re getting right now is honestly just a box score metric with lineup data on the side.

Here's RPM's Max, Min, and Std Dev by year:

Code: Select all

 
Year   Max   Min   SD    
2014   9.08   -8.44  2.98
2015   9.34   -6.87  2.94
2016   9.79   -6.27  2.77
2017   8.42   -5.69  2.57
2018   6.52   -5.4    2.19


Are they changing something with the calculation? Wouldn't be the first time.

[Dr Spaceman]

PPG is automatically giving cretit just for shooting, even though you could be the worst scorer ever. Oh noes.

Not even close to the same thing.

it actually kind of is.

But let's ignore that since its bothering you. Don't these formulas usually give you credit for attempting 3 pt shots even if you aren't good at making them? With the idea that just taking 3's helps spacing? If we are okay with that notion, we should be okay with the height for defense one.


I actually think there are too many "subjective" decisions being made and then selling the end product as "objective", but I don't really want to be labeled the anti +/- guy despite my continue belief that none of these versions do nearly as good a job isolating individual player impact as they claim they do.

I agree here to a large extent but I want to make a couple points:

1. Correct me if I’m wrong, but I don’t think “pure objectivity” was ever the main selling point of the RAPM family. To me, the idea is “we have all of these ways to measure what a player is doing on the floor, but none of them try to measure the thing we’re actually concerned with as analysts. Let’s at least try”. Obviously RAPM is very very imperfect and should only be used as one slice of the analysis pie. But it’s value comes from the fact that it at least aims higher than any other stat ever did, in taking a stab at which players actually affect winning. That’s a great thing, because of course that’s the goal of team sports.

2. If subjectivity here bothers you, don’t ever sit in as a science journal article is being edited . The truth is all of science is subjective. The whole point is to find things that are relatively consistent about the way the world works, and then use those truths to make a statement about the world- one that can be tested. Every journal article you’ve ever read has 2 sections: 1. “Here are the results” I.e. pure numbers that nobody cares about “ 2. Here is what we think the results mean, and here’s how other scientists can prove us right or wrong

I personally don’t take most stats all that seriously anymore. I take RAPM for what it is, flawed but with the right intentions. My own personal method is sloppy and imperfect as well, and I can get it wrong to a much greater degree than someone who is more stat-focused. But that’s a judgement call, because my method gets me closer to where I want to actually be - determining which players actually move the needle in helping their teams win.

In other words, aim for the higher target, even if sometimes you miss by a lot.

[bondom34]

I think (hope?) nobody takes RPM/RAPM as gospel, or some ultimate player ranking. I get that feeling at times, and I don't know if I'm lumped into that by my comments itt, but it wasn't ever intended for that and shouldn't be used as such. It, like anything else to me, is a part of what you can use as information in evaluating players. If someone is particularly high/low and seems out of place, its more a flag of "What is going right or wrong here?" for said player.

[mikejames23]

LeBron this year reminds me of 2011 Kobe. The raw numbers hadn't necessarily changed, but the aging had started to show in the +/- reports. If the East were a little stronger, I would confidently say this is the year LeBron doesn't make the finals... as of now I don't know.

[Texas Chuck]

2. If subjectivity here bothers you, don’t ever sit in as a science journal article is being edited . The truth is all of science is subjective. The whole point is to find things that are relatively consistent about the way the world works, and then use those truths to make a statement about the world- one that can be tested. Every journal article you’ve ever read has 2 sections: 1. “Here are the results” I.e. pure numbers that nobody cares about “ 2. Here is what we think the results mean, and here’s how other scientists can prove us right or wrong

Subjectivity doesn't bother me one iota. Subjectivity wrapped up and sold as objective analysis bothers me a lot.

[dhsilv2]

I think there's some weird stuff going on with RPM this year. It looks more constrained at the top and bottom (pulled towards 0) compared to previous seasons.

Anybody?

I don’t take RPM results seriously until we start getting really close to the playoffs. Well, to the extent that I ever take RPM seriously, which these days is not all that much.

It’s telling to me that JE will release RPM so early into the season but withholds actual RAPM until the last second. That, combined with certain results, leads me to the conclusion that what we’re getting right now is honestly just a box score metric with lineup data on the side.

Here's RPM's Max, Min, and Std Dev by year:

Code: Select all

 
Year   Max   Min   SD    
2014   9.08   -8.44  2.98
2015   9.34   -6.87  2.94
2016   9.79   -6.27  2.77
2017   8.42   -5.69  2.57
2018   6.52   -5.4    2.19


Are they changing something with the calculation? Wouldn't be the first time.

I've noticed this as well. They have to be messing with the scaling, this wouldn't be new for espn to do and then not retroactively fix. Hollinger's VA stuff makes no sense if you look at the early years of it, and that's one where the formula is even published and clearly wasn't being used.

[dhsilv2]

Not even close to the same thing.

it actually kind of is.

But let's ignore that since its bothering you. Don't these formulas usually give you credit for attempting 3 pt shots even if you aren't good at making them? With the idea that just taking 3's helps spacing? If we are okay with that notion, we should be okay with the height for defense one.


I actually think there are too many "subjective" decisions being made and then selling the end product as "objective", but I don't really want to be labeled the anti +/- guy despite my continue belief that none of these versions do nearly as good a job isolating individual player impact as they claim they do.

If it's giving credit for taking and missing 3's, that's just as bad. I'm not okay with anything like that.

Also, it's not the same. PPG doesn't give credit for just taking shots, because they still actually have to go in, and PPG isn't a measure of impact, it's just how many points you scored. This is trying to assign how much lift you give a team, and gives you bonus points for a physical trait that may or may not have any impact at all for a particular player, and isn't even a correct measure for MOST NBA players.

A prior used for any player getting decent minutes so going to be so small that it wouldn't even show up in rounding. If they don't publish out 3 decimals why do you care?

[Jaivl]

Wouldn't make much sense to publish 3 decimals with standard errors that are always gonna be bigger than 1.00.

[dhsilv2]

Wouldn't make much sense to publish 3 decimals with standard errors that are always gonna be bigger than 1.00.

I'd love to see some data on what the error level is on these metrics. I'd venture to guess it's somewhat large relative to the scores and would give us a better way to look at the data. Instead of so and so was ahead, we'd be discussing groupings of players that RAPM/RPM say are within range of each other. It would really elevate the conversations.

[Jaivl]

I'd love to see some data on what the error level is on these metrics. I'd venture to guess it's somewhat large relative to the scores and would give us a better way to look at the data.

I remember APM had absurd errors, bigger than the signal itself. Single year RAPM had avg sqrt errors* around ~2.0 or so. https://web.archive.org/web/20150206043736/http://stats-for-the-nba.appspot.com/ratings/2014.html

* square root of the mean squared error

Instead of so and so was ahead, we'd be discussing groupings of players that RAPM/RPM say are within range of each other. It would really elevate the conversations.

Aren't we already doing that?

[dhsilv2]

I'd love to see some data on what the error level is on these metrics. I'd venture to guess it's somewhat large relative to the scores and would give us a better way to look at the data.

I remember APM had absurd errors, bigger than the signal itself. Single year RAPM had avg sqrt errors* around ~2.0 or so. https://web.archive.org/web/20150206043736/http://stats-for-the-nba.appspot.com/ratings/2014.html

* square root of the mean squared error

Instead of so and so was ahead, we'd be discussing groupings of players that RAPM/RPM say are within range of each other. It would really elevate the conversations.

Aren't we already doing that?

Wow would be nice to have that with RPM.

I think sometimes people use the grouping method, butif you look at how large those errors are, it makes far bigger groupings.

[Dr Positivity]

So no one's going to comment on the huge discrepancies between RAPM and RPM?

Fine. I will. I think Engelman has rigged RPM to give an answer that the fans want to see- ie, that the most popular players have similar ratings. The reality is better reflected by the Raw numbers and RAPM- That Curry is terrorizing the league, Harden is overrated and James is extremely overrated this year.

RAPM-
Curry- 1st by a long ways.
Harden- 53rd.
James- 78th.

Raw On/Off- Curry is way ahead of Harden and obliterating James.

RPM- Top players ranked close to the same.

I thought earlier in the season boxscores are more important than they end up being later. Thought we aren't that early in the season.

[Dr Positivity]

If it's giving credit for taking and missing 3's, that's just as bad. I'm not okay with anything like that.

Yeah but every time you take a 3 and miss, your TS% goes down. So if the decrease in value because of shooting % is more impactful than the positive increase of adding a 3 FGA attempt, then taking a 3 and missing still hurts you on the whole every time it happens

[Dr Positivity]

Height isn't the boxscore. That's what I'm saying, you don't know what's really used for a fact. Is age a factor? I know tons of statistics of this sort include age in the calculation but can I say RPM doesn't? Not really. And as far as predictive power goes all of these statistics have strong predictive power so that's a moot point. Wins produced is a generally panned statistic but did you know it has the strongest predictive power of all of these stats? It has a 95% correlation to wins (based on a 40 year sample). That's generally what happens when you work backwards. Adding to that RPM is meant to predict future level of play. A +8 RPM for Curry and a +5 RPM for Kyle Lowry (random example) isn't saying Curry played like a +8 and Lowry like a +5. It's saying that's the level we can expect them to play at in the future. Honestly RPM is just a completely misused statistic even if we say it does hold value.

WP is not a good predicting stat, bad predictions and repeated attempts to predict the upcoming season and falling flat on their face, is basically what showed people it doesn't work. When their predictions are compared to APBR board contest entries, Vegas, etc. it always gets beat badly, whereas RPM is supported by same contests

[Clyde Frazier]

Gotta chime in here quickly as I didn't realize height was used to calculate RPM. While I agree with the notion that on average, bigs have a greater defensive impact than smalls, listed heights in the NBA are wildly inaccurate.

Players openly have admitted that their listed height is more or less what they want it to be within reason (although with Durant specifically, I wouldn't even call it that). It's a bizarre part of the NBA that basically just gets ignored.

Now, if the NBA performed yearly measurements league wide similar to the combine, using that data would make sense. Otherwise, I consider a lot of noise in using current data and an odd decision.

[clyde21]

Gotta chime in here quickly as I didn't realize height was used to calculate RPM. While I agree with the notion that on average, bigs have a greater defensive impact than smalls, listed heights in the NBA are wildly inaccurate.

Players openly have admitted that their listed height is more or less what they want it to be within reason (although with Durant specifically, I wouldn't even call it that). It's a bizarre part of the NBA that basically just gets ignored.

Now, if the NBA performed yearly measurements league wide similar to the combine, using that data would make sense. Otherwise, I consider a lot of noise in using current data and an odd decision.

Does it matter if the data for all of the players is getting skewed somewhat in the same direction and in the same capacity? If everyone is +1" added to their measurements, then it doesn't really affect the output of the data.

[Clyde Frazier]

Gotta chime in here quickly as I didn't realize height was used to calculate RPM. While I agree with the notion that on average, bigs have a greater defensive impact than smalls, listed heights in the NBA are wildly inaccurate.

Players openly have admitted that their listed height is more or less what they want it to be within reason (although with Durant specifically, I wouldn't even call it that). It's a bizarre part of the NBA that basically just gets ignored.

Now, if the NBA performed yearly measurements league wide similar to the combine, using that data would make sense. Otherwise, I consider a lot of noise in using current data and an odd decision.

Does it matter if the data for all of the players is getting skewed somewhat in the same direction and in the same capacity? If everyone is +1" added to their measurements, then it doesn't really affect the output of the data.

I don't think it's nearly that consistent. Players want to be listed at different heights for different reasons. My issue here ultimately is using data that's knowingly inaccurate. I'd just pass on that completely.

[eminence]

Hmm, don't really see the problem here. RPM's strength is as a predictive stat. Sure I may not understand why exactly height works to increase predictive power, but it does, so it's used. If you wanted to use it for descriptive purposes yeah, it might not be the best (just use RAPM if you want a plus/minus family stat in that case).

Seems like a case of mis-using RPM (xRAPM), which is understandable since both J.E. and Ilardi are a bit guilty of marketing their stat as a bit more than it really is.

[dhsilv2]

Gotta chime in here quickly as I didn't realize height was used to calculate RPM. While I agree with the notion that on average, bigs have a greater defensive impact than smalls, listed heights in the NBA are wildly inaccurate.

Players openly have admitted that their listed height is more or less what they want it to be within reason (although with Durant specifically, I wouldn't even call it that). It's a bizarre part of the NBA that basically just gets ignored.

Now, if the NBA performed yearly measurements league wide similar to the combine, using that data would make sense. Otherwise, I consider a lot of noise in using current data and an odd decision.

Does it matter if the data for all of the players is getting skewed somewhat in the same direction and in the same capacity? If everyone is +1" added to their measurements, then it doesn't really affect the output of the data.

I don't think it's nearly that consistent. Players want to be listed at different heights for different reasons. My issue here ultimately is using data that's knowingly inaccurate. I'd just pass on that completely.

All of these regression based stats have a LARGE error rate. If a prior improves the results over the whole but misses some players, so what? The error rates in these stats are far more impactful than a small piece of a the prior.