RealGM

I've never really found a stat that accurately measures how good or bad a player is at defense. Recently, I've been looking at RAPM (the site I found only goes back to 2001 I think) but is this really the best way to rank defense? Defensive Rating (DRtg) seems to work well for ranking team defense but it's awful for player defense.

I lean toward RAPM as the "best" (or at times "least bad") defensive stat.

I particularly dislike 1 on 1 based defensive stats (things like opponents PER). IMO they really only show the team defensive strategy (and sometimes how much a player sticks to his man at the expense of team defense).

So going by this:

http://stats-for-the-nba.appspot.com/ranking12_no_prior

Would it be accurate to say players like Dirk, Tony Parker, Vince Carter, etc were better on defense than LeBron this season? RAPM doesn't seem to be as effective in this lockout season but what's the reasoning behind those players being ranked ahead of of guys you would normally think had much more defensive impact (KG is another example)?

I think that's mostly showing the biggest weakness of RAPM - noise.

The shorter season made sample sizes smaller, and there were bigger differences than usual if you caught a team rested and healthy or tired and gimpy (or playing to get their coach fired). Large enough sample sizes work those effects out, but one shorter season wasn't enough.

The counter is to go to larger/longer sample sizes, though that adds drift issues (players getting better or worse over time).

Tony Parker played the vast majority of his minutes with Timmy--only like 400 minutes without him--and they have very similar defensive on/off numbers. The credit should probably go primarily to Timmy for that defense, but the minutes are too low to filter that out. Even if those low minutes do show Timmy should get the credit, RAPM doesn't "cheat" and extrapolate (that's the whole point of RAPM vs APM), so they share the credit. Timmy is +1.8 and TP is +1.3 on defense. With more minutes and lineup variety, these numbers would probably spread out more to give Timmy the bulk of the credit.

Same story with LeBron. He played a ton of his team's minutes, so there's just not much left to say what the team was like without him. He's got a solid defensive backup (Battier), the other starters have similar defensive on/off numbers to LeBron's, the second unit was pretty solid defensively in general, and a significant % of the non-LeBron time was probably junk minutes against junk lineups anyway. The Heat's most-used players are all in the same approximate range for defensive RAPM:
LeBron: +0.6
Bosh: +0.4
Chalmers: +0.9
Wade: +0.7
Haslem: +0.6

Chalmers probably gets the most credit here because he has the worst backup in Norris Cole. So again, with more minutes and more lineup variety, these numbers would shift.

I'll echo what others said:

APM-based stats are the one's that are not systematically biased for/against certain types of players and that's a really big deal. On the other hand they also have major issues with sample size noise. If you're looking for one stat to use and be confident in, (R)APM may or may not be good enough, but you can't really have much confidence in other stats either.

Really what you need to do is understand all the stats, and use them all together holistically. But if you want a short cut, I prefer multi-year RAPM studies when evaluating outlier players. It is Garnett's huge edge on this front that really makes me unable to side with other current defenders above him.

Doctor MJ wrote:APM-based stats are the one's that are not systematically biased for/against certain types of players and that's a really big deal. On the other hand they also have major issues with sample size noise. If you're looking for one stat to use and be confident in, (R)APM may or may not be good enough, but you can't really have much confidence in other stats either.

The issue with sample size is much overblown, especially with RAPM. There is variance in the player performance, but at about 30 to 40 games the resulting standard deviation is pretty much equal to looking at 82 games instead. With 66 games there is enough to have confidence in those numbers, and with the prior informed version you get MUCH better results than with any kind of multiyear study.

Doctor MJ wrote:Really what you need to do is understand all the stats, and use them all together holistically. But if you want a short cut, I prefer multi-year RAPM studies when evaluating outlier players. It is Garnett's huge edge on this front that really makes me unable to side with other current defenders above him.

You still haven't understood the issue with multiyear studies, haven't you? Look at the variance within a certain sample and then check out how that looks with a bigger sample. For teams we are basically at the same variance after 10 games as we have within a 82 game sample. Yeah, there are outliers in which a team has a 25 game sample of playing much better or something like that, but the difference is hardly that huge and most times explainable by health or injury issues to their opponents.


The issue with RAPM is that it doesn't know whether the better defense comes from the defensive setting (getting back on defense instead of crashing the board is a huge factor, providing offensive strength in order to give the chance to surround a player with better defensive players is another) or from individual defense. If you want to know how good a player is in terms of individual defense check out his Synergy numbers and his opponents performance level. How much effect that has can be seen by the defensive RAPM to a certain degree and for sure better than individual boxscore based DRtg numbers can do it.
Nowitzki is a pretty good post defender, but his biggest contribution to the team defense is that he prevents fastbreak opportunities by lowering the team turnovers and getting back on defense. He also has an offensive skillset which makes it possible to put better defensive players next to him which can entirely focus on defense. Nowitzki is also pretty good within the zone defense, getting even out on the perimeter and restricting penetrations with his size. That worked best when they had Chandler behind him last season, because basically no player tried to drive past Nowitzki just to run into Chandler. That's why the Mavericks are playing clearly better defensively when Nowitzki is playing. That doesn't make Nowitzki a better individual defensive player in terms of defending 1on1 or helping than LeBron James for example, but overall that clearly helps defensively.

For Tony Parker Los Soles gave a pretty good explanation. But again, that is also due to Parker's skillset making it possible to play him with certain players. We also see the same phenomena with the Spurs as we see with the Mavericks, they rather go back defensively than trying to get the offensive rebound. The same story for the Celtics. I said that for years that going back defensively helps more than trying to get each offensive rebound. The numbers are pretty clear in that aspect.

And while we might want to know how great of a defender a certain player is the whole point of the game is to win as a team. Thus, fitting skillsets and helping a team significant on offense can be the key to play better defense as well. RAPM is just noticing the overall effect, not how the effect is happing.

mysticbb wrote:You still haven't understood the issue with multiyear studies, haven't you? Look at the variance within a certain sample and then check out how that looks with a bigger sample. For teams we are basically at the same variance after 10 games as we have within a 82 game sample. Yeah, there are outliers in which a team has a 25 game sample of playing much better or something like that, but the difference is hardly that huge and most times explainable by health or injury issues to their opponents.

The issue with RAPM is that it doesn't know whether the better defense comes from the defensive setting (getting back on defense instead of crashing the board is a huge factor, providing offensive strength in order to give the chance to surround a player with better defensive players is another) or from individual defense. If you want to know how good a player is in terms of individual defense check out his Synergy numbers and his opponents performance level. How much effect that has can be seen by the defensive RAPM to a certain degree and for sure better than individual boxscore based DRtg numbers can do it.
Nowitzki is a pretty good post defender, but his biggest contribution to the team defense is that he prevents fastbreak opportunities by lowering the team turnovers and getting back on defense. He also has an offensive skillset which makes it possible to put better defensive players next to him which can entirely focus on defense. Nowitzki is also pretty good within the zone defense, getting even out on the perimeter and restricting penetrations with his size. That worked best when they had Chandler behind him last season, because basically no player tried to drive past Nowitzki just to run into Chandler. That's why the Mavericks are playing clearly better defensively when Nowitzki is playing. That doesn't make Nowitzki a better individual defensive player in terms of defending 1on1 or helping than LeBron James for example, but overall that clearly helps defensively.

For Tony Parker Los Soles gave a pretty good explanation. But again, that is also due to Parker's skillset making it possible to play him with certain players. We also see the same phenomena with the Spurs as we see with the Mavericks, they rather go back defensively than trying to get the offensive rebound. The same story for the Celtics. I said that for years that going back defensively helps more than trying to get each offensive rebound. The numbers are pretty clear in that aspect.

And while we might want to know how great of a defender a certain player is the whole point of the game is to win as a team. Thus, fitting skillsets and helping a team significant on offense can be the key to play better defense as well. RAPM is just noticing the overall effect, not how the effect is happing.

mystic, I'll admit it is a challenge following you sometimes. I enjoy having conversations from you, and learning some things from you. So, it is possible that I'm just "not getting it".

Sounds like what you're saying about Dirk and his D, is that his ridiculous numbers in prior-based RAPM right now are not due to sample size but due to fit. As in: We should not look at this as a statement of Dirk's generalizable abilities as a defender, but on Dallas, were the team to not age, we should expect Dirk's defensive RAPM to keep looking DPOY worthy.

That's a cohesive worldview.

Consider my perspective though as someone who views outliers skeptically. When I see that pure APM has a crazy number and that RAPM has something more reasonable, I take heart in RAPM. This happens a good amount too.

At the same time though, sometimes RAPM comes off with the crazier numbers. Your preferred method has Dirk listed as easily the most "impactful" player in the league right now. Meanwhile the other +/- metrics don't. Stuff like this clouds my faith in 1-year prior RAPM studies. Not that I'd refuse to use the in conjunction with other things, but in terms of them being truly reliable enough that I don't have to worry about sample size, yeah, makes me nervous.

And I need to make clear I'm not saying this simply for the above reason. The thing is that I know that RAPM gets made by, to some degree, de-emphasizing our trust on the actual results of play. I understand the reasons for this and why this is in general a good move for improving reliability. That improvement though is by no means a guarantee that every player's RAPM value will be more accurate than his corresponding APM value, just that in general it's a better estimate.

So a question: What does it mean then when a technique meant to make results come down to earth gives us an outlier value NOT seen in our ungrounded metrics?

I'm curious your thoughts. Obviously for me, it makes me wary. When we only see an outlier value when we essentially pump in fake data, how can that not make us question whether that data truly has a sufficient sample size?

Dirk's defensive RAPM has risen dramatically in the last two years. Did he actually improve that much? Doubt it, although it's very possible that he has gotten more crafty defensively. I think he's a solid overall defender: good combo of mobility and length, hard working, and intelligent. But I think the shift in his RAPM numbers is situational.

1) He plays a ton of minutes, so you just get a low sample size for what's happening when he's not playing. Dallas hasn't had a clear plan when he sits because they haven't had a clear 2nd string PF. So that means lots of hack units.
2) His backup options have been really terrible: Brian Cardinal, the artist formerly known as Lamar Odom, and Shawn Marion.
3) Marion is an intriguing situation: I don't think Marion is a poor defensive 4. He's undersized, so players can shoot over him at times, but I really don't think that's the issue. I think Dallas has been better defensively with Dirk at the 4 primarily because that allows Marion to be in at the three. This became a very serious problem when Caron Butler got injured last season, especially with Josh Howard gone: who do you play at the 3? When Dirk sits and Marion goes to the 4, Terry also comes in and suddenly you're small at the 2, 3, and 4. Or you play Peja at the 3, which doesn't help the defense any.

That whole convoluted mess is not something RAPM has the minutes/lineup variety to filter through.

Pick a random, scrappy, unskilled 2nd-unit 4 (like 80% of teams have) and insert him as Dirk's consistent backup, and I think Dirk's defensive RAPM numbers fall back to normal.

Doctor MJ wrote:Consider my perspective though as someone who views outliers skeptically. When I see that pure APM has a crazy number and that RAPM has something more reasonable, I take heart in RAPM. This happens a good amount too.

Beware the golden mean fallacy, however.

Doctor MJ wrote:Sounds like what you're saying about Dirk and his D, is that his ridiculous numbers in prior-based RAPM right now are not due to sample size but due to fit. As in: We should not look at this as a statement of Dirk's generalizable abilities as a defender, but on Dallas, were the team to not age, we should expect Dirk's defensive RAPM to keep looking DPOY worthy.

Indeed, but that is not just the case for Nowitzki, but for all involved players. Los Soles explained some of the stuff, but he looked at that from a rather negative perspective. The thing is Dirk Nowitzki playing means playing Shawn Marion as SF, which helps defensively in comparison to playing Marion at PF. Nowitzki's skillset allows that move, if he would be different, Marion would probably play the PF defensively too. But we have also the effect that a more efficient offense can lead to a more efficient defense. The most efficient scoring opportunities are coming after a steal, then defensive rebound and last is the inbound after a made basket. Additional to a big going back defensively is reducing fastbreak opportunities and lowers the efficiency in a possession after a defensive rebound. Thus, Nowitzki's defensive RAPM value is not just a result of fit, but also a result of a specific defensive setting (add the zone defense to it, which Nowitzki is used to since playing for the German national team).

To understand that point better, we might want to look into the situation with James and Wade. Neither player was actually worse last season or this season than in 2010 or 2009 and yet there +/- values are down. That can be explained by fit.

Doctor MJ wrote:Consider my perspective though as someone who views outliers skeptically. When I see that pure APM has a crazy number and that RAPM has something more reasonable, I take heart in RAPM. This happens a good amount too.

At the same time though, sometimes RAPM comes off with the crazier numbers. Your preferred method has Dirk listed as easily the most "impactful" player in the league right now. Meanwhile the other +/- metrics don't. Stuff like this clouds my faith in 1-year prior RAPM studies. Not that I'd refuse to use the in conjunction with other things, but in terms of them being truly reliable enough that I don't have to worry about sample size, yeah, makes me nervous.

RAPM is per se better than APM, there is no discussion about it. The math is pretty clear, the result in out of sample tests AND retrodiction tests is also completely clear. In the end it is similar to use eFG% instead of FG%, you wouldn't revert back to FG% only because you think the eFG% is crazy, wouldn't you?

Nowitzki is also way ahead of everyone else in 2yr APM:

http://basketballvalue.com/topplayers.p ... order=DESC
http://basketballvalue.com/topplayers.p ... =2011-2012

Nowitzki is also the best starting player in terms of no-prior informed RAPM for the 2011/12 season. His high prior-informed RAPM is not an outlier, it is actually seen in the other data too.

Doctor MJ wrote:And I need to make clear I'm not saying this simply for the above reason. The thing is that I know that RAPM gets made by, to some degree, de-emphasizing our trust on the actual results of play. I understand the reasons for this and why this is in general a good move for improving reliability. That improvement though is by no means a guarantee that every player's RAPM value will be more accurate than his corresponding APM value, just that in general it's a better estimate.

The reason for using such technique is the variance in the player performance. We can bascially use the baysian interpretation here, which means we assume a normal distributed performance with a overall mean of 0 in order to understand that concept. That indeed helps to work better at predictor for ALL players.

Doctor MJ wrote:So a question: What does it mean then when a technique meant to make results come down to earth gives us an outlier value NOT seen in our ungrounded metrics?

I'm curious your thoughts. Obviously for me, it makes me wary. When we only see an outlier value when we essentially pump in fake data, how can that not make us question whether that data truly has a sufficient sample size?

The 2yr APM is producing an even higher outlier, if you want to call it like this.

And to understand the stuff happing with APM, we can take a look at the Lakers with Gasol and Bryant. Do you honestly think that Bryant was nearly 20 points worse last season than Gasol? Look at the RAPM values for both and they are making much more sense.

Doctor MJ wrote:At the same time though, sometimes RAPM comes off with the crazier numbers. Your preferred method has Dirk listed as easily the most "impactful" player in the league right now. Meanwhile the other +/- metrics don't.

Which ones don't? Two year APM from BV has him at #1 by far. He had two really good years in a row, especially 2010-2011, and so any multi-year APM system that gives more weight to recent performance is going to rate him really high. This is amplified by the fact that:

1) LeBron and Wade have been stealing each other's thunder for the past two seasons.
2) Until Shaq and Rasheed retired and the Celtics traded Perkins, they were pretty darn thick in the frontcourt. Garnett wasn't quite as crucial a commodity as usual.

That leaves Dirk, Nash, and Chris Paul is the primary candidates for APM supremacy in recent multi-year studies, and it's pretty hard to argue with Dirk. Ultimately APM is about great basketball, which the Mavs obviously played with Dirk in 2010-2011. Kidd-Terry-Marion-Dirk-Chandler played some of the best basketball we've ever seen. And Dirk was the ONLY absolutely essential piece.



That's something I like to call "beating the sh*t out of everybody." That's with or without Kidd, with or without Terry, with or without Marion, and with or without Chandler. But always with Dirk.

mysticbb wrote:RAPM is per se better than APM, there is no discussion about it. The math is pretty clear, the result in out of sample tests AND retrodiction tests is also completely clear. In the end it is similar to use eFG% instead of FG%, you wouldn't revert back to FG% only because you think the eFG% is crazy, wouldn't you?

Nowitzki is also way ahead of everyone else in 2yr APM:

http://basketballvalue.com/topplayers.p ... order=DESC
http://basketballvalue.com/topplayers.p ... =2011-2012

Nowitzki is also the best starting player in terms of no-prior informed RAPM for the 2011/12 season. His high prior-informed RAPM is not an outlier, it is actually seen in the other data too.

I feel the conversation drifting here. I realize I made a long post, but if I've given the impression that I favor APM over RAPM generally, let me clarify: I don't.

That said: I do object to making the eFG to FG analogy and I find myself frustrated that you're going there even though it's in line with how you've been talking. eFG is simply counting more accurately. Take any nugget of data of FG and it is guaranteed to be equal or better measured by eFG. How can you possibly say that about RAPM? You KNOW there is an error involved with it still so how could you ever be certain that any particular bit of results is superior to APM?

I understand it's superiority in general, but that's not the same as guaranteed granular dominance.

Also we were talking about whether the prior-method single season RAPM has sample size adequate than variability is never an issue, yes? Just want to make sure that stays in the conversation.

mysticbb wrote:The 2yr APM is producing an even higher outlier, if you want to call it like this.

And to understand the stuff happing with APM, we can take a look at the Lakers with Gasol and Bryant. Do you honestly think that Bryant was nearly 20 points worse last season than Gasol? Look at the RAPM values for both and they are making much more sense.

I understand why you wrote this response, but to me this just seems to be pointing out how likely it is that the Dirk's huge values from '10-11 are playing a major role in his '11-12 prior method RAPM, which one of my fears when I look at the study.

In my ideal world of stats, I'd have perfectly discrete bits of data separating between seasons. A study that comes right out and says it's just combining seasons together has issues, but it's a more intuitive bit of data to make sense of than the stuff you champion. Dirk's not paying as good this year as last, and studies like these let what he did last year influence the "this year" stat in ways too subtle for me to be able to separate. It just makes me uncertain how to use the data.

Again, this is not my way of saying "So I prefer APM over RAPM" but rather my way of saying I'm not comfortable jettisoning multi-year studies entirely. I'll keep using them, and I'll use the ones you like, and I'll use other things.

Do you understand? I don't need to be convinced to use prior-based studies, I need to be convinced to use them exclusively, which will come not be knocking the other methods, but by addressing my concerns.

Los Soles wrote:

Doctor MJ wrote:At the same time though, sometimes RAPM comes off with the crazier numbers. Your preferred method has Dirk listed as easily the most "impactful" player in the league right now. Meanwhile the other +/- metrics don't.

Which ones don't? Two year APM from BV has him at #1 by far. He had two really good years in a row, especially 2010-2011...

Okay, when I say "right now" I was taking it as a given that the "especially 2010-2011" component was obviously what I was trying to exclude. I've yet to hear anyone disagree with the notion that Dirk didn't look as good this year as he did last, so when "this year" studies that use priors still favor Dirk by large amounts, my concern is that I'm not actually sure how much of what I'm getting is due to Dirk's work this year.

Now as I say this, I realize that Engelmann has a non-prior rating as well. This is nice to have, and part of the stuff I look at, but it seems to me that mystic is saying I should be so happy with the prior study that I'll need none other. And that, is where I lack sufficient faith.

Doctor MJ wrote:That said: I do object to making the eFG to FG analogy and I find myself frustrated that you're going there even though it's in line with how you've been talking. eFG is simply counting more accurately. Take any nugget of data of FG and it is guaranteed to be equal or better measured by eFG. How can you possibly say that about RAPM? You KNOW there is an error involved with it still so how could you ever be certain that any particular bit of results is superior to APM?

I know I repeat now what I said many times before, but let me try it again: RAPM beats out APM easily in terms of out-of-sample tests AND retrodiction tests as well as explaining wins in hindsight. The result of RAPM has an all-around better explanatory and predictive power. No idea what else is needed in order to proclaim RAPM to be superior to APM.

Doctor MJ wrote:I understand it's superiority in general, but that's not the same as guaranteed granular dominance.

What else should that be?

Doctor MJ wrote:Also we were talking about whether the prior-method single season RAPM has sample size adequate than variability is never an issue, yes? Just want to make sure that stays in the conversation.

Sample size is not an issue in the way it is an issue with APM. APM is producing incredible stupid and wrong results even after more than 100 games played. Look at Gasol's and Bryant's 2yr APM, there were still nearly 9 points apart while prior informed RAPM had them within 1pt. What is more reasonable? That Pau Gasol is some kind of MVP caliber player while Bryant is mediocre or that both are rather equal while neither is among MVP caliber players?
For sure, if a player hits a hot streak for 20 or 25 games, RAPM will see him better than he is while in reality the normal variance allows such things. But eliminating the overfitting issue is HUGE.

Doctor MJ wrote:I understand why you wrote this response, but to me this just seems to be pointing out how likely it is that the Dirk's huge values from '10-11 are playing a major role in his '11-12 prior method RAPM, which one of my fears when I look at the study.

Nowitzki has the 3rd highest no-prior informed RAPM behind Gibson and Bonner; both are role players excelling in a certain role. Nowitzki is the best starter in no-prior informed RAPM. He started out not that well and RAPM captured that with having him barely above average while his prior-informed RAPM went back to be 3rd overall behind Chris Paul and Manu Ginobili. When Nowitzki came back and played stronger again, all of his numbers went up. Nowitzki had 25.1/7.4 on 58 TS% per 36 min over the last 43 games of the season. He showed all signs of being as impactful as last season during that stretch.

Doctor MJ wrote:In my ideal world of stats, I'd have perfectly discrete bits of data separating between seasons. A study that comes right out and says it's just combining seasons together has issues, but it's a more intuitive bit of data to make sense of than the stuff you champion. Dirk's not paying as good this year as last, and studies like these let what he did last year influence the "this year" stat in ways too subtle for me to be able to separate. It just makes me uncertain how to use the data.

How is the no-prior informed version of RAPM influenced by the data from last season? There is no last season data involved in that measurement, the prior is set to 0 for all players. And still Nowitzki beats out basically all players. The advantage of prior-informed comes into play when making predictions, because the out-of-sample test showed that prior-informed is just better as predictor.

Doctor MJ wrote:Again, this is not my way of saying "So I prefer APM over RAPM" but rather my way of saying I'm not comfortable jettisoning multi-year studies entirely. I'll keep using them, and I'll use the ones you like, and I'll use other things.

The issue with multi-year studies (including multi-year RAPM studies) is that we not only have the normal variance included for a player within a year, but also add another variance which is caused by the development curve of a player over his career. That is an issue which might be better understandable, if you think about the principle of superposition of waves. Imagine you have one wave with a rather short wavelength (player performance over a year) and another one with a longer wavelength (player performance over a career). Within a certain amount of time you will not see effects of the longer wave, but at a certain point the overlapping of both waves will cause constructive or destructive interference. What does that mean for our problem? Well, some players will see their longterm value being amplified and other will see theirs being reduced just by the nature of things. Exactly what I showed with the examples for Nash, Garnett, Nowitzki and their respective teammates. We could eliminate the effect by including development curves for each player, otherwise the increased sample size is not solving issues, but is introducing another one.

Doctor MJ wrote:Do you understand? I don't need to be convinced to use prior-based studies, I need to be convinced to use them exclusively, which will come not be knocking the other methods, but by addressing my concerns.

To be quite frankly: I don't even understand your concerns at all, especially when you are using APM data in order to justify things. It just looks like you want to use the stats which are best fitting for your preconception, while you are not using those which are best. I only use APM as a point of reference when the prior-informed RAPM isn't available or not as good due to the lack of sample size (2002 and 2003, because 2002 has only a few games included, which screws up the overall result and makes the prior for 2003 also not good at all.)

Los Soles wrote:That's something I like to call "beating the sh*t out of everybody." That's with or without Kidd, with or without Terry, with or without Marion, and with or without Chandler. But always with Dirk.

We can increase that by including this years lineups:



The first lineup without Nowitzki and playing at least 50 minutes is listed 14th for the Mavericks. One of the dumbest things Carlisle did in the series against OKC was not using West-Carter-Marion-Nowitzki-Haywood.

mysticbb wrote:

Doctor MJ wrote:That said: I do object to making the eFG to FG analogy and I find myself frustrated that you're going there even though it's in line with how you've been talking. eFG is simply counting more accurately. Take any nugget of data of FG and it is guaranteed to be equal or better measured by eFG. How can you possibly say that about RAPM? You KNOW there is an error involved with it still so how could you ever be certain that any particular bit of results is superior to APM?

I know I repeat now what I said many times before, but let me try it again: RAPM beats out APM easily in terms of out-of-sample tests AND retrodiction tests as well as explaining wins in hindsight. The result of RAPM has an all-around better explanatory and predictive power. No idea what else is needed in order to proclaim RAPM to be superior to APM.

Hmm I basically take it as a given that in anything technical here you're at least as knowledgeable as me, so I'm not sure why I'm talking past you here. Perhaps I'm just not using the standard vocabulary?

When I say "granular dominance", as in Method A dominating Method B I'm talking about the confidence that no matter what bit of data we have, there is never a time where Method B outperforms Method A. Meaning there cannot possibly be even one player whose APM is more accurate than his RAPM. I just don't know how you can possibly see this as "proved". You know there is error in RAPM unlike something like eFG. Doesn't that error have to mean that with any competing analysis that has error overlap there is the possibility that in that case the other metric performed more accurately?

And again: This is not about which is the overall superior metric. I'm talking about the assumption that the on-average better metric is superior in every single granule of data.

mysticbb wrote:Sample size is not an issue in the way it is an issue with APM. APM is producing incredible stupid and wrong results even after more than 100 games played. Look at Gasol's and Bryant's 2yr APM, there were still nearly 9 points apart while prior informed RAPM had them within 1pt. What is more reasonable? That Pau Gasol is some kind of MVP caliber player while Bryant is mediocre or that both are rather equal while neither is among MVP caliber players?
For sure, if a player hits a hot streak for 20 or 25 games, RAPM will see him better than he is while in reality the normal variance allows such things. But eliminating the overfitting issue is HUGE.

I certainly understand that RAPM decreases reliance on sample size, my issue is that I just don't see any basis for saying it's no longer an issue. Perhaps you could go into specifics as to what exactly made you say "Okay, now that's good enough."

mysticbb wrote:How is the no-prior informed version of RAPM influenced by the data from last season? There is no last season data involved in that measurement, the prior is set to 0 for all players. And still Nowitzki beats out basically all players. The advantage of prior-informed comes into play when making predictions, because the out-of-sample test showed that prior-informed is just better as predictor.

Obviously I'm not talking about the no-prior model when I talk about concerns relating to the prior. C'mon now. I like the no-prior and I use the no-prior, I just don't have faith that it totally deals with sample size.

mysticbb wrote:The issue with multi-year studies (including multi-year RAPM studies) is that we not only have the normal variance included for a player within a year, but also add another variance which is caused by the development curve of a player over his career. That is an issue which might be better understandable, if you think about the principle of superposition of waves. Imagine you have one wave with a rather short wavelength (player performance over a year) and another one with a longer wavelength (player performance over a career). Within a certain amount of time you will not see effects of the longer wave, but at a certain point the overlapping of both waves will cause constructive or destructive interference. What does that mean for our problem? Well, some players will see their longterm value being amplified and other will see theirs being reduced just by the nature of things. Exactly what I showed with the examples for Nash, Garnett, Nowitzki and their respective teammates. We could eliminate the effect by including development curves for each player, otherwise the increased sample size is not solving issues, but is introducing another one.

I get the issues with APM. What I need is more confidence in RAPM. Not simply "RAPM > APM" confidence, but confidence that I can use short time spans.

mysticbb wrote:To be quite frankly: I don't even understand your concerns at all, especially when you are using APM data in order to justify things. It just looks like you want to use the stats which are best fitting for your preconception, while you are not using those which are best. I only use APM as a point of reference when the prior-informed RAPM isn't available or not as good due to the lack of sample size (2002 and 2003, because 2002 has only a few games included, which screws up the overall result and makes the prior for 2003 also not good at all.)

Well here's what I say: It's possible the resistance is based on wanting to believe that the weird values I see in RAPM (or APM for that matter) is sample-sized based because the alternative means that this data is farther from my perceptions of what the best players are and this discourages me.

I won't claim to be completely objective on the matter, but you keep talking as if I'm not using things that I keep telling you that I AM using. Meanwhile, I just don't see where you've ever really given the full argument for why RAPM stats are so good that they don't have some of the same issues APM does. You explicitly state RAPM doesn't have those issues, but how am I supposed to be convinced?

Doctor MJ wrote:Hmm I basically take it as a given that in anything technical here you're at least as knowledgeable as me, so I'm not sure why I'm talking past you here. Perhaps I'm just not using the standard vocabulary?

Oh, I fully understand what you want to say, the issue is that it seems as if you don't really understand the technical aspect here. There is a mathematical theorem which states: There is always a lambda for which the MSE of the ridge regression is smaller than the MSE of the OLS.
That is PROVEN! There is no discussion about it. We have a ill-posed problem and thus it is a given that ridge regression will produce a better result. There is NOTHING in the math which would indicate that in "some" cases the OLS would produce better results. AND we can ONLY take the WHOLE set of coefficients and NOT pick randomly some. It makes ZERO sense to select a few values from APM over the RAPM values only because you FEEL they are more accurate. In fact if we calculate the "standard error" for RAPM (which makes not much sense due to the introduced bias, but whatever) via bootstrap, we will see a value of in average 2.5 after half a season of data, a value of about 2 after 2/3 of a season and a value of about 1.5 after a full season played. Check out the average SE for 2yr APM and you will realise how much better ridge regression really is. After TWO full seasons played the SE for the players with the lowest SE is at about 2.3. In average we are ending up with about 3 as the SE for a dataset consisting of two full seasons. That is something we can see for RAPM after 25 to 30 games played. What else do you want to have? We have the theoretical aspect, the proven math, we have the results in out-of-sample tests, retrodiction and the results of the SE.

Doctor MJ wrote:When I say "granular dominance", as in Method A dominating Method B I'm talking about the confidence that no matter what bit of data we have, there is never a time where Method B outperforms Method A. Meaning there cannot possibly be even one player whose APM is more accurate than his RAPM. I just don't know how you can possibly see this as "proved". You know there is error in RAPM unlike something like eFG. Doesn't that error have to mean that with any competing analysis that has error overlap there is the possibility that in that case the other metric performed more accurately?

Honestly, your point makes no sense at all, because we don't have independent results for each player. The coefficients are depending on the values the other players are assigned to as well. You can either accept the whole result or not, but you can't just randomly pick out coefficients from APM and RAPM just because your feeling tells you they are somehow valid. The fact is the results of the RAPM are per se more valid than the results of APM.

Doctor MJ wrote:And again: This is not about which is the overall superior metric. I'm talking about the assumption that the on-average better metric is superior in every single granule of data.

I can imagine that using a APM value for a player instead of a RAPM value can even improve the prediction, but in average it will get worse. So, if you make an analysis in which you replace the RAPM value with the APM for a certrain player, I would be surprised, if you find many players for which the result of an out-of-sample test would be better.

Doctor MJ wrote:I certainly understand that RAPM decreases reliance on sample size, my issue is that I just don't see any basis for saying it's no longer an issue. Perhaps you could go into specifics as to what exactly made you say "Okay, now that's good enough."

For sure sample size is an issue, just not in the way it is an issue with APM. APM needs a big sample in order to not come up with insane results due to overfitting, that issue is eliminated with RAPM. Essential: While APM is trying to seperate player performances by all means, RAPM just says that if there is not enough data to seperate them, they might as well be equal in terms of value.
What we have to overcome within the sample is the normal variance of the player performances, and in average we see enough full cycles for each player within 25 games or so. A lot of players have a big game-to-game variance, some play 3 good games and 1 bad or whatever, but we hardly see players having 25 good games in a row while then 25 bad games in a row.

Doctor MJ wrote:I get the issues with APM. What I need is more confidence in RAPM. Not simply "RAPM > APM" confidence, but confidence that I can use short time spans.

I was talking about the issue with multi-year studies not about the difference of APM or RAPM here. The same problem applies to RAPM as it applies to APM. Increased sample size is adding another variance curve to the sample, as I tried to describe with short and long waves. When we have two waves overlapping, we can see amplification or reductation or even destruction of the wave pattern. No idea what is so hard to understand about that part.

Doctor MJ wrote:Well here's what I say: It's possible the resistance is based on wanting to believe that the weird values I see in RAPM (or APM for that matter) is sample-sized based because the alternative means that this data is farther from my perceptions of what the best players are and this discourages me.

Why do you want to use those values to justify your opinion about a player? Why aren't you using the lamppost to enlight you? ;)

Doctor MJ wrote:Meanwhile, I just don't see where you've ever really given the full argument for why RAPM stats are so good that they don't have some of the same issues APM does.

What? I have stated MULTIPLE times that there always exists a lambda for which the error of the ridge is smaller than the error of the OLS. That is mathematically proven. And that is also all you need to know about it in order to justify RAPM > APM!

Doctor MJ wrote:You explicitly state RAPM doesn't have those issues, but how am I supposed to be convinced?

By understanding the math behind it? Ridge regression in a case of an ill-posed problem like we have is per se better than OLS. Honestly, if you have any mathematical proof of the opposite, please present it. Otherwise you are just ignoring the facts here in order to justify your behaviour.

Ok, here's what we agree on:

1) RAPM/APM is the best system for evaluating individual defense. It definitely has its limitations, but nothing else comes close.
2) Both RAPM and APM get significantly better as the sample size and sample variety increases, so we all agree that either multi-year or prior-informed studies are better than single-year studies. The problem with this is that players change, for better or for worse. A multi-year or prior-informed study doesn't help us figure out just how bad Lamar Odom was this year.
3) In general, RAPM > APM, especially for smaller sample sizes. As sample size increases, this difference evens out somewhat.

This leaves a lot of gray areas. How big of a sample size is big enough? What's the ideal length of time for a multi-year study? How much more weight do we place on recent performance in a multi-year/prior-informed study? Is RAPM always better than APM?

I believe that basketball is situational enough that the answer to all of those questions is situational. There won't be a holy grail of defensive statistical analysis because the whole thing is eternally convoluted. Help defense/zone defense/crashing the boards vs. getting back/injuries/egos/terrible backups/great backups...whenever player/coach interactions change, so do players' RAPM/APM scores. LeBron's offensive RAPM score has fallen from 7.1 to 4.2 in the last two years: did he get worse? No, his situation changed and he became less critical to his team's success. If Steve Nash and Chris Paul joined the same team, their offensive RAPM scores would plummet. These same kinds of interactions are happening constantly, but on a much smaller scale.

Telfair-Redd-Childress-Morris-Lopez +12.1 in 147.7 minutes.
Telfair-Brown-Redd-Morris-Lopez -17.0 in 107.9 minutes.

Yet RAPM says that Brown is the better player. So, what's going on?? FIT Fit Fit fit FIT!!!

RAPM/APM should always be evaluated alongside the lineup data.

Los Soles wrote:A multi-year or prior-informed study doesn't help us figure out just how bad Lamar Odom was this year.

Yes, but what we also need to know is what can we expect from Lamar Odom next season. Is it reasonable to assume that he will continue to play like he did this season or is it more reasonable to assume that he will play better? Is the data on Odom for this season a good starting point to make a prediction for next season or is the prior informed data or multi-year data a better point of reference? I go as far as saying that I would rather expect Odom to bounce back and be even better next season than his prior informed RAPM suggest. That is obviously made under the assumption that he gets his psychological problems under controll (which isn't unlikely).

And that's where the value of RAPM comes into play (especially the prior informed version) in comparison to APM. What exactly do I have from a result which is entirely based on hindsight while all I want to know is what the guy can bring to the team in the future? RAPM is per se better as a predictor and prior informed is the best at this. I think it is very important to include how data can be used.

Anyway, for defense I think the combination of defensive RAPM (which gives the effect) and Synergy Stats+opponents stats (like 82games OppPER) is the best way to go. Someone who is doing better in both things is likely the better overall defender. If a player constantly scores good in both aspects, he most certainly is a good defender. If someone constantly scores average in both, he is very likely just an average defender (which isn't a bad thing by all means, because the majority of players is just around average). If a player is constantly bad in both things, he is for sure a bad defender.