Introducing BBB's VORP/Player Value for Review

[Chicago76]

Defensively, you should re-read how I'm calculating DVORP. I'm not making any assumptions about opponent data; everything is based solely on a player's stats, his team overall rating, and the league average. I already acknowledged earlier that we should expect that this would correlate extremely highly with Team DRTG. If I sound defensive, it's because I don't enjoy the implication that I'm being "suspicious" or "cheating" in my analysis.

I understand how you're calculating DVORP. The big component of DVORP is when you compare player drtg to team drtg and team drtg to lg drtg. You can just cancel these two terms out and call it player drtg compared to lg drtg. Drtg in and of itself, when you minutes weight it by player for a given team, will reconcile to team drtg. We're just stating the obvious, which is that you are largely anchoring player DVORP to team drtg, so it should reconcile fairly well. That reconciliation is only good to the extent that individual drtg is a good metric.

Offensively, I'm not sure how you're finding that calculated ORTG falls 3-5 pts away from actual 20% of the time. I have 2 teams that fall outside a 3 pt boundary - BOS and NYK - and one that sits right at 3 (MIL). While not ideal, that's half your stated rate.

My apologies. I was working from memory. I should have stated 2-5 pts from average. I don't have your full data set for low min players, nor did I allocate players who were traded to teams, but based upon what I do have (which still accounts for the overwhelming majority of team minutes), this is what I saw.

BOS 4.5 pts off
NYK 3.4
MIL 3.2
GSW 2.5
ATL 2.4
UTA 2.0
TOR 2.0
WAS 2.0
PHO 2.0

So 30% of the league is off by 2 pts (rounded to nearest tenth) or more. That's not good. Keep in mind some of these may not be off by as much when you include traded players or those falling below the minute threshold. Others I haven't listed may also appear too, however. This is based upon 4.11 OVORP = lg avg, which may be slightly different when including lower min players.

Anyways... as a validation I looked at year-over-year change at a player level, the idea being that if VORP works well, we should see either minimal unexplainable changes year-over-year.

http://www.buildingbetterball.com/2013/ ... ar-change/

Summary is that we do indeed see some significant swings in DVORP, but primarily among backup big men who get new (usually bigger) roles. However, among big minute players, the variance is significantly less. I also believe the OVORP variances observed are perfectly indicative of changes in player performance.

This type of test doesn't necessarily validate your study. It only tells you that similar players producing similar numbers get similar results year over year. The results may very well still be wrong. If you want to validate the study, I would suggest doing an out of sample test at the team level. Calculate OVORP and DVORP for 2-3 prior years for each player. From this calculate team Ortg and Drtg. Then look at the error terms between your implied team o and d rtgs and actual team o and d rtgs. Compare these error terms to other studies. Example: dsmok1 has an Off +/- adjustment he has included in his spreadsheet. A 0.4 adjustment = a team error of 2 pts (04. x 5 players on the court). Also, why do you believe that, "OVORP variances observed are perfectly indicative of changes in player performance"? Is it because you want to believe? Is it because you've tested the variances in OVORP compared to ORAPM and note similar consistency? Some other reason?

Finally, IMO, you still haven't addressed the issues that someone looking at this critically will bring up:

1) How did you select the weights in your analysis? Is there anything to it other than these are the weights you came up with on your own?
2) How can you explain that a replacement level player plays better D than an average NBA player? Did you look at the commentary I directed you to from Dsmok1? Did you consider that first or second year 10-day players aren't true replacements?
3)Have you considered at least making a forced adjustment for each players OVORP and DVORP such that the sum of player values for a team will reconcile to actual team OVORP and DVORP?
4) How did you empirically derive your position adjustment?
5) Have you attempted to validate your data vs. SPM or other means?

[azuresou1]

There's actually a difference between comparing player DRTG to league DRTG, and adding a player DRTG to team DRTG comparison. Example: Tony Allen has a 98 DRTG, compared to his team at 100.3 and the league at 103. If we just take personal vs. league, he gets an 5. With the team adjustment, he gets a 7.3. Andris Biedrins also has a 98 DRTG. However, his team has a 105.5 DRTG. With the team adjustment, he gets a 12.5.

As for validation, I agree with you on the need for a out-of-sample test. My belief that the variances are indicative of changes in player performance are subjective and based off of an assessment of how much their VORP has changed combined with my appraisal of the player's season. You'll forgive me for not testing variance compared to RAPM, given that I think it's a metric that has no value outside of fitting a regression model well.

To answer the specific criticisms:
1. Weights were based off of my judgment and refitting until I had numbers that subjectively seemed appropriate for each sub-category. I suppose I could run everything through CART/TreeNet to get better weighting accuracy. But yes, the weights are subjective.
2. I did see the commentary you had about replacement level defense, and truthfully I haven't had time to really look into it in great detail. That said, I don't know I agree with your assessment that the D-League callups aren't true replacements. Guys like Damion James, James Anderson, Garrett Temple, CDR - they've been in the league before. It's not their first trip to the rodeo. They're getting called up not because teams think they've found the next Jeremy Lin, but because they need bodies. Temple in particular played for his sixth NBA team this past season, at age 26. Is there a reason that you believe an NBA player is a significantly better defender than a replacement player, given that the majority of teams appear to pick up players primarily due to offensive talent?
3. Yes, I've considered it quite heavily actually. The problem is that I'm having difficulty figuring out how to properly adjust, given that there's no clear indicator ATM of what is causing specific teams to be over/under forecast. Maybe running it through TreeNet would give me an answer. In lieu of that, do you have any ideas about how to adjust?
4. This is probably the single LEAST defensible part of what I did - I eyeballed it entirely I should actually run a validation based on unadjusted VORP and see how that fares...
5. I did a QnD validation against RAPM, which indicated a RSQ of ~.4 on offense, ~.15 on defense, and in between for overall. I didn't validate against any other models