Introducing BBB's VORP/Player Value for Review

[azuresou1]

Hey guys, I've put together two new stats - VORP and Player Value - that are designed to be all-encompassing stats for a player's actual production. Since this is a introductory forum post, I'll be cutting back on the technical details and speaking to the broader, conceptual arguments. I'll link to the details for those who want to take a more in-depth look.

Player Value is the expected value in salary the player will contribute but is largely a function of VORP and minutes, so I'll be explaining VORP.

Overview of VORP

VORP is the sum of OVORP and DVORP. I only looked at players who played this past season (so no Derrick Rose), and I prior-informed one-year back. The idea is to combined actual production with sabermetrics to produce an all-encompasing assessment of a player. Here's a small sample:

Top 5 SGs by VORP O/D/T
1 James Harden 950% 1% 951%
2 Kobe Bryant 951% -31% 920%
3 Dwyane Wade 787% 56% 843%
4 Manu Ginobili 676% 95% 771%
5 Monta Ellis 522% 15% 537%

How is VORP different from existing stats like WARP or RAPM?
While similar to WARP, VORP is broken down into offense and defense, which I believe adds value, particularly for situational players like Steve Novak or Andris Biedrins. Unlike RAPM which has a similar goal, VORP utilizes actual production, rather than being driven purely by on/off. I believe this produces more accurate and intuitive results, and I've done some (cursory) analysis comparing the two. I'll link the comparison at the end of the post.

Explaining OVORP and DVORP
--------------------------------------------------------------------------------------------------------------------------
Offensive VORP (OVORP) is broken down into three sections - Scoring Ability, Shot Creation, and Team/Positional Adjustment.

Both volume and efficiency should come into play when talking about scoring, and so Scoring Ability is a function of per-minute scoring and TS% to get an overall sense of how good a scorer someone is. To simplify our team adjustment for spacing later, we then cheat a little and bake in a bonus for 3PT shooting, again volume and efficiency.

Scoring obviously isn't the only part of offense though - the ability to create for yourself and others without turning it over is a pretty vital piece. Shot Creation attempts to recreate this, by looking at a player's AST% in relation to his TOV% and USG.

Finally, Team Adjustment attempts to incorporate the remaining pieces of offense by looking at how a player's personal ORTG relates to both his team's overall ORTG as well as league average. This greatly rewards guys like Tyson Chandler, who are neither prolific scorers or good passers but who greatly enhances his team's offense simply by being on the floor, extending offensive possessions, and taking the right shots, while penalizing guys like Josh Smith who are better scorers and passers but whose offenses crater due to bad offensive tendencies.
--------------------------------------------------------------------------------------------------------------------------
Defensive VORP (DVORP) centers primarily around the relationship between a player's DRTG and his team's/the league average. Normalizing for team is important, because a guy like Jamal Crawford shouldn't get a free pass because his team is defensively good, nor should a guy like Andre Drummond get slammed because he plays on the Pistons. While many are not a fan of individual DRTG due to its derivation from boxscore stats, I believe when used in conjunction with team DRTG it's pretty valuable.

We then take this figure and adjust for STL%/BLK%, which DRTG captures, but is IMO insufficient when it comes to overall impact. Finally, we apply a positional weighting, under the premise that big man defense is more pivotal to team performance than wing defense.
--------------------------------------------------------------------------------------------------------------------------

Top 10 by VORP O/D/T
1 LeBron James 1510% 215% 1725%
2 Kevin Durant 1363% 142% 1505%
3 Tim Duncan 635% 679% 1314%
4 Chris Paul 1219% 78% 1297%
5 A. Drummond 463% 715% 1178%
6 Dwight Howard 530% 455% 984%
7 Stephen Curry 1016% -54% 962%
8 James Harden 950% 1% 951%
9 R.Westbrook 902% 31% 933%
10 Kobe Bryant 951% -31% 920%

Thoughts, comments, and pageviews obviously appreciated

Comparison between VORP and RAPM results

Full list of VORP/Player Value with actual calculations

[Chicago76]

The big question I would have is why bake in bonuses, team adjustments, and all of that stuff? Why not just use DRAPM and ORAPM to compute VORP?

For example, if we know that ORAPM replacement level is -2, and we know a player is +2 and he plays 36 minutes per game, isn't VORP on the offensive end just [2 - (-2)]*36/48, = +3?

[azuresou1]

While I am a fan of what RAPM aims to do, I don't believe it accurately teases out the noise created by various lineups. For example, do I believe Tony Parker is a top 10 defender and the equivalent of Tim Duncan? No, I believe that is noise created by having worse defensive subs behind him, such as Gary Neal, and also playing many minutes with great defenders like Duncan/Splitter. Now it wouldn't be fair to pick on this specific example, but that general flaw (as I see it) discredits RAPM in practice for me.

I linked a comparison between RAPM and VORP, and you can ser some major differences, which should answer why I rejected it.

[Chicago76]

You're correct in your Parker example. Sometimes lineups, sample size of various lineups, etc lead to larger error terms. In this case, Parker's DRAPM is extraordinarily high compared to anything he's done from 03-12, where his median/avg rating hovers around 0.2. Let's not throw the baby out with the bathwater though. RAPM has one strength in terms of validation: it reconciles to overall team performance (with a small rounding error). It definitely reconciles to lg average, where minutes (or possession) wtd average of all players is lg average (or zero). You can look at the bottom 5% or so of players to figure out a proxy for replacement (roughly -1.5 for O and -1 for D). There are misallocations at times (like in your Parker example), but if you take a long-term view, player values tend to center in certain ranges for particular players.

We can validate your results by minutes-weighting the individual players of a given team or league to arrive at aggregate VORP for defense . Minutes weighting your defensive rating gives a league average of -.03. This is obviously not valid. Assuming that the DRAPM vs. VORP scaling in your comparison is correct, 100% = 1 pt of RAPM. On a team level, this means that there are 5-6 teams out there that are 3 to 7 pts worse than replacement. It's possible that there is a team or two out there in history that might actually have a defense so awful that replacement players could play that end of the court better. But 20% of the league is not playing 3 to 7 pts worse than replacement in a single season. No matter if the scaling above is correct, it still intuitively is not possible that the lg average player is negative vs. replacement defense.

Scaling on the offensive end is a bit different. It still appears to be 100% to 1 pt, but the intercept is 0 DRAPM = 400%. We'll look at it two ways. First 100% OVORP = 1 pt better than replacement w/ no intercept adjustment. If this is correct, then the lg average player is 4.1 pts better than replacement and an average team is 20 pts better than replacement. A replacement team would have lg average D (from above), but would be 20 pts worse on O, resulting in an expected win total of 4 games.

If we assume that 400% = replacement, then the formula is OVORP - 4 = RAPM. The league average player is now only 0.11 pts better than replacement. This has the same issue as the D above.

What is driving these issues is the lack of empirical weighting placed upon your adjustment variables. Things get squared, multiplied by 2, divided by 300, subtracted from one another etc with as best I can tell no empirically derived rationale. At a bare minimum, you should be able to show that every team in the league is playing

What you are trying to do, it seems to me, is to derive a less noisy alternative to RAPM. These exist in the form of statistical regressions of the RAPM data. Rather than say we believe X is important, so we're going to weight it this way, run a multivariate regression with your variables, test for significance and overfitting, and let the coefficients speak for themselves. You'll get something that makes sense on a league-wide basis, and almost always a team basis as well.

[ceiling raiser]

Cool study! Can you describe the positional adjustments for OVORP/DVORP in more detail? I'm interested in what sort of values you use for each, and how you decided on them.

[azuresou1]

While I was building the stat, I also noticed that average DVORP was sub-zero (-8% median), and asked myself whether that made sense at all or not. Ultimately I concluded that while maybe a bit surprising, it's entirely possible. I should point out that since DVORP:DRAPM a 100%:1 translation, a -3% DVORP translates to a -.003 DRAPM and not -3 DRAPM.

As a validation, I pulled the list of D-League call ups and looked at what the average D/OVORP was. Most of those players are not listed on the website due to not playing sufficient minutes to qualify, but the minute-averaged DVORP for those players was very close to 0 (-5%). In contrast, the players who were called up had an OVORP of 144%.

I don't know why you would assume that 400% = replacement when it is meant to be value over replacement. I centered 400% OVORP with 0 ORAPM due to the median OVORP being 320% and the median RAPM being -0.9. A replacement player is not coming in and playing at the level of a guy like Al Harrington or Rudy Gay (~385% OVORP). They're coming in and playing at the level of Jamaal Tinsley or Tyler Honeycutt.

This conclusion this seems to indicate is that the average NBA player on a roster is not any better than a replacement level player on defense, which I'm sure many will question. I think it's not out of the question that NBA players primarily distinguish themselves from D-League replacement players based on their level of offensive talent, and that guys who do get called up from D-League largely do it based on their offense.

I am not trying to derive a less noisy RAPM. I reject the idea that RAPM is valid at determining the best players in the league. Is a lineup of Andre Miller, Tony Allen, Vince Carter, Amir Johnson, Taj Gibson better than a lineup of Steph Curry, Paul George, Kevin Durant, Lamarcus Aldridge, and Tim Duncan? Not in a million years. Team A gets crushed. Yet RAPM would have you believe that the first team would produce better results (4.5 difference), or at the very least that the former players have a bigger impact than the latter. That to me is insanity.

[mysticbb]

How good is that "metric" in terms of explaining and predicting? From the first look at that, it doesn't look any better than PER in that regard. And that's what a metric should be based on, not what you believe is somewhat "valid".

RAPM is also not telling you that a lineup of Miller-Allen-Carter-Johnson-Gibson is per se better than other lineups, but that the players in their respective positions on the court within the framework of their respective team are influencing the scoring margin per 100 possessions at a certain level. The idea here is that players are usually used in a similar fashion, transferring them into a different position will likely change their influence on the result. That is a big interpretation issue here, but that is something seen a lot of times, in most cases by those who actually don't even know how the method works.

A similar thing can be said about boxscore-based metrics. They are a bit more stable, but in the end the biggest factor for the year-to-year consistency is that players are used in a similar fashion, even when they are changing teams.

[azuresou1]

That's actually the next step for VORP - proving its worth as a explaner/predictor of performance, so I'm glad that you brought that up.

I'm a little disappointed though that you don't think it looks any better than PER, especially since VORP not only folds in ORTG relative to team/league, but also has a defensive component.

RAPM tells you that Thabo Sefolosha has a bigger impact to OKC as a defender than Kevin Durant does as a scorer. Which is, again, to me absurd. You'll note that in the Miller/Allen/Carter/Johnson/Gibson lineup, I am not asking anyone to play out of position, nor in a role that they're not especially suited for. The possible sole exception is VC will now have to take a few more FGA/game, but I don't buy that as a groundbreaking change.

[mysticbb]

RAPM tells you that Thabo Sefolosha has a bigger impact to OKC as a defender than Kevin Durant does as a scorer. Which is, again, to me absurd. You'll note that in the Miller/Allen/Carter/Johnson/Gibson lineup, I am not asking anyone to play out of position, nor in a role that they're not especially suited for. The possible sole exception is VC will now have to take a few more FGA/game, but I don't buy that as a groundbreaking change.

You are clearly missing the point here, which comes down to a wrong interpretation of the numbers. Sefolosha helps the Thunder to play better defensively than an hypothetical average player would do. RAPM may tell (not quite sure what numbers you are talking about, what numbers you have access to) you how much difference that makes per 100 possessions. RAPM can't deveiver whether that is due to his defense alone or due to some offensive skills as well, it just measures the effect here. RAPM also can't tell you how much impact Durant makes as a scorer, it is his combined impact in comparison to an average player offensively, which might be lowered due to his high turnover rate for example. Really, your interpretation makes not much sense when you actually look at what RAPM is actually doing.

And whether you "buy" into that being a "groundbreaking change" or not is not the relevant thing at hand. Miller for example is not solely used as PG, but also gets minutes as SG, where the Nuggets are playing better than when he is the PG, Allen gets a few minutes at SF, the Grizzlies are also playing better in such a case than when he played SG, Carter as SG has the Mavericks playing better than when he is the SF, Johnson gets minutes at C with the Raptors playing better than, at Gibson gets his majority of minutes at PF. But then again, those players are plugged into roles, in which they can succeed, anything else wouldn't make much sense for the respective teams.

RAPM is not supposed to be a player ranking tool, it is a tool which helps determining which player helps a team over an replacement player for the respective position. Trying to paint that as a 1-single-number which would completely tell you everything (like your interpretation suggests) is just foolish. It is actually the combination of that RAPM with an effective boxscore-based tool, which makes it much more possible. And it is proven that such a way will give a better prediction.

[azuresou1]

If RAPM states that Durant's presence on the floor helps his team over a replacement by 3 pts/100 possessions on offense and 0 points on defense, and Thabo's presence on the floor helps his team by 4.4 points on defense but hurts his team by 1.1 points on offense, RAPM indirectly states that Thabo's presence on the floor has a greater impact than Durant's.

In any case, you agree with me regarding my initial assertion: that RAPM is invalid at determining the best players in the league, and is merely useful at letting us know how players produced in certain roles.

VORP is, however, intended to be a player ranking tool, which makes it different from RAPM.



All RAPM numbers I reference are from here: https://sites.google.com/site/rapmstats ... d-march-30

[mysticbb]

If RAPM states that Durant's presence on the floor helps his team over a replacement by 3 pts/100 possessions on offense and 0 points on defense, and Thabo's presence on the floor helps his team by 4.4 points on defense but hurts his team by 1.1 points on offense, RAPM indirectly states that Thabo's presence on the floor has a greater impact than Durant's.

Under specific circumstances that might be even true? Then you take into account that this is per 100 possessions, take the possessions played into account, and you may become a different picture. Sefolosha may not be able to play more minutes whether due to stamina or lack of skills doesn't really matter.

VORP is, however, intended to be a player ranking tool, which makes it different from RAPM.

VORP, btw, exists already as an acronym for a player metric. dsmok is using that, and he is actually basing his metric on a regression of boxscore numbers on multiyear RAPM. His metric is also pretty good at predicting. I suggest, you rename yours in order to avoid further confusion.

[azuresou1]

Under very specific circumstances, sure. Over the close to 4K possessions that they shared the floor together though? I find that hard to swallow.

I will relabel the thread to reflect that it is the BBB specific VORP for clarity, until I think of a better name. I'll also get on putting together some validation.

[azuresou1]

How good is that "metric" in terms of explaining and predicting? From the first look at that, it doesn't look any better than PER in that regard. And that's what a metric should be based on, not what you believe is somewhat "valid".

A quick and simple validation is up on the website as the first post.

If you're loathe to link, there's an extremely high correlation between Team minute-weighted O/DVORP and O/DRTG, and Player Value and Wins.

[mysticbb]

If you're loathe to link, there's an extremely high correlation between Team minute-weighted O/DVORP and O/DRTG, and Player Value and Wins.

First, you are not checking how good the model is in terms of prediction, but rather how good it can explain something. To see how good it can predict, an out-of-sample test would be necessary. You can use a retrodiction test, where you use the values derived from 2012 to predict the outcome in 2013.

Also, a graph is nice and so, but getting a number is better. What are the correlation coefficients for those 3 graphs, respectively.

[azuresou1]

If you're loathe to link, there's an extremely high correlation between Team minute-weighted O/DVORP and O/DRTG, and Player Value and Wins.

First, you are not checking how good the model is in terms of prediction, but rather how good it can explain something. To see how good it can predict, an out-of-sample test would be necessary. You can use a retrodiction test, where you use the values derived from 2012 to predict the outcome in 2013.

Also, a graph is nice and so, but getting a number is better. What are the correlation coefficients for those 3 graphs, respectively.

That's a great idea, actually. I'll need to recrunch the numbers to get prior-informed 2012 VORP. From there, what do you think would be more useful - a comparison of 2012 to 2013 VORP on a player basis, or an ORTG/DRTG/Wins prediction based on 2012 VORP?

I'm inclined to simply do a 2012-2013 comparison, since a prediction would assume that I have perfect knowledge of players MP, which is a big assumption (e.g. Steph Curry being healthy all year, Bynum missing the full season). A simple 2012-2013 comparison should also directionally give some sense of how volatile VORP is.

However, I'm completely willing to concede it'd also be much harder to glean the predictive power based solely on a 2012-2013 comparison.

I'll need to check the correlation coefficients, but off the top of my head:
OVORP-ORTG was around a .8
DVORP-DRTG was around a -.95
Total Player Value - Wins was around a .9

I really should upload my spreadsheets somewhere...

[mysticbb]

I'll need to check the correlation coefficients, but off the top of my head:
OVORP-ORTG was around a .8
DVORP-DRTG was around a -.95
Total Player Value - Wins was around a .9

Are those R or R²? Is it possible that you mixed the offense and defense up? Because the boxscore is biased towards offense and usually metrics are getting high correlation for offense, but not that high for defense.

[azuresou1]

R Scores:
OVORP-ORTG: .8475
DVORP-DRTG: -.9798
Total Player Value-Wins: .9014

Rsq:
OVORP-ORTG: .7183
DVORP-DRTG: .9600
Total Player Value-Wins: .8142

DVORP should have a higher correlation to DRTG than OVORP to ORTG, since DRTG factors in so heavily for DVORP. I'd actually be worried if the correlation between them wasn't at least a .8.

[mysticbb]

First, the connection you made on the defensive part looks really suspicious. Somehow it seems as if you just had every player's defensive valued based upon team defensive rating, which doesn't tell you that much about the individual defense. Based on individual boxscore stats, the variance of the defensive strength for the overall team is covered by about 50% (R² of 0.5). That's the best you can come up with. Only if you assume certain things about the relationship between the opponents data and the individual boxscore numbers, you can raise that. What you are basically doing on the defensive end is "cheating", and that to a very high degree. I suspect you will get a rather low year-to-year consistency for that part, because players changing teams will have a bigger influence here, it suggest a lack of reliability. That will drag down the predictive power of your overall metric.

On the other hand your offensive part is way lower than expected. An R² of 0.72 is worse than PER, while adding a similar defensive "adjustment" like yours to PER would drive that up for the total wins correlation to 0.9+ (R² is that). I don't see that this part is really valid.

I don't want to discourage you, but I'm really not impressed by those correlation coefficients. Metrics like Win Shares, ASPM or my SPM are beating that rather easily (my SPM translated via pythagorean expectation to wins has an R² of 0.9469 to wins in a dataset from 1978 to 2012, and my adjustment for strength of schedule is actually hurting a bit in that regard, but it improves the predictive power). But in the end only a real out-of-sample test would give you an idea about the value of that metric as player evaluation tool.

[Chicago76]

^The problem is he's assigning his own weights, so they're not regressed (as in SPM) and they aren't at least intuitive (as in PER). If we know avg over replacement for both O and D (which is lg avg VORP), we can compute suggested ratings for teams as well.

Defensively, his values reconcile quite well to team drtg, but as you pointed out, he is relying heavily on individual drtg, which in and of itself must reconcile to team drtg. That doesn't mean it's a good metric per se, just that it reconciles well.

Offensively, there are some pretty huge error terms between actual team ortg and team ortg implied by aggregating player values in this study. In a lot of cases, we're talking something that falls 3-5 pts away from actual team ortg. This occurs roughly 20% of the time. Not good at all when you consider just how many more thing we can track on O than on D.

Something is wrong with the implied replacement levels on O and D too. He's looking at 10-day contract DLeaguers as replacement level. In reality, some young guys are given a spin based upon the concept of potential. They won't play much, it's low risk, moderately high reward to find a young guy who might fit on the end of your bench that you can groom into a long term rotation player for cheap. Lg min mid-season FA pickups or D-League guys who have been out of school for at least 2 years are typically better players. They don't have the upside, but they are readily available, any team can sign them, and on average they perform better. Most decent SPM studies I have seen (like DSMok1's) have replacement level O at about 2.0-2.5 pts/100 worse than lg average and D at about 0.5 pts/100 worse than league average. It boggles the mind that a reliable study could come up with lg replacement on D being better than lg avg. I know DSMok1 had the same initial conclusion with his study until he went back and thought about replacement level in a more thorough manner.

[azuresou1]

Defensively, you should re-read how I'm calculating DVORP. I'm not making any assumptions about opponent data; everything is based solely on a player's stats, his team overall rating, and the league average. I already acknowledged earlier that we should expect that this would correlate extremely highly with Team DRTG. If I sound defensive, it's because I don't enjoy the implication that I'm being "suspicious" or "cheating" in my analysis.

Offensively, I'm not sure how you're finding that calculated ORTG falls 3-5 pts away from actual 20% of the time. I have 2 teams that fall outside a 3 pt boundary - BOS and NYK - and one that sits right at 3 (MIL). While not ideal, that's half your stated rate.





Anyways... as a validation I looked at year-over-year change at a player level, the idea being that if VORP works well, we should see either minimal unexplainable changes year-over-year.

http://www.buildingbetterball.com/2013/ ... ar-change/

Summary is that we do indeed see some significant swings in DVORP, but primarily among backup big men who get new (usually bigger) roles. However, among big minute players, the variance is significantly less. I also believe the OVORP variances observed are perfectly indicative of changes in player performance.