Why I'm not a WP fan

[Idunkon1stdates]

Here is the current TOP of the FM-guys:

Code: Select all

      Player       Age  Tm   Q FM48  FM
Kobe Bryant          32 LAL  1 0.676 24.2
LeBron James         26 MIA  1 0.594 22.7
Kevin Durant         22 OKC  1 0.592 22.6
Dwyane Wade          29 MIA  1 0.593 20.9
Derrick Rose         22 CHI  1 0.525 19.9
Kevin Martin         27 HOU  1 0.591 19.3
Amare Stoudemire     28 NYK  1 0.501 19.0
Dwight Howard        25 ORL  1 0.453 16.8
Dirk Nowitzki        32 DAL  1 0.567 16.7
Russell Westbrook    22 OKC  1 0.427 16.0
Monta Ellis          25 GSW  1 0.375 15.8
Carmelo Anthony      26 DEN  1 0.484 15.6
Blake Griffin        21 LAC  1 0.404 15.5
Eric Gordon          22 LAC  1 0.450 14.5
LaMarcus Aldridge    25 POR  1 0.343 14.3
Deron Williams       26 UTA  1 0.375 14.1
David West           30 NOH  1 0.368 13.6

I called it FM48 and FM in honor to floppymoose.

LOL. FM48 destroys WP48.

Kobe, LBJ, Durant, Wade and Rose vs.
Love, Paul, Howard, LBJ, Randolph.

Kevin Martin is Floppy's biggest outlier. WP is, um, Kris Humphries.

Kris Humphries isn't the biggest outlier. He is one among many. Reggie Evans has a higher wp48 than Lebron. Marcus Camby is a perennial MVP candidate according to Wins Produced. Ben Wallace was an MVP candidate. Dennis Rodman meant more to the Bulls than Michael Jordan.

[Vinsanity420]

LOL, this WP stuff should be published. Berri keeps firing off how he's a Ph.D. and there's no way we know better... there we go, we don't need any Ph.D.'s to properly analyze basketball.

[Jimmy76]

epic stat nerd showdown

wow

I wish could contribute but I think I'll just show my appreciation

[mysticbb]

Well, the DEF_A is a small adjustment that will not change the ranking of the players much ....

To show that this is true, I just ran a correlation analysis of the rank with and without the defensive adjustment:

Code: Select all

Correlations
                         Rank_Pts     Rank_adj
Rank_Pts  Pearson Correl          1       ,994**
          Sig. (2-tailed)                    .000
          Sum of Squares8271032.000   8217375.000
          Covariance      17902.667     17786.526
          N                     463           463
Rank_adj  Pearson Correl    ,994**              1
          Sig. (2-tailed       .000
          Sum of Squares8217375.000   8271032.000
          Covariance      17786.526     17902.667
          N                     463           463


As I said the adjustment doesn't change much in the player ranking, but that doesn't prove that it is not important for the correlation coefficent.

Here is the result of the correlation analysis without the defensive adjustment:

Code: Select all

Correlations
                               PTS_P           Win%
PTS_P       Pearson Correl               1       ,680**
            Sig. (2-tailed)                         .000
            Sum of Squares         469.918        66.054
            Covariance                .564          .079
            N                          834           834
Win%        Pearson Correl         ,680**              1
            Sig. (2-tailed            .000
            Sum of Squares          66.054        20.096
            Covariance                .079          .024
            N                          834           834


As everyone can see the correlation coefficent goes down from 0.97 to 0.68. Similar thing happens with Berri's WP. Getting a correlation coefficient of around 0.7 with the data of the boxscore isn't such a huge accomplishment at all.

[Idunkon1stdates]

LOL, this WP stuff should be published. Berri keeps firing off how he's a Ph.D. and there's no way we know better... there we go, we don't need any Ph.D.'s to properly analyze basketball.

The thing is, he has been roundly criticized by other sports economics, those with PhDs and those who are amateurs, for over 10 years. His first book was roundly criticized in peer-reviewed journals for its sloppy statistics and dubious assertions, and his second book was not reviewed at all by peer-reviewed journals. This isn't the first time people have made Berri look the fool, and it certainly won't be the last.

[mysticbb]

Well, this answer by nerd numbers gave me the motivation to do a bit more:

To clarify why those of us who use the Wins Produced Metric like it
1) It’s based on a good theory. Possessions and Points = Wins.
2) It correlates well with Wins.
3) It is predictable (a players AdjP48 from prior years explains about 80% of future years)

For FM48

1.) It's based on a good theory, Scoring per 100 possessions + Defense = Wins.
2.) It correlates well with Wins (0.97 correlation coefficient)
3.) It is predictable (a players FM48 from a previous year explains 80+% of future years.

The last part I compared 2010 FM48 values with those from 2011 for 292 players which played enough minutes in both seasons (250+). The result:

Code: Select all

Correlations
                               FM48_2010         FM48_2011
FM48_2010    Pearson Corre                   1        ,852**
             Sig. (2-tailed)                             .000
             N                             292            292
FM48_2011    Pearson Corre             ,852**               1
             Sig. (2-taile                .000
             N                             292            292


Now, FM48 explains wins better than WP48 and it seems to be more stable from year to year.

At the end we know that FM48 is based on a simple model, it is easy to apply and don't need a positional adjustments. Well, we apply Occam's razor here ...

[floppymoose]

I'm glad you covered that mystic. As I went to sleep last night I knew that the predictability of WP would be brought up as another prong of the defense of WP, and I figured that FM passed that test, but it's nice to see you have run the numbers.

[andre316]

I have to admit I do follow some of the WP guys semi-regularly. Even though I have a lot of problems with the metric, I do think they frequently "stumble" on some very good points and place more value on things other metrics like PER and EFF don't care about as much as they should, most notably efficient scoring.

Speaking of which, and I don't feel like starting a new thread on the topic but would if anyone thinks it's worth discussing, why do metrics like PER that assign arbitrary weights anyway not just set the league average or some other common sense threshold as the break-even point? I think one of the attractions of WP is that it rewards true scoring efficiency, while PER is fine with 30% chuckers.

[floppymoose]

Hey, I enjoyed "Wages of Wins" when I read it. I think the idea that you should take a fresh look at what correlates with winning in sports stats is a great starting point for learning things about a sport, and possibly shaking up conventional wisdom. I've got no problem with that.

But you need a critical eye on your own calculations. You need enough understanding of what is going on to draw some conclusions about *why* you are getting the correlations you get, and what they really mean. Are they causal, or are they an effect of something else? Do the relationships really hold for individual players, or just for complete teams of players as a whole?

This is where WP kinda fell down.

To me the natural next step is to carry the torch into on/off data. Let's bring the full force of analysis to that data and see how good a model we can build there, with the goal being a player metric that holds well for individual players, and that has solid predictive power about team wins.

Having said that, I'm sure some work has already been done there. I welcome any good links on the subject.

[mysticbb]

I'm glad you covered that mystic. As I went to sleep last night I knew that the predictability of WP would be brought up as another prong of the defense of WP, and I figured that FM passed that test, but it's nice to see you have run the numbers.

Before we (or anyone else) are using predictability as an argument, we have to understand where that is coming from.
There is only a limited amount of players anyway and from year to year the players with major minutes usually don't change teams. They are still in a similar role with the similar amount of minutes. Rosenbaum&Lewin 2007 is showing that. You can use minutes per game adjusted for positions and defense and it will have a higher correlation to the team's success than those other boxscore metrics. The high correlation is again controlled by the defensive adjustment. The rest comes from the defined role of that player.

Every player has his strength and his weaknesses, the coaches and scouts are able to determine that and will usually put those players into appropriate roles. Due to injuries, slightly changes in roles due to added rookies or free agents the values will change a bit, that's why the correlation will not be at 100%. But the value of 80+% is mostly explained by the fixed roles of the players. That is no different for any boxscore metric, WP falls under the same rules regarding this as FM48 is. A player with his primary role as a scorer will most likely be a scorer next season again. The numbers are showing it. The same goes for rebounders with limited offensive roles. Thus WP48 will still be rather constant from year to year.
Btw., the 2 or 3 year average values for APM are also pretty constant over the career of a player. The sample size is really a bigger issue here than with boxscore numbers, but even 1 yr values are useful, especially when they are combined with boxscore numbers like Dan Rosenbaum is doing it. Completely dismissing those values with the comment they are "inconsistent" is stupid, because that comment doesn't take into account why the boxscore numbers seems to be more consistent from the year-to-year standpoint.

If we understand all that, we know that predictability isn't characteristic of the model, but of the whole setting of the NBA. Each boxscore based model will have a similar trend regarding the year-to-year correlation. If there are more role changes for players within the league, the year-to-year correlation will be lower. If there are less, it will be higher. We could also add something like an aging curve for the players and we can probably improve the correlation a bit. But overall the year-to-year correlation isn't something which tells us anything about the model's ability to evaluate players. But the latter is something the coaches and scouts could use. To evaluate the performance of a player we can use those other metrics and can set them up regarding our preferences, with the right team adjustments the metric will still have a high correlation to winning and the predictablity will be similar to all other metrics.

FM48 is showing that, we set our preferences at scoring and we know that scoring per 100 possessions plus defense will give us a good estimation of the team's total wins. Using a linear model here is something rather simple, in the APBR community the pythagorean expectation is usually used with an exponent between 14 and 16.5. I determined the value on a dataset from 1980 to 2005 and came up with a value below 14 as the best approximation.

WP isn't doing anything else, they are using scoring per possession and opponents scoring per possession to determine the win%. Nothing wrong with that. But they are using two formulas to determine the possessions which are true for teams, but NOT for individual players. In fact the formula for PA isn't even defined in all cases. Not quite sure why NOBODY of the reviewers ever saw that. Berri is using a formula which isn't defined for the case that somebody gets an offensive rebound of a teammates miss and converting it right away via putback or tip-in. The whole underlying model is flawed. If Berri would have tried to publish such thing in a natural science journal, it would have been rejected.

And here is proof for this:

The formula PA = FGA+TO+0.47*FTA-ORB describes the amount of possessions. Points is the amount of positive credits someone gets for putting the ball into the basket. A close shot like a putback or tip-in will give a credit of 2 points. If a player attempts a shot, it will be counted as FGA. If a player grabs a missed shot, it will be counted as ORB.
To calculate the scoring efficiency (PPP), we have to do:

PPP = Points/PA

Assuming a player A is just in the game for an offensive possessions. One of his teammates takes a shot, but he missed. Player A is able to grab the offensive rebound and convert that right away for two points. After that sequence he is taken out of the game. His stats are 2 pts, 1 FGA and 1 ORB, everything else is 0. That means:

PPP = Points/PA
PPP = Points/(FGA+0.47*FTA+TO-ORB)
PPP = 2/(1+0+0-1)
PPP = 2/0

We are using a normal arithmetic setting, which means 2/0 is not defined. The model is invalid. And the statement is true, because we only need one example in which that model is not working.

And that is the real problem. Everytime a player gets the offensive rebound from a teammates missed shot, he used one less possession for his scoring. That is the reason we can find so many offensive rebounders on top of WP48. They are scoring points in Berri's model without having touched the ball. All those other things are minor and the defense thing is something every boxscore based model will face.

Well, I should probably just write a paper or a letter about that and publish it.

[floppymoose]

It's interesting that we came at this from different places. You from the desire to move people to a better box score metric. And me from the desire to show that seemingly positive attributes for a box score metric (win% correlation, stability over seasons) weren't really anything special.

In my case I want to show these things so that I can eliminate that as a prop for defending WP, so that we can move on to models that incorporate on/off data. I think on/off data will help us identify good defenders and differentiate between good rebounders and selfish rebounders. And who knows what else. It should tell us everything box score data does, plus more. Because at it's heart it's game flow data, which is box score data on steroids.

[mysticbb]

Actually I'm using On/Off Court and APM numbers quite often. I know how limited boxscores are, especially, as you pointed out, for defense. I ran a regression for the entries in the boxscore (adjusted for minutes and pace) and the DRtg of the teams, and the result was a linear model with a -0.5 correlation coefficent. Similar things were done by other people with a similar or same result. That just means that 50% of the variance of the DRtg can be explained by the boxscore stats, the rest is not described in any way with boxscore numbers. Thus using On/Off Court and defensive adjusted PM numbers will give us a better evaluation in terms of numbers. Everything else we have to either watch by ourself or we can use Synergy Sports data.

I want the people to use better boxscore metrics (I think Win Shares and WS/48 are doing the best overall job here (well, if we ignore my own rating :D)). The reason is that boxscore numbers are more intuitive and even more complexe boxscore metrics are easier to handle than On/Off Court numbers or APM numbers. But overall we want to look at every possible metric and decide which player might be the best in terms of impact. I guess real basketball purists, like a couple of guys here on this forum, are more interested in the player's impact than in his production. Well, that is the reason I tried to get a descriptive boxscore based model which can emulate the Net+/- and APM numbers. It is far from perfect, but I actually like it. It is closer to Net+/- than APM, but that is something which we can expect.

[floppymoose]

dberri has thrown in the towel. He has decided to block further comments from me on his WoW journal site. After blocking my post he posted this:

Thanks Jeremy. A very good response. I do understand all the critical comments your post inspired. Unfortunately, I find these criticisms to be unconvincing (but also have no desire to continue a back-and-forth with these people).

...which is funny since there was no back and forth with him. He never contributed anything to the discussion.

[DSMok1]

dberri has thrown in the towel. He has decided to block further comments from me on his WoW journal site. After blocking my post he posted this:

Thanks Jeremy. A very good response. I do understand all the critical comments your post inspired. Unfortunately, I find these criticisms to be unconvincing (but also have no desire to continue a back-and-forth with these people).

...which is funny since there was no back and forth with him. He never contributed anything to the discussion.

Blocking disagreement is not a good way to further understanding.

[ElGee]

Uh oh - my latest comment is now "awaiting moderation." Does this mean I'm in a for blacklisting? :/

[floppymoose]

Yup. I'm betting mystic will find the same thing.

[ElGee]

Just so you know, my final comment on the matter was:

Dave,
Not following you at all here.

1. When did I acknowledge that prior beliefs shouldn’t be used to evaluate models? (I believe my last post suggested the opposite.)
2. How can you not use a prior belief in evaluating a model? If I run a regression on CEO height and business success can I forecast the entire economy next quarter based on the results? Or will the first evaluation of that be based on prior beliefs about successful businesses?
3. I’ve quite clearly stated the predictive ability of WP is important here BECAUSE of the outliers identified by points 1 and 2. Again, WS has nothing to do with that, nor does wingspan as an evaluative tool. The model wouldn’t give us any knowledge if we didn’t have prior beliefs, and the prior beliefs couldn’t identify outliers if we ignored them. What other way, besides predictive measures, could we confirm that the model is telling us something new and useful?

I have no horse in this race, so the only metrics I prefer are ones that tell me something useful.

To which Berri issued the following email:

Thanks for the comments, but this discussion isn't really going anywhere. You clearly prefer Win Shares (which is fine, you can like whatever you want). But why you prefer this model is unclear. You say that forecasting is important, but Win Shares is less consistent than the model I developed. So if consistency over time is important (and that is what forecasting is all about), you wouldn't prefer Win Shares.

In the end, it does appear you are just looking at whatever model confirms what you believe. And that is fine. But I don't think that is how any model should ultimately be evaluated.

Best,

Dave

I suppose this is par for the WP course. I'm not going to start a battle over it - i have nothing against the WP community - I just think the problems with the metric are glaring and I've never seen them remotely addressed. Doing a dance about prior knowledge, ignoring all specific questions and concerns and simply saying "you don't understand" or "my method has the highest correlation to wins" isn't exactly convincing.

I gave Berri many opportunities, he apparently didn't want them. I suppose he doesn't think outliers in his model should be substantiated, he just thinks everyone's prior knowledge about the sport is wrong. :shrug:

[floppymoose]

I don't get his claim "consistency over time is [...] what forecasting is all about". What consistency is he talking about?

Accurate predictions are what forecasting is all about.

[mysticbb]

Yup. I'm betting mystic will find the same thing.

Correct, Sir! My latest comment was blocked too.

I wrote Berri an e-mail that he can delete comments as much as he wants, that will not change the facts. WP is not evaluating players, but overall teams. Using the formulas he did will make that sure. As I pointed out the ability "to predict" doesn't come from the model, the correlation to winning comes from point differential which is triggered by the team defensive adjustment. The defensive adjustment doesn't add anything to a deeper evaluation of individual players, because it just assumes that every player on the team has the same defensive impact.
The way he is evaluating the model (and the reviewer did), doesn't tell anybody anything about the model's ability to evaluate players.

Blocking disagreement is not a good way to further understanding. :(

Berri isn't about understanding, he is about attention and book sales. Thus any critic is bad for him, especially when he has not a convincing answer to those. He shows that by blocking the comments.

I wrote a comment to the 76ers, because he claimed an average center would be enough to push them to 50 wins. He wrote that 0.100 WP48 is average, but he didn't account for overtime minutes. And when someone writes with such a precision, he should also use the precise number (which is 0.099 and not 0.100). I also pointed out that DJ MBenga will be a free agent next season, he had 0.105 WP48 at this time. But I somehow doubt that MBenga will be the center who can push a team to 50 wins. He deleted that comment.

Well, ElGee, you should prefer Win Shares, because Win Shares doesn't think that a player can score points without touching the ball.

[DSMok1]

Mystic: Do you agree that the following measure would probably be the best valuation of a metric? I outlined this to Berri a little while back. Some variation on this would be good, I think:

Let me pose the following "Test"; please tell me if it is statistically valid.

Take Year Y. Calculate measure for each player for Year Y. Compile full per-minute rates for each player. (Alternatively, add regressing this value)

Take Year Y+1. Apply Year Y values to Year Y+1 minutes (or possessions, to be more precise) and generate an expected efficiency for each team.

Compare Measure 1 to Measure 2 by looking at how expected efficiency for Y+1 compares to measured efficiency for Year Y+1.