Why I'm not a WP fan

[floppymoose]

WP = Wins Produced, as per the book Wages of Wins and the associated website http://dberri.wordpress.com/

Those who have discussed nba stat with me before will know I'm much more a fan of metrics that measure on/off court distinctions, than I am of metrics that are purely box score based.

WP is box score based, and as such is flawed in my opinion. It's difficult to prove this to the WP crowd, though. Finally a specific prediction based on WP came up that I think will reveal the problem: David Lee's trade to the Warriors and how that will impact their record. WP measurements suggest that his presence alone, holding other factors constant (such as the rest of the team being injured, Biedrins suddenly sucking, no improvement from young players, etc), will lead to *14* more wins by the Warriors.

There is an article on this here:
http://dberri.wordpress.com/2010/07/14/ ... on-50-wins

The article goes on to make some not unreasonable assumptions about the state of the team that gets the WP prediction all the way up to 56.9 wins.

!

I've been trying to convince them that this will demonstrate a problem inherent in WP as a tool for evaluating nba player performance. I'm "ohreally" in the comments section of that article. What do folks here think about WP, and about my criticisms of it?

[azuresou1]

Dave Berri has no idea what he's talking about, and dismisses any criticism, legitimate or otherwise, by challenging the person to produce it in a peer-reviewed journal i.e. he's a complete tool.

[Doctor MJ]

I'm not a fan. Two things:

1- Berri's one of these statisticians really talks as if he has the only answer, which is honestly impossible. Winston's also like this. Hollinger's not nearly as bad, and yet still does it enough I don't like him.

Aside from simply raising my ire, thinking you have the one answer just tells me that you really don't understand or respect the sport. These guys come in and clearly treat those involved with the sport like they're morons incapable of anything except superstition. No statistical analysis should be done this way.

2- I simply don't buy Berri's results (whereas Winston & Hollinger, as long as you don't go too far with them have useful stats). I've previously brought up the case of Marcus Camby -

In his last 2 years as a Nugget, Camby had a WPs of 17.6 and 21.0 respectively, and the next best player on the team had 6.3 and 9.5. This is an absolutely stunning statement of Camby being an MVP-level player, and being many times more valuable than accepted stars.

What happened when Camby left the next year? Did the team fall apart? No, of course, they got better, despite having now no one on the team with a WP in the double digits. Now defenders of the stat will point to injured players getting healthy etc, but the reality is that Camby has a career history of putting up comparable box score stats, and no significant track record of giving teams huge lifts in performance. In Ilardi's adjusted +/- stats, evaluating all active players as of '08-09 over the seasons from '03-04 to '08-09, Camby was 75th in the league. There's only one conclusion: What Camby contributes, can be mostly be done by plenty of others.

Now, I've seen places where Berri has made statements about the fact that his stat doesn't actually attempt to measure value over replacement, he's measuring how important different box score stats are for the team to win, and then crediting the players who got those stats. If he admitted this in a different way, there'd be nothing to be pissed off about. The problem is that he promotes the stat like it's gospel, and while if pushed he'll admit there's things he's uncertain about, he's never actually done anything to really sanity check his stat, like compare it to how teams actually seem to do with and without players. It's fine to give a conclusion with small caveats, but when others analyze the caveats and find huge red flags about your conclusion, you need to do so big-time soul searching, and he doesn't do this.

[Doctor MJ]

floppy, read up on the thread, and I think you sound very reasonable. This will be a good test for WP, thought stuff like this frustrates me:

Do the doubters of this not see the 500 ifs?

Many of these ifs won’t happen. Nelson isn’t going to allocate the minutes like the author states most likely, they never do.

Berri and his followers are very well versed at picking the slightest deviations from what they think the coach/GM should do, and using that to blame for under-performance. The nerve it takes to simply assume that if the pros would only listen to Joe Internet about minute allocation, teams would be winning 10+ more games is just astounding.

Again I think +/- is a really good thing to look at here. What +/- tends to show is that certain rotations aren't working, and which stars are really lifting teams. What has become clear it doesn't show is that there are a group of non-superstars of any particular mold (say, guys who get a ton of rebounds) that are consistently resulting in massive +/-. To be clear, I don't hold +/- to be the gold standards because of variability of it in addition to the fact that there is no intrinsic narrative. However, if there's a revolution to be had about the value of rebounds, etc - you sure think you could seem some glaring +/- trends to go along with it.

[floppymoose]

Did you see that Berri jumped in against me? He dismisses me as being "not well read". He also criticizes my understanding of WP without being in any way specific. He also states that plus minus is less suited to predictions because it is not consistent, which I don't think is a valid argument on it's own. Finally, he tells me all of this is covered ground if I would only read his book, Stumbling on Wins.

[Doctor MJ]

Did you see that Berri jumped in against me? He dismisses me as being "not well read". He also criticizes my understanding of WP without being in any way specific. He also states that plus minus is less suited to predictions because it is not consistent, which I don't think is a valid argument on it's own. Finally, he tells me all of this is covered ground if I would only read his book, Stumbling on Wins.

Classic Berri.

The most frustrating thing about him is that his strategy appears to be working. Other than Hollinger, he seems to be getting more mainstream pub than any other stat guy. I was utterly horrified when Malcolm Gladwell jumped on the Berri bandwagon. Gladwell is a guy who I've respected, and seems to find logical insight in areas with tons of assumptions, and here it's like he thinks Berri started the statistical revolution and everyone else is just a hater. That Gladwell would get involved, and choose the one big stat guy that everyone in the stat community thinks is a manipulative hack really undermines his credibility generally.

[andyhop]

The most frustrating thing about him is that his strategy appears to be working. Other than Hollinger, he seems to be getting more mainstream pub than any other stat guy. I was utterly horrified when Malcolm Gladwell jumped on the Berri bandwagon. Gladwell is a guy who I've respected, and seems to find logical insight in areas with tons of assumptions, and here it's like he thinks Berri started the statistical revolution and everyone else is just a hater. That Gladwell would get involved, and choose the one big stat guy that everyone in the stat community thinks is a manipulative hack really undermines his credibility generally.

Isn't his prominence in the mainstream a consequence of a lot of the other stats guys with different approaches taking jobs with teams and so not being able to continue to expand and publish their work openly.

[mysticbb]

Isn't his prominence in the mainstream a consequence of a lot of the other stats guys with different approaches taking jobs with teams and so not being able to continue to expand and publish their work openly.

This. And as far as I'm concerned Berri lacks the understanding of the game to get a job in the NBA. Nobody would hire him, because of his ability to ignore the flaws of his stats.

It is nothing wrong with the way Berri approached that, it is a good way to get the informations out of a regression analysis. But in the end he ignored some of the correlations the variables have which also screwed up his results. It should be rather obvious that the amount of rebounds in a game is somehow connected to the missed shots. He didn't account for that. He also assumed that letting the shot clock expire is a better alternative than taking a bad shot. Somehow he is missing the points and when you look at the complete results you will see that he makes scoring completely useless. Over the whole season scoring is canceling out, but that isn't true for the rest of his stats. If you go with the average scoring efficiency you should also use the average trb% for an adjustment. He isn't doing that. And telling him those kind of things will result into a request for writing a paper about that. Yeah, for sure, give me a job at your department and I will write a paper about that.

Berri is the stats guy with the least amount of knowledge about the game. And he is showing that in each of his blog entries.

[Doctor MJ]

The most frustrating thing about him is that his strategy appears to be working. Other than Hollinger, he seems to be getting more mainstream pub than any other stat guy. I was utterly horrified when Malcolm Gladwell jumped on the Berri bandwagon. Gladwell is a guy who I've respected, and seems to find logical insight in areas with tons of assumptions, and here it's like he thinks Berri started the statistical revolution and everyone else is just a hater. That Gladwell would get involved, and choose the one big stat guy that everyone in the stat community thinks is a manipulative hack really undermines his credibility generally.

Isn't his prominence in the mainstream a consequence of a lot of the other stats guys with different approaches taking jobs with teams and so not being able to continue to expand and publish their work openly.

A decent point, however:

1) There are vastly superior statisticians who are not directly involved with teams. The guys at b-r.com for example.

2) Berri is not separate from the greater statistical community. The other stat guys are in debate with him on his site, and a quick google of "basketball statistics" brings up other well respected stat guys but doesn't bring up Berri on the first page. For a random fool to become a fanboy of Berri is one thing, for someone like Gladwell to do it - someone known for being a stellar researcher, is disturbing.

[azuresou1]

Gladwell is overrated. A good 80% of what he writes about is common sense. What, you need breaks in life and hard work as well as talent to go places? Shocking!

[Doctor MJ]

Gladwell is overrated. A good 80% of what he writes about is common sense. What, you need breaks in life and hard work as well as talent to go places? Shocking!

His article on "The Quarterback Problem" which talked about teaching was excellent.

[ElGee]

Gladwell's one of my favorite writers -- a great story teller with highly readable prose.

That said, he does do narrow research at times, which is not even necessarily his own fault. He just tries to tackle complicated subjects from, what a professor once dubbed, "the mile wide and inch deep" approach. The guy can't be an expert in everything (they go "a mile deep and an inch wide"), and he can't be a New Yorker writer and convey big sweeping ideas if he reduced every story to scientific research and peer-reviewed journals.

His criticisms stem from the experts within each field he writes about noting how he used a poor source or missed a nuanced point. Fair enough. But I'd rather have Gladwell's imperfect stories and thought-provoking ideas, with online communities correcting him, than no Gladwell at all.

Btw, I liked his lecture on pasta sauce: http://www.youtube.com/watch?v=iIiAAhUeR6Y

[floppymoose]

My latest comment on that thread manages to put into words my thinking on box-score based player measurements, perhaps better than I have before (which wouldn't be hard):

http://dberri.wordpress.com/2010/07/14/ ... ment-74282

The gist of it is that WP has impressive correlation with wins because the *sum* of the stuff it is counting across a team full of players is correlated very strongly to wins. It's basically counting up a teams scoring chances and scoring efficiency and dividing those up among the players. But we have no good way of knowing that it's dividing up those values correctly. As long as the sum total is correct, it will appear to correlate strongly with wins.

Then I give what I hope is a powerful anecdotal example to illustrate the point.

[mopper8]

My latest comment on that thread manages to put into words my thinking on box-score based player measurements, perhaps better than I have before (which wouldn't be hard):

http://dberri.wordpress.com/2010/07/14/ ... ment-74282

The gist of it is that WP has impressive correlation with wins because the *sum* of the stuff it is counting across a team full of players is correlated very strongly to wins. It's basically counting up a teams scoring chances and scoring efficiency and dividing those up among the players. But we have no good way of knowing that it's dividing up those values correctly. As long as the sum total is correct, it will appear to correlate strongly with wins.

Then I give what I hope is a powerful anecdotal example to illustrate the point.

That was a very nice post. I didn't realize Wins-Produced is just another traditional-box-score compilation stat. Those are teh lame-o IMO.

[Ripp]

WP seems to be a very weird stat. Here is a description of how it is calculated:

http://www.wagesofwins.com/CalculatingWinsProduced.html

So it seems to be a linear weights sort of formula, with several odd team/position adjustments made afterwards. Here is a criticism I found from Dave Berri (a +/-) guy:
http://sonicscentral.com/apbrmetrics/vi ... .php?t=877

Basically, the team/position adjustments he makes afterwards seem to cause WP to closely fit the number of wins a team actually gets....but unfortunately, it doesn't appear that the WP rating actually does a good job of predicting wins and losses. So it seems to be overfitting the dataset, and thus have pretty limited predictive value.

As an example...I could come up with my own system that gives each player a rating of zero, then tack on a "team adjustment" factor to each player that is just the total number of wins Team X got divided by the total number of minutes they played. Clearly, this stat would have nearly zero value, since your rating is just a function only of the number of minutes you played and number of wins you got...it doesn't say if you are actually a good player or not.

Berri's stat is a bit more insidious than that...it appears to do fairly meaningless linear weights for each player that are then washed away by huge team adjustment factors.

[mysticbb]

Berri's stat is a bit more insidious than that...it appears to do fairly meaningless linear weights for each player that are then washed away by huge team adjustment factors.

Yes, without the team and position adjustments the correlation to winning isn't good at all, in fact it is lower than NBA EFF. We had that discussion here before in another thread.

But the biggest problem with Berri is his antics, his believe (or claim) that he is right and has found the holy grail of basketball stats. While in reality only a very few people are taking him serious.

[floppymoose]

So it seems to be a linear weights sort of formula, with several odd team/position adjustments made afterwards. Here is a criticism I found from Dave Berri (a +/-) guy:
http://sonicscentral.com/apbrmetrics/vi ... .php?t=877

Thanks for the link. I think you meant to say "Dan Rosenbaum", though. Dave Berri is the WP guy.

[Ripp]

^-- Heh, good call.

[Ripp]

Hrm, so one of the extremely bad aspects of WP is that if you are a stat stuffer but a bad defender, the negative impact of your defense is shared among all of your teammates, not just you (See step three of here: http://www.wagesofwins.com/CalculatingWinsProduced.html)

This explains why a guy like Jose Calderon ranks so highly in WP48.

[adambigbucks]

Hey guys, kind of new to this section of the board, but I'm a pretty big advanced metrics guy. Anyway, I seem to share a distaste for those who believe that what their regressions spit out are the word of god and have no chance of being flawed (call it Sagarin-itis)...Here is a great article/blog post about Berri and Gladwell that I feel you would find enjoyable (I know it is a couple of weeks old, sorry if it has already been posted):

http://mgoblog.com/content/sports-econo ... mgoblog%29