Chronz wrote:So the consensus(amongst the APBR community at least) is that the very reason he defends it (that the individual markers correlate almost perfectly at the team level) is not only irrelevant but proof of its shortcomings?
What? No, that is not at all the critism of WP48. The method is inconsistent, that is one issue, the other issue is that things on a team level do not scale down to the individual level. Berri assumes two specific things he NEVER showed are true. First, that each boxscore metric has an intrinsic value which can be determined by regression, and second that each player can be looked at as if he would be a "team". Neither things were shown by Berri nor are those two things true.
Berri's way of "proving" his metric is completely inconsistent and does not show anything. You can basically take the uniform numbers, adjust them by "position" and then make a team adjustment and the result will be a pretty much "perfect" correlation to the overall team wins. The high correlation is controlled by the adjustments not by the initial unadjusted metric. I showed that in another thread by just taking a scoring rate, adjust that with a team defensive adjustment and I get an even better correlation than Berri. I can make the same steps to "prove" that this metric has some value and get the very same result as Berri. Well, Berri is ignoring two very important biases: selection bias and hindsight bias.
The players are pre-selected by scouts and coaches and will in almost all cases be used in a similar way. No coach will try to make his "enforcer" to be his main "scorer" for example. The players will have similar roles and minutes in consecutive seasons. That just a simple fact. And using hindsight data to predict something which already happened is not something which is really impressive. In fact, we have a better method to predict team wins, pythagorean expectation, than the linear relationship Berri is using.
Berri's method is inconsistent, because at the beginning he is proposing that boxscore entries would have intrinsic value which can be determined by regression. Well, Berri is doing one regression and one regression only, he uses linear regression to determine the win% by using offense and defensive efficiency as independent variables. After that Berri is using league average values to determine the marginal values of each boxscore entry. So far, so good. The issue is that now we get a pretty good differentiation between players, bigs are incredible valuable while smalls are not. At that point Berri is doing something which is completely unscientific, he is introducing an adjustment for somewhat arbritrary positions. Well, if Berri would have stayed with his original hypothesis, he would have just taken the results and would have concluded that bigs are in fact more valuable than smalls. By changing that make the assumption that each "position" would be equally valuable, he also changes the marginal values for each different position. In that way 2pt scoresd by a center are less valuable than 2pt scored by a point guard. Makes no sense to everyone except to Berri and his followers.
That step also is very important in regard to the consistency. Because adjusting for position just means that boxscore entries do not have an intrinsic value, thus we can't use regression in order to determine such value. That makes the whole thing complete bogus.
Well, and at the end Berri does also assume that every player on a specific team has the same impact on the overall team defense. He is basically saying that each player is the same in terms of help or weakside defense. That is obviously complete nonsense, but Berri believes that this is true. That adjustment is rather important, because it controls the correlation between WP48 and overall team wins.
Can we show that all what I wrote is actually true? Yes, we can. Two things in which WP48 would need to perform worse are correlations to lineup performance level and in out of sample tests. WP48 can't say anything about the lineup performance, we can use a coin and will have the same ability to predict lineup performances as WP48. Other boxscore metrics such as Win Shares or my SPM are MUCH better regarding this. The other question is the out of sample test. Even when we take the real values in the respective season for rookies into account WP48 is worse than basically everything else except of PER. WP48 as a tool is worthless in terms of predicting outcome of games in the future.
Chronz wrote:Yet Berri is praised by the vast majority of people in HIS field for lending his work to be analyzed openly, which to my knowledge Hollinger/Rosenbaum do not do. Any particular reason why you value APBR community above his fellow economist?
Which fellow economists? Seriously, Berri is publishing his stuff in low-impact journals, sometimes Berri himself is in the editorial board or was. Berri is basically publishing his stuff without any kind of serious peer-review. He is not cited by any meaningful people and the only citations you can find are those from people in his staff or those who are critizing the method.
We also talk about a group of people which have not a strong mathematical background. They can use SPSS in order to make a regression, that's basically it. The APBR comunity consists of people with a PhD in math, physics, etc., a much stronger mathematical background than that of those economy people. As a geophysicist I had more math in my first 2 semesters than Berri in his whole life, just to make clear what kind of differences we are talking about.
Hollinger does not claim that his method is scientific, but his methods are all open. No idea how you can say otherwise. Rosenbaum has also described his method in great detail and the work by others is influenced by that. The difference between those +/- based people and Berri is: A lot of them are actually really working for teams and thus do not publish their results openly. But with enough math knowledge we can easily reproduce such things. Well, Aaron Barzilai is even kind enough to provide the results for all teams but the Grizzlies, because he works for the Grizzlies.