Least compelling argument for or against advanced stats?

[Hambone93]

i agree with you in general. the problem is that stats will never be able to perfectly quantify something that isnt measureable. how do you measure intimidation, fatigue, emotion, distraction, and attitude? these are all things that need to be observed, as well as tested and measured.

the issue here isn't measuring these things, its measuring the effects. Ultimately what matters is the score on the scoreboard, not a teams attitude, or confidence, or whatever.

Also, everything is quantifiable, you just need enough information.

Something I like to do is ask people to list things that they feel are "unmeasurable" and list ways to measure them.

[Doctor MJ]

I would say the least compelling common arguments for advanced stats come from those who parachute in with a bit of academic knowledge and simply assume that they talking talking with a bunch of neanderthals. The underlying argument is always something along the lines that math is the way to solve problems and if you disagree with the specific approach then it must mean you don't understand the math. The most typical fatal symptom I see in these folks is that they will avoid get into discussions of the pros and cons of the stats they use and to me that reveals that basketball isn't really the issue. The issue is that they've learned a great deal of mathematical knowledge without learning to think scientifically, and as a result no matter where they go among the disciplines they are essentially tone deaf musicians with fast fingers.

I would say the least compelling common arguments against advanced stats come from those who simply believe they "know" the game too well for there to be anything actually relevant that they don't have the foggiest idea about. The most common refrain from them is "I watch the games!" of course, and the stats basically say to them "...but you don't actually understand what you see". The truth is the stats simply say "but there are details you can't completely process", which is what good observers already know, but the stubborn faith they have in what they see is already based on a drastic overestimation of their own faculties that simply has the benefit that other people can politely ignore the elephant in the room. What the stats do then is simply remove the ability to cover baseless bravado in fog, which leaves them with no choice but to attack.

[Tim Lehrbach]

I would say the least compelling argument for advanced stats comes from Dave Berri.

Edited for brevity.

[BoutPractice]

My issue with some of the advanced stats geekery is the same as that with the general "data" religion of the times: whenever it shows lack of depth, understanding, or "meta" thinking. Pretty much the same thing as economists developing sophisticated statistical models in the ultimate objective of unlocking "growth" without questioning what growth is in the first place, or why we should want it.

When you start relying on one single measure, there is the danger of forgetting what it is you're measuring and why you're measuring it. Measuring is valuing. When we say that stats rely on" assumptions" what we mean is that they depend on thinking you've already done, consciously or not (fast or slow?) on what's of value and what's not, what's better and what's worse. So it's not just lack of awareness of the potential flaws and blind spots in the models (although that can hurt too), it's lack of awareness of the "objective" of the model and where exactly you're coming from that's sometimes problematic.

Who's to say, for example, that it wouldn't be interesting to value the "effort" of athletes, regardless of, or in conjunction with the "results"? The physical act of grabbing a specific number of rebounds, putting up a certain number of shots, one after the other, etc. Isn't the pure accumulation of box score stats worthy of praise in its own way, kind of like a "wow" factor?

Or on the contrary, why not value "energy efficiency" and purposely devalue effort, just to see what it would do?

When we measure "averages", should we be happy with that, or do we think that consistency is something worth praising and throw variance into the mix?

Do we think, on the contrary, that "outbursts" or "outliers" are worthy of praise? Should a player get extra points in our evaluation if he's shown that he could put up X points and Y rebounds, while others have not? Those are real questions we can't completely dismiss without thinking, because by dismissing them we've already thought about it whether consciously or not. Just my two cents.


Edit: A couple of other thoughts for stats people, as a non-expert...

Regarding plus/minus measures, is it possible that the mere fact of a player coming off the court may, in and of itself, have an impact on the performance of the substitute unit, in a systematic way?

For instance, some kind of leadership effect, where a certain player has such an impact on the collective that teammates don't want to "disappoint" him. Or in a somewhat related mechanism but with more selfish incentives behind it, an "earning your minutes" effect, where the substitute feels that he needs to perform up to the standards set by the starter? Then units "replacing" that player would systematically outperform units "replacing" an average player, leading to lower apparent impact measures? So let's say the player's team is +5 when he's on the court and -2 when he's off the court, perhaps they would have actually been -5 had they been replacing someone else?

Another weird way this could happen might be if a selfish or ball dominant star has the team performing well enough through his individual dominance... and then, whenever he comes off the court, a "we can finally shoot" effect mitigates the impact of his substitution, so the team is still decent but in a different way. Players relish the opportunity to be more featured in the offense and make the most of it... but had they been replacing someone else, the "boost" might have not happened.

Also, there are obvious determinants of performance like contract years, but how do we measure things like motivation, especially in the regular season (those who don't believe in the statistical impact of motivation have never watched a Boris Diaw or Andrew Bynum), bipolar patterns of self-confidence/self-doubt (Roy Hibbert?), and the "playing up to the competition" effect?

Case in point, some mediocre to merely "very good" players will start hitting all sorts of shots against the LeBrons and Kobes of this world, especially in the playoffs. Call it the Jimmy Butler, or Jason Terry, effect. Is it because a LeBron is a bad defender (in other words, should we "blame" him for the unexpected performance of his opponent) or just got lazy that night, or is it because they're fueled by the desire to beat the best on national television?
I think with LeBron in particular there must be some of that effect going on. His opponents routinely hit tough, contested shots against him. He's got the right stance, does a correct contest, but they'll still hit it. You can clearly feel it, in the way they run, the way they behave, that the mere challenge is what gets them going. These hypercompetitors want to kill the King. They're saving this kind of performance for the King, not Michael Kidd-Gilchrist or Al-Farouq Aminu.

One last point: only someone who either plays or watches basketball could formulate those kinds of hypotheses.
A robot would not see the difference between one of those hypotheses and the hypothesis that a player's performance is impacted by the number of Top 40 Hits Madonna released in a given year.

[Tinseltown]

I would say the least compelling common arguments against advanced stats come from those who simply believe they "know" the game too well for there to be anything actually relevant that they don't have the foggiest idea about. The most common refrain from them is "I watch the games!" of course, and the stats basically say to them "...but you don't actually understand what you see". The truth is the stats simply say "but there are details you can't completely process", which is what good observers already know, but the stubborn faith they have in what they see is already based on a drastic overestimation of their own faculties that simply has the benefit that other people can politely ignore the elephant in the room. What the stats do then is simply remove the ability to cover baseless bravado in fog, which leaves them with no choice but to attack.

This is essentially Stan Van Gundy's argument against advanced stats and I believe that he makes an excellent point.

http://probasketballtalk.nbcsports.com/ ... onference/

Van Gundy has spent 30 years coaching basketball and several more years playing. He prepared for the upcoming season by watching all 82 games the Pistons played last season. So when he sees data that may have been gathered by someone who can't explain a box and one defense, he's going to be skeptical. Especially when that data contradicts everything that he's observed.

The model is only as good as the data that's put into it, which is why I'm not surprised that nearly every coach rejects advanced stats

[Doctor MJ]

I would say the least compelling common arguments against advanced stats come from those who simply believe they "know" the game too well for there to be anything actually relevant that they don't have the foggiest idea about. The most common refrain from them is "I watch the games!" of course, and the stats basically say to them "...but you don't actually understand what you see". The truth is the stats simply say "but there are details you can't completely process", which is what good observers already know, but the stubborn faith they have in what they see is already based on a drastic overestimation of their own faculties that simply has the benefit that other people can politely ignore the elephant in the room. What the stats do then is simply remove the ability to cover baseless bravado in fog, which leaves them with no choice but to attack.

This is essentially Stan Van Gundy's argument against advanced stats and I believe that he makes an excellent point.

http://probasketballtalk.nbcsports.com/ ... onference/

Van Gundy has spent 30 years coaching basketball and several more years playing. He prepared for the upcoming season by watching all 82 games the Pistons played last season. So when he sees data that may have been gathered by someone who can't explain a box and one defense, he's going to be skeptical. Especially when that data contradicts everything that he's observed.

The model is only as good as the data that's put into it, which is why I'm not surprised that nearly every coach rejects advanced stats

You're lumping a bunch of things together and then damning the whole thing based one particular issue.

The concern you're pointing to of Van Gundy's specifically are about whether people are actually recording the extreme player tracking data they say they are. Basically all the stats being mentioned here aren't based on that
extreme player tracking data, and so these concerns are absolutely irrelevant to anything that's been seriously discussed here.

Do you understand the issue? There is absolutely no leg to stand on to dismiss "advanced stats" based on this concern and the issue isn't with Van Gundy being wrong.

Now, I did read the article, and Van Gundy goes on to make another point:

To me, I think that a lot of the analytic stuff can be very useful, but if you’re using that in place of sitting down and watching film yourself and seeing what’s going on, you’re making a big mistake, And I don’t want to offend anybody, but I think one of the problems with analytics — I think it’s good; I used it, I love looking at it — but one of the problems is, there are a lot of people in a lot of organizations who don’t know the game, who all they know is analytics and as a result, that’s what they rely on. And they will use that to supersede what guys like us see with our eyes. And I think that’s a major mistake. There’s no substitute for watching film over and over and over again, and the only numbers I trust are the ones that my people believe.

So here's what I'll say:
-I respect Van Gundy.
-I don't think I could do his job (obviously).

-He's absolutely right that really bad things can happen if you just rely on the numbers, and if you know me I get really frustrated with some of the people who do. The most controversial stat here is RAPM, and one of the key people who makes the stat is someone who I frequently lambast because to me he is proven he doesn't understand basketball well enough and he should really be working for someone who does. Before that guys associated with other stats (Wins Produced and WINVAL - which is precursor to RAPM)) demonstrated they were too clueless to be an actual decision maker on an NBA team.

-When he speaks of analytics "superseding" observation, whether I agree with him depends on precisely what he means by that, but realistically I'm going to guess we'd diverge to some degree, but there's plenty we could agree on.

The danger I'd see with him is one I was already talking about, and it relates to just basic issues with the human brain. Van Gundy is vary better at watching basketball than I am, but he's still only human, and that means he's going to struggle to tally everything up. And then it's a question of whether he truly recognizes his strengths and weaknesses, or he doesn't.

General rule is that if someone is talking about something fine-grained that they saw, the type of stuff that is in scouting reports, there's every reason to think an expert could nail that and that the thing they are nailing isn't something random but something dead useful.

However, when people talk about summing it all up. There just aren't any short cuts. It's complicated. A Van Gundy can give you his gut feel about about who he prefers between two different players without looking at any numbers and he'll get it right most of the time, but his precision isn't going to be perfect. And at least to some degree Van Gundy knows this, otherwise he wouldn't be saying "analytic stuff can be very useful".

This is where the framing of things is key, because when someone like me sticks by something he learned from the data in face of conventional opinion he's seen as arrogant or possibly just crazy. "How can you claim to know more than the experts?" But the whole reason why NBA teams uses analytics is that you can learn stuff using them that the best basketball scouts miss, and when this happens whoever sees it in the data knows something the scout doesn't, and he'll continue to know something the scout doesn't until the scout listens to him.

[itstheepp]

I think the least compelling argument against advanced statistics is that some analysts use them poorly. SVG's argument isn't at all about analytics, it's about analysts. He pretty much praises the stats themselves.

The thing about historical data is that it's almost universally appealing because it will show you things that surprise about the past. But, that's kind of a trap, because it's greater value is predicting the outcome of decisions going forward. If teams don't have a culture that is making predictions, measuring the actual outcome and adjusting the approach, then I think that team probably isn't getting superb value from their analytics dept.

This is a key part of the picture of sports analytics that is hidden from view. When teams use analytics to their benefit, we don't really know, because those predictions are going to be considered a competitive advantage and kept secret.

[Celtics2014]

My least compelling argument for advanced stats is that Gerald Wallace and John Wall have almost identical combined Rapm numbers for 2013-2014

[blabla]

My least compelling argument for advanced stats is that Gerald Wallace and John Wall have almost identical combined Rapm numbers for 2013-2014

Wall is listed at least 2 points better in every RAPM source there is?

[NZB2323]

I remember before game 5 of the 2012 NBA Finals the studio guys were talking about how "stats guys" were saying that Wade hurt his team and shouldn't play in game 5. The best argument against this one is a statistical argument, that you'd take a sample size of the whole season where Wade played well over a sample size of 4 games where Wade played poorly. Instead they said, "This is why you can't look at stats to evaluate basketball."

The best argument is that there are some things that stats can't measure. Russel caring so much about every game that he threw up before it started. Cousey taking Havlicheck under his wing. Magic being the most enthusiastic, supportive teammate of all time. Jordan telling his teammates in practice, "If you don't hustle in practice, I'll kick your ass." Allen Iverson skipping practice. Shaq in the summer of 2006 finding a way to rub the heart monitors so he didn't have to work in practice and then telling his teammates about it. Derrick Rose busting his tail in practice when Thibbs is yelling at him so other players have to do the same. Jason Williams telling rookies the wrong thing to do in plays so that way he looks better than the rookies. Stats don't measure how valuable a player is in practice.

[turk3d]

Great thread. Kudos to all you guys for your contributions.