How Unreliable is the 'Eye-Test' for NBA analysis?

[NbaAllDay]

The Eye Test is a fairly unreliable way to evaluate a player, and i'll try and point out a few ways it can deceive you. I will use an example in a discussion I had plus some real world data and concepts.

Bottom 2 notes are the TLDR;

Note 1: This isn't an argument for purely "data is better than the Eye-Test" Data only analysis also has it's issues. This is to say that utilising the majority of you analysis on what your eyes see is an unreliable form of analysis.

Note 2: I believe using data and being able to verify it with your eyes is a solid starting point, however this balance is tilted more towards 'what actually happened' over 'What I have witnessed'

Example from my friend: Kobe is a better player than Lebron because he looked like a more complete/better player.(Please don't make this the topic of discussion it's just an example!)

The eye test is only as useful as the credibility of the person the eyes belong to:
- Reports say upwards of 35% of eye witness Testimony is wrong and upwards of 70% of wrongful convictions were based on Eyewitness Testimony alone.
- Should be 'eye opening' enough as to the glaring issues with basing analyis off of the eye-test.

We Percieve everything through our worldview and beliefs which is often highly biased:
- You'll see someone name a player as a 'killer' or 'a great scorer' and you'll find this leans into style over substance. What makes them a killer and why are they a great scorer? They could have great body control and a great looking shot but are they actual efficient? Do they actual take/make good shots. (Expanded example on my friends take)

- If you believe player x is a great shooter, you will more easily ovelook anything they do that goes against your belief. When someone shows you data you will be more likely to dismiss or look for reasons to debunk it.
This is actually something most encourage, however is the intent to try and dig deeper? Or simply refute it because it goes against your belief.

Eye-Test is often used to dismiss someone for "not watching them play or being around during the 'insert decade':
The eye test general requires you to know what you are looking at, watch a big portion of film (not just highlights), do a similar watch through for every player/team you are involving in your comparision.
- Even if this were possible, you are also subject to your own worldview and biases. You are also relying on a memory that is often scewed by Nostalgia and factors you arn't even aware of. A number doesn't change over time, our thoughts often do.
- Watching game/films after doing years/decades of analysis will more often yield a more balanced view, rather than relying on what you watched and how you perceived it 20 years ago.

Inability to understand the data leads to dismissing it for something more 'simple'
- Simply put, watching a game and analysing it from there is a lot easier than actually digging into the data and matching what you saw versus what actually happened.
- We do this with almost everything in life, so NBA doesn't need to be an exception for you, we all have limited time, however it is often the reason as to some steering clear. Most will go as far as box score stats and call it a day.
- Listening to Sports Media and their opinion is a lot easier than forming one based on your own research/thoughts.

There are more points I was going to cover but the post is getting long enough!

Would be keen to see what others think and how they go about analysing players/teams and the balance they use.

[NO-KG-AI]

Extremely. The number of people that can put aside their bias and preconceived notions, and the amount of people that actually are open to being wrong or correcting an opinion they held is miniscule.

[The-Stallion70]

It's reliable if someone is good at it but not many are.

Scouts in the high school level have to use eye test above all else in order to brief NBA teams on who's who. There isnt a ton of data and what bit there is isnt always reliable because the prospects don't play against each other in their leagues. They are always beating up on non-pro bound competition.

This is why the elite basketball invite camps are always a big intrigue because it's a rare opportunity to see the big prospects playing against each other.

[ChuckChilly]

Darko Milicic

[Wallace_Wallace]

Depends on how many times you use it. If you are only using one game, obviously it’s not enough evidence. The eye test to me is to see how someone reacts to a certain situation on a consistent basis.

If a QB can’t make certain throws against the blitz on a consistent basis, then defense can take advantage of that. Eye test would conclude, “He can’t do well under pressure.” or “He can’t go through reads.”

[SNPA]

So flawed a premise.

Data. You use it as though only numbers are data. That is wrong. You also use it as though this type of data has no “bias.” Also flat wrong.

Everything not quantitive you lump into one category and call it the eye test. So everything that impacts a basketball game that can’t be put on a spreadsheet is eye test.

[Tor_Raps]

I watch a stupid amount of basketball so I trust my eye test in that scenario. Now for things I don't watch enough, sure I'll use statistics.

[xxSnEaKyPxx]

I think you have to use the eye test to have any kind of opinion.

For one, defense is not well quantified by statistics. You can see blocks, you can see steals, and so on, but any of the more advanced metrics usually involve the entire team to one degree or another. So you really aren't going to see half the game of basketball from looking at a box score, at least not for an individuals defense. Even blocks and steals can be very misleading when talking about how good a defensive player is.

[zimpy27]

Depends on who's eye.

That's what makes it extremely unreliable IMO.

I would trust data only over 99% of posters on these boards though.

There are a few I trust on eye test because they provide lengthy reasoning and analysis on players from many different teams. But even these eye tests use data.as they should. Otherwise you rank a guy like Poole far better than he is based on moves he makes, finishing is key even something as simple as fg% is needed at a minimum.

[BlzMwt]

Another problem with the eye test is the fragility of of the human brain and memory.

Even if you dismiss any notion of bias (which you cannot), you still cannot retain and recall basketball that you saw ten years ago, the same as you can the basketball from last year, versus the basketball you've consumed over the last week.

Maybe macro level trends and major things stick out. But the details get washed away. And the details are what can really make the difference between surface level and indepth critical analysis.

[jazzfan1971]

Not only all that... But, even our concepts of what makes a player great differ.

For instance, I might say, 'Rudy Gobert is a better basketball player than Carmello Anthony.'
And you might reply, 'What??? Gobert can't do anything outside of 3 feet and Carmello was one of the top scorers in the league for a decade!'

And the problem here isn't eye vs. Stats. Although those two things are significant here. The main difference is the definition of what defines greatness being vastly different between two different individuals.

[WemBA Time]

You need the eye test and advanced stats to complement each other. If a player's advanced stats look good but maybe they don't look spectacular when they're on the court to the casual viewer, someone who knows the game well can give a good, in-depth explanation of how they're making they're impact. Could also be that advanced stats over or underemphasize a player's impact, and you need the eye test to put that into context. I remember a decade or so ago, there was a professor who developed his own advanced stat who called for guys like Brandon Knight or Kenneth Faried to get lots of minutes. They weren't looking great on the court but through his umbrella stat they looked like stars, but clearly that didn't pan out.

That said, all of your arguments are reasonable, but they don't capture situations when eye tests can actually be useful or when we become too reliant on advanced stats.

[NbaAllDay]

So flawed a premise.

Data. You use it as though only numbers are data. That is wrong. You also use it as though this type of data has no “bias.” Also flat wrong.

Everything not quantitive you lump into one category and call it the eye test. So everything that impacts a basketball game that can’t be put on a spreadsheet is eye test.

I never said it was unbiased, I even said it has its flaws. The numbers are what they are, they don't change. However people can use them in a biased way, or emphasise data that is of limited impact, relevance. It is however harder to do so than how one can manipulate their experience through the 'eye analysis'

The premise does not say that using data is the standard. It simply states that using the eye test for the majority of your analysis is flawed.

What else am I 'lumping in'? And how do you evaluate a player/team?

[NbaAllDay]

Not only all that... But, even our concepts of what makes a player great differ.

For instance, I might say, 'Rudy Gobert is a better basketball player than Carmello Anthony.'
And you might reply, 'What??? Gobert can't do anything outside of 3 feet and Carmello was one of the top scorers in the league for a decade!'

And the problem here isn't eye vs. Stats. Although those two things are significant here. The main difference is the definition of what defines greatness being vastly different between two different individuals.

Yes people who have conflicting views often have wildly different criteria by how they define greatness. Especially when it comes to player ranking.

I find those that go by mostly what they 'see' on the court will rate those who have more aesthetically pleasing games versus those who actually have impact (Style over substance).

This is not always the case, however you will find some of these people consistently ranking certain players higher because of it. Or lower if the opposite is true to them.

[NbaAllDay]

I think you have to use the eye test to have any kind of opinion.

For one, defense is not well quantified by statistics. You can see blocks, you can see steals, and so on, but any of the more advanced metrics usually involve the entire team to one degree or another. So you really aren't going to see half the game of basketball from looking at a box score, at least not for an individuals defense. Even blocks and steals can be very misleading when talking about how good a defensive player is.

Yes I think Defense can be something a little more nuanced in it's analysis. We continue to have stronger data points when evaluating however they can only take you so far.

Defense is actually a part of the game I believe you need to have more of a variety of information (including film) to properly be able to rank.

[SNPA]

So flawed a premise.

Data. You use it as though only numbers are data. That is wrong. You also use it as though this type of data has no “bias.” Also flat wrong.

Everything not quantitive you lump into one category and call it the eye test. So everything that impacts a basketball game that can’t be put on a spreadsheet is eye test.

I never said it was unbiased, I even said it has its flaws. The numbers are what they are, they don't change. However people can use them in a biased way, or emphasise data that is of limited impact, relevance. Agreed. It’s not the numbers fault. It’s the fault of the people collecting, displaying, analyzing and in general consuming quantitive data.
It is however harder to do so than how one can manipulate their experience through the 'eye analysis' Why?

The premise does not say that using data is the standard. It simply states that using the eye test for the majority of your analysis is flawed. If one spends more time watching a spreadsheet than basketball which one are they really a fan of most?

What else am I 'lumping in'? Everything not numerical. Literally…everything. And how do you evaluate a player/team? Peak.

[FarBeyondDriven]

it's THE most important means to analyze players and project NBA prospects. Seeing guys on the court actually playing tells the complete story. Someone can "officially" measure whatever but if you've been watching basketball for your whole life you can tell how tall they actually are in comparison to the lines on the court. Athleticism might be impressive in drills but it might not be functional. BBIQ needs to be seen to gauge and appreciate. Hustle and leadership is best gauged by watching full games. Advanced stats mean nothing without taking into account all the variables and context. The reason why so many of the supposed draft gurus fail so miserably every year is because they rely too much on box scores, "official measurements" and advanced stats instead of actually watching them play. My guess is the "tape" they watch are just highlights which can be very misleading. And relying so heavily on the opinions of other "scouts" hurts them as well. It's very common for groupthink to take over.

Of course you use stats as well. But the scouting communities increased reliance on it and often in place of eye-test is where things break down. I'll use Reed Sheppard as an example. I'm currently way lower on him as an NBA prospect than most of the "scouts" and people on the draft board. He's among my favorite college players but I don't think he's going to a starter on a great team. His supporters opposed to this opinion point to his advanced stats, 3 pt shooting and defensive metrics. And yes, if you base your opinions on these you'd come away thinking he's the next Steph Curry and one of the best college defenders of all-time.

But if you actually watch the games and see how he arrives at these numbers you'll start to see the forest from the trees. I'd wager 9 out of 10 of his three point shots are wide open because his team is loaded with multiple future NBA players with multiple capable slashers collapsing defenses and leaving guys like him, Dillingham, and Reeves wide open and he's often going against even smaller backcourts than him and if he's not wide open he can still get his shot off over them. Now, that doesn't take away from the fact that he's still making them at an incredible rate. But when he was the man in high school and not afforded wide open looks because he had to create for himself more as well, he struggled as a 3 pt shooter. Projecting forward into the NBA those wide open looks won't be there as much. There are better and longer athletes, and because of his size he'll need to alter his release speed and angle to account for better closeouts. It will most likely affect his percentages greatly. Since he can't create for himself in college he has no chance of doing so in the NBA. If you then take away or mitigate the one elite skill he has, three point shooting, then what is he?

If you watch the games you'll see that he struggles mightily on-ball to defend his position and has no chance at defending bigger guys. The steals and blocks are a product of him gambling and often-times getting away with it because of the strength of his opponent or because he's surrounded by NBA talent and a lot of size underneath to make up for the times it backfires. Though, he's been caught asleep and gambling quite a bit and it has helped lead to losses so it is a problem. He's a very smart player and he's got great hands and anticipation. It makes him a great team defender in college overall despite the numerous times his gambles or not paying attention has hurt his team. But once he gets to the NBA he'll get hunted so badly he'll be unplayable except as a bench unit guard kinda like McConnell or Pritchard. Those college blocks and steals won't mean squat.

[Goomba3666]

I’d rely on the eye tests over stats in a vacuum. Both are biased. How stats are defined are still based on human definitions who also have biases.

[Ryoga Hibiki]

Hwat does "eye test" mean for you? Are including actual film study or just casually watching a game?

[Mr Peanut]

I think it's complementary to advanced stats. Neither of them should really be assessed in isolation.

However all too often you'll get people on the GB posting about BPM, VORP, LEBRON, RAPTOR etc, and some of the statements they make and inferences they derive make it abundantly clear they have really watched the player/team in question.