Stat muse - crazy new stats website

[Lionlifer]

I'm betting that those response times are simulated, but that UI is on point.

Looks like they have a winner on their hands.

[Devilzsidewalk]

Anyone know what kind of search engine they're using, the queries are coming back so quick. That's the dopest part of this site.

Anyone with technical background know how they accomplished something like this search query function? would be greatly appreciated

some sort of SQL I guess - you have all the players and all the stats and seasons, then when you type "which power forwards averaged 20+ ppg, 11 rebounds on the road last season?" it runs a select query something like season = 13/14, points >= 20, rebounds >= 11, road_or_home = road, position = PF, etc. I didn't read the site yet; not sure if the visualization software is their own, or if they're running something like Tableau to get the shot charts and such, but the site is pretty neat looking.

The queries just come back so fast. I figured it may be Mongodb, MySql or SQL. But the search query is quite well coded that it seem to be semantically.

They probably only have a very small amount of the data loaded on test tables - just enough to answer the question. Once you load all the players and all the years, it'll take some time, but nobody wants to see that in the demo.

[Onus]

some sort of SQL I guess - you have all the players and all the stats and seasons, then when you type "which power forwards averaged 20+ ppg, 11 rebounds on the road last season?" it runs a select query something like season = 13/14, points >= 20, rebounds >= 11, road_or_home = road, position = PF, etc. I didn't read the site yet; not sure if the visualization software is their own, or if they're running something like Tableau to get the shot charts and such, but the site is pretty neat looking.

The queries just come back so fast. I figured it may be Mongodb, MySql or SQL. But the search query is quite well coded that it seem to be semantically.

They probably only have a very small amount of the data loaded on test tables - just enough to answer the question. Once you load all the players and all the years, it'll take some time, but nobody wants to see that in the demo.

I noticed that Michael Adams came up on a search result for the 24+ ppg and 8+ ast per game. He played until 96, so it seemed like they were able to go back somewhat far. But then again who knows, but yea no one wants to see that in the demo

[Devilzsidewalk]

The queries just come back so fast. I figured it may be Mongodb, MySql or SQL. But the search query is quite well coded that it seem to be semantically.

They probably only have a very small amount of the data loaded on test tables - just enough to answer the question. Once you load all the players and all the years, it'll take some time, but nobody wants to see that in the demo.

I noticed that Michael Adams came up on a search result for the 24+ ppg and 8+ ast per game. He played until 96, so it seemed like they were able to go back somewhat far. But then again who knows, but yea no one wants to see that in the demo

Yea who knows what they got going on behind the scenes for their demo. Even if they're working off the full dataset, they can cache the queries I guess. I've worked on Oracle BI systems that did this, so the first time I'd run the query, it'd take forever, but subsequent runs would pull the data almost instantly.

[floppymoose]

This data doesn't change except for the current season. So you can index all the other seasons by every column that you allow searches on, making for very fast response times. It's not really much data. Modern servers can hold the data and the indexes in memory, so search time should be fairly instant.

[PowerSteele]

Have to pay for it later?

[thinktellectual]

Anyone know what kind of search engine they're using, the queries are coming back so quick. That's the dopest part of this site.

Anyone with technical background know how they accomplished something like this search query function? would be greatly appreciated

some sort of SQL I guess - you have all the players and all the stats and seasons, then when you type "which power forwards averaged 20+ ppg, 11 rebounds on the road last season?" it runs a select query something like season = 13/14, points >= 20, rebounds >= 11, road_or_home = road, position = PF, etc. I didn't read the site yet; not sure if the visualization software is their own, or if they're running something like Tableau to get the shot charts and such, but the site is pretty neat looking.

The queries just come back so fast. I figured it may be Mongodb, MySql or SQL. But the search query is quite well coded that it seem to be semantically.

1. They first need to "translate" from natural language to something they can use to query a database.
That part also takes some processing, and you probably need some dictionary so this part probably also needs some sort of database.

To me, that's the part that's more impressive than returning results ultra fast.
Given enough time and money for hardware, you can achieve the fast response times with a blend of code & data storage optimization (indexes in SQL, for example) and better hardware.

2. The second part, after building the query from the natural language input, would be to query the stats database itself.

Of course you can have super fast stuff. Google does, for example. But the question is, how much are you willing to pay in order to have it. A basketball stats web site doesn't have Google money and can't afford Google's datacenters. They also more than likely don't have Google's army of engineers that work to optimize everything.

So, regardless of what they use to hold the data in, to have such response times you need super optimized code + probably relatively expensive hardware.

BTW, MySQL IS SQL. Just like Oracle, DB2, MS SQL Server, PostrgreSQL and probably others that I forget about.

Now that I think about it, maybe it's feasible that the entire stats DB of the NBA is not that big, so maybe they have a machine with tons of memory and they keep everything in memory. This could explain the literally lightning fast response times, since the hard disks are usually the bottleneck.

3. There is a 3rd step to it, and it's the display. It looks simple, but I am not sure how processor intensive the building of the graphs is. My guess, though, is that it's HTML 5 and thus it runs on the user's computer, so it uses its processor and memory.

[KingDavid]

If they're able to do this for the NFL...Someone is going to be terrifyingly rich.

[Onus]

some sort of SQL I guess - you have all the players and all the stats and seasons, then when you type "which power forwards averaged 20+ ppg, 11 rebounds on the road last season?" it runs a select query something like season = 13/14, points >= 20, rebounds >= 11, road_or_home = road, position = PF, etc. I didn't read the site yet; not sure if the visualization software is their own, or if they're running something like Tableau to get the shot charts and such, but the site is pretty neat looking.

The queries just come back so fast. I figured it may be Mongodb, MySql or SQL. But the search query is quite well coded that it seem to be semantically.

1. They first need to "translate" from natural language to something they can use to query a database.
That part also takes some processing, and you probably need some dictionary so this part probably also needs some sort of database.

To me, that's the part that's more impressive than returning results ultra fast.
Given enough time and money for hardware, you can achieve the fast response times with a blend of code & data storage optimization (indexes in SQL, for example) and better hardware.

2. The second part, after building the query from the natural language input, would be to query the stats database itself.

Of course you can have super fast stuff. Google does, for example. But the question is, how much are you willing to pay in order to have it. A basketball stats web site doesn't have Google money and can't afford Google's datacenters. They also more than likely don't have Google's army of engineers that work to optimize everything.

So, regardless of what they use to hold the data in, to have such response times you need super optimized code + probably relatively expensive hardware.

BTW, MySQL IS SQL. Just like Oracle, DB2, MS SQL Server, PostrgreSQL and probably others that I forget about.

Now that I think about it, maybe it's feasible that the entire stats DB of the NBA is not that big, so maybe they have a machine with tons of memory and they keep everything in memory. This could explain the literally lightning fast response times, since the hard disks are usually the bottleneck.

3. There is a 3rd step to it, and it's the display. It looks simple, but I am not sure how processor intensive the building of the graphs is. My guess, though, is that it's HTML 5 and thus it runs on the user's computer, so it uses its processor and memory.

the first part is really interesting to me. I know a few companies are trying to do this for their particular search algorithms rather than the normal boolean, and this is most likely easier to do with basketball than other industries but it appears to be done really well.

Yes I know mySQL is SQL based, and all the other iterations of it. I'm not as technical as I want to be so I'm just trying to figure out what's out there and what their actual applications are. So thank you what you wrote helps explain a bit more

[thinktellectual]

The queries just come back so fast. I figured it may be Mongodb, MySql or SQL. But the search query is quite well coded that it seem to be semantically.

1. They first need to "translate" from natural language to something they can use to query a database.
That part also takes some processing, and you probably need some dictionary so this part probably also needs some sort of database.

To me, that's the part that's more impressive than returning results ultra fast.
Given enough time and money for hardware, you can achieve the fast response times with a blend of code & data storage optimization (indexes in SQL, for example) and better hardware.

2. The second part, after building the query from the natural language input, would be to query the stats database itself.

Of course you can have super fast stuff. Google does, for example. But the question is, how much are you willing to pay in order to have it. A basketball stats web site doesn't have Google money and can't afford Google's datacenters. They also more than likely don't have Google's army of engineers that work to optimize everything.

So, regardless of what they use to hold the data in, to have such response times you need super optimized code + probably relatively expensive hardware.

BTW, MySQL IS SQL. Just like Oracle, DB2, MS SQL Server, PostrgreSQL and probably others that I forget about.

Now that I think about it, maybe it's feasible that the entire stats DB of the NBA is not that big, so maybe they have a machine with tons of memory and they keep everything in memory. This could explain the literally lightning fast response times, since the hard disks are usually the bottleneck.

3. There is a 3rd step to it, and it's the display. It looks simple, but I am not sure how processor intensive the building of the graphs is. My guess, though, is that it's HTML 5 and thus it runs on the user's computer, so it uses its processor and memory.

the first part is really interesting to me. I know a few companies are trying to do this for their particular search algorithms rather than the normal boolean, and this is most likely easier to do with basketball than other industries but it appears to be done really well.

Yes I know mySQL is SQL based, and all the other iterations of it. I'm not as technical as I want to be so I'm just trying to figure out what's out there and what their actual applications are. So thank you what you wrote helps explain a bit more

No problem.
Just keep in mind that what I wrote are educated guesses.
But it does seem that I wasn't the only one thinking the same (in memory data, etc).

[thinktellectual]

If they're able to do this for the NFL...Someone is going to be terrifyingly rich.

Rich from what ?

Are millions of fans gonna pay for access ?
I doubt it.
Just like basketball, I assume (I don't follow the NFL) most fans are casual and most fans don't care much about stats.
In any case they won't care enough to pay a substantial amount for access to nicely displayed stats when I suppose there's a free equivalent of basketball-reference for the NFL (or in any case ESPN, Yahoo and others have the stats for free).

And if you're thinking about ad-revenue from clicks, you have to consider how much money it might cost to run the website (hardware, software etc).

Keep in mind that some of the most used websites had (and still have) problems monetizing the user-base.
I think Facebook operated at a loss for years before starting to turn some profit.
IIRC Twitter, Linkedin and many others have HUGE user bases, but can't properly monetize them.

[KingDavid]

If they're able to do this for the NFL...Someone is going to be terrifyingly rich.

Rich from what ?

Are millions of fans gonna pay for access ?
I doubt it.
Just like basketball, I assume (I don't follow the NFL) most fans are casual and most fans don't care much about stats.
In any case they won't care enough to pay a substantial amount for access to nicely displayed stats when I suppose there's a free equivalent of basketball-reference for the NFL (or in any case ESPN, Yahoo and others have the stats for free).

And if you're thinking about ad-revenue from clicks, you have to consider how much money it might cost to run the website (hardware, software etc).

Keep in mind that some of the most used websites had (and still have) problems monetizing the user-base.
I think Facebook operated at a loss for years before starting to turn some profit.
IIRC Twitter, Linkedin and many others have HUGE user bases, but can't properly monetize them.

Ah so generating money from a website is not as cut and dry as I thought it was. I was thnk more along the lines of NBA/NFL want to buy the rights off the creators to use this since it seems so simple and user-friendly.

When considering popularity, I think the ratio of diehards to casual fans are relatively the same between the NFL and NBA.

Enlightening as usual thinktellectual, thanks for the insight.

[Onus]

If they're able to do this for the NFL...Someone is going to be terrifyingly rich.

Rich from what ?

Are millions of fans gonna pay for access ?
I doubt it.
Just like basketball, I assume (I don't follow the NFL) most fans are casual and most fans don't care much about stats.
In any case they won't care enough to pay a substantial amount for access to nicely displayed stats when I suppose there's a free equivalent of basketball-reference for the NFL (or in any case ESPN, Yahoo and others have the stats for free).

And if you're thinking about ad-revenue from clicks, you have to consider how much money it might cost to run the website (hardware, software etc).

Keep in mind that some of the most used websites had (and still have) problems monetizing the user-base.
I think Facebook operated at a loss for years before starting to turn some profit.
IIRC Twitter, Linkedin and many others have HUGE user bases, but can't properly monetize them.

Nailed it on the head. Everyone assumes that they're huge corporations because everyone knows about them so they must be making lots of money. But many times it takes a long time for them to figure out how to monetize their user base.

LinkedIn for one has looked into various outlets and avenues to monetize but they haven't found the best way yet. They're pretty much throwing darts at a dart board and hoping one turns into the best.

[thinktellectual]

If they're able to do this for the NFL...Someone is going to be terrifyingly rich.

Rich from what ?

Are millions of fans gonna pay for access ?
I doubt it.
Just like basketball, I assume (I don't follow the NFL) most fans are casual and most fans don't care much about stats.
In any case they won't care enough to pay a substantial amount for access to nicely displayed stats when I suppose there's a free equivalent of basketball-reference for the NFL (or in any case ESPN, Yahoo and others have the stats for free).

And if you're thinking about ad-revenue from clicks, you have to consider how much money it might cost to run the website (hardware, software etc).

Keep in mind that some of the most used websites had (and still have) problems monetizing the user-base.
I think Facebook operated at a loss for years before starting to turn some profit.
IIRC Twitter, Linkedin and many others have HUGE user bases, but can't properly monetize them.

Ah so generating money from a website is not as cut and dry as I thought it was. I was thnk more along the lines of NBA/NFL want to buy the rights off the creators to use this since it seems so simple and user-friendly.

When considering popularity, I think the ratio of diehards to casual fans are relatively the same between the NFL and NBA.

Enlightening as usual thinktellectual, thanks for the insight.

Thanks for the kind words.


I don't know if the NBA/NFL would even be interested in buying such a website, no matter how cool it is.
Maybe ESPN, but not even them. All of them already have their own stats websites fully integrated with their other stuff, like fantasy etc. Just having the website would be nothing if it's not integrated with all their other stuff - which is a lot of work in itself, at which point, they might as well develop their own stuff. I don't think there's anything on that website that can be considered proprietary.

In case you want a little more info ....

There are a whole bunch of companies that are valued at BILLIONS that have never turned a profit after years of being active. The valuation is based mostly on the number of users.
A few examples:
Whatsup was sold for 19 BILLION
Twitter is worth about 30 BILLION (according to the value of their shares on the market)
Viber was sold for 900 million
Linkedin is worth about 25 BILLION (according to the value of their shares on the market)

I don't think any of these has ever turned a profit, and if they did, I am pretty sure it was way too small to justify these values.

Bigger companies buy them for their potential, and because sometimes they have ideas of how to monetize them in ways that might be impossible to do for the original owners.
For example, youtube was bought by Google with 1 billion, but they integrated their ad serving and I think they're making quite some money from it now. The original owners didn't have Google's reach and all the ecosystem that allows Google to do that (Google sells ads - or ad space, that's what gives them most of their money).

Here, for example, is an article on how Facebook might monetize WhatsUp:
http://www.forbes.com/sites/ericjackson ... ian-peers/

Some of these transactions work out, but many of them fail.

An example of failure is, IMO, EBay buying Skype for 2.5 BILLION (plus some bonuses, not exactly sure how much). They sold it later on for more or less the same amount. Might have been better to keep it, since Microsoft later bought it for 8.5 BILLION.

Even if some of these transactions look much better after a few years, IMO that's a new bubble forming up.
The more companies get sold for these ridiculous prices without ever turning profit, the more people will invest in them hoping for a big payday when Google/Facebook/Microsoft/Apple/whomever buys their company for billions. And I think at one point the market will suffer a crash - strictly the tech/web 2.0/social media companies market and companies will be more properly evaluated.

[MagnusPinus]

This mania with stats is bollocks. Too much emphasis on the single players. This is not basketball.

[RealRapsFan]

If they're able to do this for the NFL...Someone is going to be terrifyingly rich.

Rich from what ?

Are millions of fans gonna pay for access ?
I doubt it.
Just like basketball, I assume (I don't follow the NFL) most fans are casual and most fans don't care much about stats.
In any case they won't care enough to pay a substantial amount for access to nicely displayed stats when I suppose there's a free equivalent of basketball-reference for the NFL (or in any case ESPN, Yahoo and others have the stats for free).

And if you're thinking about ad-revenue from clicks, you have to consider how much money it might cost to run the website (hardware, software etc).

Keep in mind that some of the most used websites had (and still have) problems monetizing the user-base.
I think Facebook operated at a loss for years before starting to turn some profit.
IIRC Twitter, Linkedin and many others have HUGE user bases, but can't properly monetize them.

pay for access?


Current market cap:
Twitter = 26 Billion dollars
Linkedin = 25 Billion dollars

Zucherberg started facebook locally in 2004... by 2007 he was worth a billion


Rich from what you say? The IDEA that a lot of people are gonna click on your website.... thats all it takes. That you have to 'turn a profit' is nonsense.... you make a lot more money from people THINKING you will make a profit than from actually doing so.

Don't mistake these companies operating at a loss because they have invested so much back into their company with not being able to make money.....

[Bolivar]

Ah so generating money from a website is not as cut and dry as I thought it was. I was thnk more along the lines of NBA/NFL want to buy the rights off the creators to use this since it seems so simple and user-friendly.

Reporting technologies, interfaces and the possibilities in querying big data volumes have grown quite a lot in the past few years so there is nothing particularly innovating or proprietary in this. NBA, NFL, ESPN or whatever can invest in a bit of work hours in doing the exact same thing if they so wish.

That being said, it does look great in action and I'm pretty sure it will query the results fast. I'm kind of old-school so I probably still want to go to basketball-reference for my usual "all time points list" or "Dirk Nowitzki stats", but this or a similar site should supplement it perfectly.

Will also probably give good fodder for a thread of mostly irrelevant but kind of new and interesting findings from stats.

[thinktellectual]

If they're able to do this for the NFL...Someone is going to be terrifyingly rich.

Rich from what ?

Are millions of fans gonna pay for access ?
I doubt it.
Just like basketball, I assume (I don't follow the NFL) most fans are casual and most fans don't care much about stats.
In any case they won't care enough to pay a substantial amount for access to nicely displayed stats when I suppose there's a free equivalent of basketball-reference for the NFL (or in any case ESPN, Yahoo and others have the stats for free).

And if you're thinking about ad-revenue from clicks, you have to consider how much money it might cost to run the website (hardware, software etc).

Keep in mind that some of the most used websites had (and still have) problems monetizing the user-base.
I think Facebook operated at a loss for years before starting to turn some profit.
IIRC Twitter, Linkedin and many others have HUGE user bases, but can't properly monetize them.

pay for access?


Current market cap:
Twitter = 26 Billion dollars
Linkedin = 25 Billion dollars

Zucherberg started facebook locally in 2004... by 2007 he was worth a billion


Rich from what you say? The IDEA that a lot of people are gonna click on your website.... thats all it takes. That you have to 'turn a profit' is nonsense.... you make a lot more money from people THINKING you will make a profit than from actually doing so.

Don't mistake these companies operating at a loss because they have invested so much back into their company with not being able to make money.....

I fear you don't really understand how these things work.

Zuckerberg was worth a billion because his company was valued at billions DESPITE not turning a profit.
It was because the investors hoped the values of Facebook shares will keep rising.

Yes, a lot of money has been invested in Facebook, and there was little to no return for them.
If a company can't monetize their users and can't make sustainable profits, the growth is artificial, it's based on speculation. People buy shares thinking they will make their money back not from dividends, but from those shares increasing in value over time and them selling those shares later.

If there are no profits and this process keeps repeating where people buy shares because they expect the price to go up, the demand drives the prices further up. The whole process becomes a pyramid scheme (a Ponzi scheme, as known in the US). And it collapses when there aren't enough people willing to trust the price to go further up. No more people demanding stock means the value of stock will stop growing. When that happens, people who own stock will probably start selling, causing a sudden drop in stock value - basically a correction of the stock price to a realistic value. That's a bubble bursting.

Well, it's not that simple, and the bubble bursting will probably send the prices down too much initially, causing a lot of problems, before they recover to a more realistic value.

Same thing happened to the housing market. People kept buying thinking the values will forever go up. And then they didn't, and a lot of people lost a lot of money.


Just FYI, the value of the shares is supposed to be correlated to the dividends you expect to receive in the future, and not to the expected growth of the value of that share. It's the speculative nature of the capital market that's at least partially responsible for the bubble-burst, bubble-burst evolution of the stock market. If there were some rules to discourage speculative actions (for example if you buy some stock, you have to hold it for at least X months or pay a fine if you want to sell it in advance), I think the stock market would be much more stable and more conducive to REAL economic development.

PS: Just to make it clear, if you are ahead of the game and invest early and get your money out at the perfect time, sure, you'll be rich. But when the **** hits the fan, someone else will be poor. And if you don't get your money out at the right time, you might be the one that's poor.