It's always tempting to build it yourself. Initially we built our own data store too, partly because it was a great geek challenge, and then went back to InnoDB and NoSQL solutions once we figured we can't compete with 1000's of open source devs and battle hardened products.
Our hardware bill for our entire server cluster, which we own, is only slightly more than Druid's monthly cloud bill. [My company is the largest real-time analytics provider on the web]
Few people realize that Redis was created by Salvatore to provide a real-time analytics product - and then it grew into so much more. Also InnoDB's clustered indexes are spectacular when it comes to answering questions like the one's the OP has posed.
How's Redis' overall persistence/performance? I'm always wary of having it handle critical data in fear that it will just up and lose some (like MongoDB has done for me), but I love it otherwise. Have you had any bad experiences with using it in production?
On their home page CB currently claim 1,977,762 visitors across all sites. We see the following across our network:
397,727,080 visits per month.
1,002,792,862 impressions per month.
209,727,427 absolute uniques per month.
We use a third party to report this data to ensure objectivity when we're doing reports for outsiders. This is from the period March 30 to April 29 2011.
ChartBeat's numbers are concurrent users. No idea how that translates on web but I'd expect it's more than 200m uniques - we do 120ish off 100k - 200k concurrents.
Our hardware bill for our entire server cluster, which we own, is only slightly more than Druid's monthly cloud bill. [My company is the largest real-time analytics provider on the web]
Few people realize that Redis was created by Salvatore to provide a real-time analytics product - and then it grew into so much more. Also InnoDB's clustered indexes are spectacular when it comes to answering questions like the one's the OP has posed.