This is the latest in a long string of irritations that will have me looking for a replacement logging datastore post-haste. I've already replaced Logstash with Heka; I guess ES is next.
Good luck with that searching, and remember to share with HN.
I find ElasticSearch somewhat brittle (I need to restart it every few weeks or
so, because it stops accepting any data or queries), and I really do want to
replace it, since it's a memory hog (no data, and it already needs 230MB RAM),
but I haven't found any sensible log storage yet. All I got is this document
searching engine.
That was my experience exactly. The value of being able to search through ours logs was immediately apparent, so when it kept tanking every couple weeks it was fairly easy to talk my boss into paying a tiny bit more for Loggly. AFAICT Loggly is the ELK stack with a different theme applied to the UI, and I don't have to worry that it's down when I need it.
The preceding has been an unpaid endorsement for Loggly.
I would prefer much more to have something hosted by me and being open source
at the same time (I'm looking at you, Splunk). I'm desperate enough to write
such thing myself (and I'm already doing some research on indexing
semi-structured data), even if it would turn out not-too-scalable (I only have
~1.5GB/day of uncompressed logs), but I'm scared by the need of tolerance for
abrupt shutdowns and hitting a free disk space limit, i.e., database
recoverability.
Hey, I looked at Splunk in the last few days as well, but it seems really expensive once you want to get beyond 500MB a day. They don't have an OSS "community" version, do they (?) because I totally feel you on that.
Basically I need ELK, but I need it to be a lot more stable and documented better.
I suggest taking a look at the logz.io - it's essentially ELK as a service. We take care of many problems ES has and also have great features extra such as Alerting, Apps and Insights. I'm one of the core developers there so I'm fluent in pricing but last time I checked it's wasn't expensive and definitely worth your time and hosting costs. One the the great things about it is that you can simply migrate your dashboard and just ship to our Url.
230MB of Heap is very small for most ES usages, and is likely the reason for you needing to restart it often. I run some very large clusters and even with older versions of ES have seen them run billions of indexing and query operations without problems for 4-6 months without any restarts.
230MB RSS, and this is not "ES usage", it's just after starting it with no
data whatsoever. I've seen databases which use less memory than that when
working.
> I run some very large clusters [...] without problems for 4-6 months without any restarts.
Well, good for you. Being a sysadmin, I understand very well that my ES server
may be a specimen combining bad kernel compilation, broken JVM version,
unstable ES release, and ground water emanating bad energy, so I don't
complain much. Still, I would be very, very happy to see any alternative for
storing JSONified logs.
Which version? This reason I ask is that 2.x is a lot more stable for me than previous versions. I have to restart much less often and I haven't had ES "forget" my index since upgrading.
/aside: I get rewriting for the n-th time some utility (say a db driver) in language du jour, but (possibly poorly) replicating things like Lucene never made sense. Lucene is a fairly solid tech by all accounts.
