Hey everyone. I'm a co-founder / CTO of LogDNA. We were in Y Combinator's W15 batch working on a eCommerce marketing platform (PushMarket), only to realize we built a powerful logging system many of our friends wanted.
We had our “slack moment” and decided to pivot. We were frustrated with the current logging landscape and built LogDNA around 3 philosophies:
1. More Storage - give away ample storage so you can log everything
2. Longer Search - faster and longer search retention
3. UI/UX - a much cleaner experience to interact with your log tails
Your marketing seems to emphasize storage heavily. Is that the biggest pain point in logging today? If so, my guess would be that retention period could be a bigger differentiator than sheer gigabytes.
What's the killer use case or differentiator (vs. loggly+kibana+elasticsearch for example)? With Slack, it is integrations. If you could do something similar, like a bugsnag + new relic that aggregates all of our logs and notifies us when things happen (either individual incidents or aggregates like "nginx error log above 1 event/10s), you can have all my money.
Another idea would be to standardize transactions across the stack, so I could trace a request from nginx through rails/phoenix, to postgresql, etc. Again, take my money.
A better UI only helps me if I know when to look at it or what I'm looking for.
It's not the log storage that we care about, it's what we can do with those logs.
I've been through three logging vendors in the last two years (Loggly, LogEntries, now Sumo Logic) and I'm looking to move once again. The secret sauce is not the storage volume; our volume is relatively small: 4.3 GB/day. The secret sauce is the intelligent structuring of unstructured log data, analytics+alerting on the now-structured data, and live tail'ing of logs.
Sumo Logic does an okay job at the parsing/querying/alerting/analytics piece but live tailing is a brand-new feature for them and it has limitations. Sumo Logic's pricing is also frustrating--they don't apply volume discounting until you're pushing something like 500 GB of logs each month. Again, we need affordable analytics; storage is a small piece of the puzzle.
Honestly, our next move may be to an internal "HEK" stack: Heka, Elasticsearch, and Kibana. I've looked at the cost of doing this and it's actually more expensive when you consider what we'd spend to host a big ES cluster internally, but cost is not our #1 consideration, usability is.
If I were you, I would put a little emphasis on the security of your service. Logs can definitely contain sensitive data and companies rely on you to store that data in a "proper" way.
At least, as a possible customer, that is one of my biggest concern.
But maybe I'm just too focused on security given that our clients ask us about it pretty much every day. So as a possible vendor (for us or other SaaS businesses) and being part of our chain of trust, I'd expect some words on how you handle security and how much it is important for you. That is lacking on the website if I'm correct.
Anyway, congrats on the service! I think that there is room for LogDNA in the logging-warehousing market
Yes, thanks for the feedback. We'll be adding this and a FAQ section soon. We do encrypt everything in transit, end-to-end, just like most in our industry. We do not encrypt at rest at the moment but will be looking into it.
100GB/month doesn't matter when you only store the logs for 2 days... Why does your free plan have such a short retention time?
Is the 2 day retention time because you sync logs to the user's s3 bucket? If so then it would help conversion to say that next to the 2 days retention copy.
We had a specific audience in mind when we chose 2 days. We were targeting users who just needed to debug their apps, or a low traffic production app. For this, 2 days is plenty. The space is there mainly so they won't have to think about that 5GB/month may equal to 160mb/day or if a bug floods their logs and pushes them over their quota or dealing with overages.
If you care about retention, chances are you're also building a serious product and you want to rely on LogDNA to be around. For that, we hope you'll be on a paid plan.
We want to build the best product we can and by charging appropriately, we can afford better people.
Logging definitely sucks. It all sucks. Graylog. Elasticsearch and Kibana. Don't even get me started on Logstash which is some of the worst software I've ever used.
When you have a Docker/Kubernetes solution I will be (at another YC company) very interested in putting it through it's paces. Also I'd agree with others here that having a command line tail interface as well as a gui for browsing and savable queries would be bitchin.
Looks really great at first glance. Is there anything you guys do that differentiates yourself from other log management platforms (eg Kibana, Splunk, etc)
We've been using the product for a few months now after have looked at most of the alternatives. The biggest differentiators for me have been:
1. Really straightforward, scriptable installation
2. Intuitive view of the log data
3. Very quick and easy to select a time frame
4. UI is oriented to cope with lots of server creation/destruction (very useful for auto-scaling environments and not well handled in some of the other products)
We are very happy with the product - great job guys!
Technically, building a scalable infrastructure to ingest an order of magnitude of GBs more than other platforms was a bit challenging (and still is) but I think we're on our way to achieving that.
Are there any client libraries for LogDNA or does it just stream /var/log/* and other logs to LogDNA? Also, you provide APTs and RPMs, but no sources. Is there any way I can build LogDNA and run it on Arch Linux?
Client libraries are coming soon! We decided on an agent mainly for ease of installation and so we can self-update the agent, so you don't have to manage it (especially for auto-scaling environments).
We will be open sourcing our agent soon, we literally launched this week...some things we couldn't get to for launch.
And yes, it currently streams /var/log/* by default and any other paths your specify to the agent.
The CLI looks interesting. Can I use it to pipe logs to you?
That would be really useful for, say, running scripts and maintenance jobs on a bunch of boxes. That would allow me to collect data and inspect it later (or tail it live).
Oh, and I see you provide an OS X package for the tool. I'd vastly prefer a Homebrew package over that.
I'd love to have all logs stored from day 1 (ie unlimited retention).
Makes looking for bugs a million times easier, when you don't need to open up archived logs from S3.
Might be an interesting technical challenge to the keep search fast with 100gbs of logs.
They delete the logs after 2 days (free plan) and 60 days (most expensive plan available). The 100GB is only monthly transfer.
Would be nice to have a cloud log service with unlimited retention that's searchable, but for now self-hosting is the best choice using logstash [1] for logging and statsd [2] for metrics.
The more storage thing seems disingenuous . Splunk's biggest plan is 3x yours. It seems that your real offer is just price per storage atm. More for the same money. infact only splunk light competes with you on price but on not storage GBs.
I don't think we'll be targeting enterprise users like Splunk...especially since their featureset will take a long time to build out. We might be one day but not today.
Yes, you are correct, more value for your money is our current offer. We want to build a great product and to do that, you have to hire great people, so we came up with a pricing that'll hopefully achieve that.
But it's not set in stone and we'll adjust just like any other company.
It's just astounding how web sites still insist on replacing the normal, perfectly fine scroll behavior with weird, buggy, slow, crazy, half-assed, dizzying, and totally unnecessary custom scroll implementations.
* Can or will you add name value pairs? Logging for us is no longer just a single line of text. We have attached metadata. Kibana handles this nicely.
* How good is your security? This is actually our biggest reason why would not use a service like this. Not that we put passwords or user sensitive data in logs we still have to keep them extremely secure due to compliance.
I'm sure you have decent security but I would recommend emphasizing that and/or proving it.
Yes, just send us JSON in a line and we'll parse it. You can actually search on that today, we just haven't made a way for you to see the parsed fields yet (coming soon!)
We're fully encrypted end-to-end in transit (but not at rest, currently).
Yeah we need documentation mainly and couldn't get that in time for launch.
I'd recommend making your pricing model simply per GB and then on a rate tier, don't play games with your customers. The quota system ends being a game, and simply ends up wasting management's time trying to ensure the company isn't either getting charged for overages or not using up the quote they have. I'd quite frankly not recommend companies like Sumologic solely based on the massive amounts of time they waste with these things.
Our team at Pantelligent has been happily using the LogDNA beta for the last few months. The product is well designed and easy to use for many members across engineering, devops, and marketing roles. Very impressed so far, and this is only the beginning of where this product can go! Setup takes about 2 minutes -- give it a try today.
Thanks for the kind words and it's awesome to have your team using our product. Your product feedback on how we can make this platform better has been golden!
Question: Why does a new startup/solution for this pop up every few months? Why is this space so difficult?
I've definitely thought about building this myself, but haven't gotten around to it. It's just sending some text to a server, isn't it? What's so hard about it?
Obviously I'm missing something. I'd love to know what :)
I wasn't aware there were several startups every few months in logging. For us, it was an interesting space. Lots of changes happening.
Yes you are somewhat right. It is sending text to a server. But then you send more text and more text and you need to search for all this text in a split second. If you've ever run your own ELK stack for example, you'll realize quickly that you're spending most of your time scaling elastic search to handle the data volume. There is a challenge to doing that.
It's idiotic. You're not bringing in any major differentiation. Your idea will be implemented by your competitors in less than a week and you will have no competitive advantage and way less clients.
What AWS stack will let you build something like this? I don't think I would trust a 3rd party with all of my log data for...I can't really see what the benefit of moving to cloud would be, a log file sits on a local disk or if you had to move it to the cloud you could use S3 or SQS even no?
Nothing on AWS will give you the UX that LogDNA is delivering. And a good experience working with logs can be a huge productivity and visibility boost to your team.
But the infrastructure components are certainly there on AWS.
CloudWatch Logs will ingest tons of data at minimal cost. It has simple API for searching for events, but it's extremely slow to return results if you have lots of data in a LogGroup. It has a simple API to get the latest events but it has a 10s delay and its a bit tricky to get all the events from all the LogStreams in the right order.
Kinesis will ingest tons of data at minimal cost, and let you stream it back in real time.
Lambda can subscribe to all the CloudWatch Log or Kinesis Events for parsing, filtering, and forwarding.
AWS ElasticSearch can ingest all the logs and give you a richer query language and visualization tools.
I've been building all this into Convox, an open source AWS toolkit. Take a look at this CloudFormation template for an example of how to configure a CloudWatch Log Group and Lambda filter in your own AWS account:
Mainly so you can avoid tailing or grepping from 1 server at a time, or setting up an elaborate log shipping method of your own. We'll do that for you. :)
Another benefit is if you have many services then viewing the logs in timestamped order can make debugging across services far easier. This is really quite difficult without a single service aggregating and indexing all your logs
It gets especially useful when you start to get unique ids across "jobs". Imagine a single flow of work hits 10 services, but you can search for job_id=123 and get all the logs for the job . Now its easier to see what went wrong and where in the flow.
Other than the post by their founder above, I'm not really familiar with them. But overall, the 3 points I mentioned in my opening comment is generally similar. But to really find out if you'll like us, you should just try installing us. We have a working live demo on https://logdna.com
Readit News
We had our “slack moment” and decided to pivot. We were frustrated with the current logging landscape and built LogDNA around 3 philosophies:
1. More Storage - give away ample storage so you can log everything
2. Longer Search - faster and longer search retention
3. UI/UX - a much cleaner experience to interact with your log tails
I’m happy to answer any questions you may have!
What's the killer use case or differentiator (vs. loggly+kibana+elasticsearch for example)? With Slack, it is integrations. If you could do something similar, like a bugsnag + new relic that aggregates all of our logs and notifies us when things happen (either individual incidents or aggregates like "nginx error log above 1 event/10s), you can have all my money.
Another idea would be to standardize transactions across the stack, so I could trace a request from nginx through rails/phoenix, to postgresql, etc. Again, take my money.
A better UI only helps me if I know when to look at it or what I'm looking for.
It's not the log storage that we care about, it's what we can do with those logs.
I've been through three logging vendors in the last two years (Loggly, LogEntries, now Sumo Logic) and I'm looking to move once again. The secret sauce is not the storage volume; our volume is relatively small: 4.3 GB/day. The secret sauce is the intelligent structuring of unstructured log data, analytics+alerting on the now-structured data, and live tail'ing of logs.
Sumo Logic does an okay job at the parsing/querying/alerting/analytics piece but live tailing is a brand-new feature for them and it has limitations. Sumo Logic's pricing is also frustrating--they don't apply volume discounting until you're pushing something like 500 GB of logs each month. Again, we need affordable analytics; storage is a small piece of the puzzle.
Honestly, our next move may be to an internal "HEK" stack: Heka, Elasticsearch, and Kibana. I've looked at the cost of doing this and it's actually more expensive when you consider what we'd spend to host a big ES cluster internally, but cost is not our #1 consideration, usability is.
At least, as a possible customer, that is one of my biggest concern.
But maybe I'm just too focused on security given that our clients ask us about it pretty much every day. So as a possible vendor (for us or other SaaS businesses) and being part of our chain of trust, I'd expect some words on how you handle security and how much it is important for you. That is lacking on the website if I'm correct.
Anyway, congrats on the service! I think that there is room for LogDNA in the logging-warehousing market
Is the 2 day retention time because you sync logs to the user's s3 bucket? If so then it would help conversion to say that next to the 2 days retention copy.
If you care about retention, chances are you're also building a serious product and you want to rely on LogDNA to be around. For that, we hope you'll be on a paid plan.
We want to build the best product we can and by charging appropriately, we can afford better people.
When you have a Docker/Kubernetes solution I will be (at another YC company) very interested in putting it through it's paces. Also I'd agree with others here that having a command line tail interface as well as a gui for browsing and savable queries would be bitchin.
Looks really great at first glance. Is there anything you guys do that differentiates yourself from other log management platforms (eg Kibana, Splunk, etc)
1. Really straightforward, scriptable installation 2. Intuitive view of the log data 3. Very quick and easy to select a time frame 4. UI is oriented to cope with lots of server creation/destruction (very useful for auto-scaling environments and not well handled in some of the other products)
We are very happy with the product - great job guys!
Technically, building a scalable infrastructure to ingest an order of magnitude of GBs more than other platforms was a bit challenging (and still is) but I think we're on our way to achieving that.
We will be open sourcing our agent soon, we literally launched this week...some things we couldn't get to for launch.
And yes, it currently streams /var/log/* by default and any other paths your specify to the agent.
That would be really useful for, say, running scripts and maintenance jobs on a bunch of boxes. That would allow me to collect data and inspect it later (or tail it live).
Oh, and I see you provide an OS X package for the tool. I'd vastly prefer a Homebrew package over that.
> curl: (7) Failed to connect to heroku.logdna.com port 443: Connection refused
Might be an interesting technical challenge to the keep search fast with 100gbs of logs.
Would be nice to have a cloud log service with unlimited retention that's searchable, but for now self-hosting is the best choice using logstash [1] for logging and statsd [2] for metrics.
[1] https://github.com/elastic/logstash
[2] https://github.com/etsy/statsd
Yes, you are correct, more value for your money is our current offer. We want to build a great product and to do that, you have to hire great people, so we came up with a pricing that'll hopefully achieve that.
But it's not set in stone and we'll adjust just like any other company.
Dead Comment
* Love the price
* Can or will you add name value pairs? Logging for us is no longer just a single line of text. We have attached metadata. Kibana handles this nicely.
* How good is your security? This is actually our biggest reason why would not use a service like this. Not that we put passwords or user sensitive data in logs we still have to keep them extremely secure due to compliance.
I'm sure you have decent security but I would recommend emphasizing that and/or proving it.
Yes, just send us JSON in a line and we'll parse it. You can actually search on that today, we just haven't made a way for you to see the parsed fields yet (coming soon!)
We're fully encrypted end-to-end in transit (but not at rest, currently).
Yeah we need documentation mainly and couldn't get that in time for launch.
I've definitely thought about building this myself, but haven't gotten around to it. It's just sending some text to a server, isn't it? What's so hard about it?
Obviously I'm missing something. I'd love to know what :)
I wasn't aware there were several startups every few months in logging. For us, it was an interesting space. Lots of changes happening.
Yes you are somewhat right. It is sending text to a server. But then you send more text and more text and you need to search for all this text in a split second. If you've ever run your own ELK stack for example, you'll realize quickly that you're spending most of your time scaling elastic search to handle the data volume. There is a challenge to doing that.
You've been through YC? How the hell did you graduate? Because you just did mistake number 4: http://paulgraham.com/startupmistakes.html
But the infrastructure components are certainly there on AWS.
CloudWatch Logs will ingest tons of data at minimal cost. It has simple API for searching for events, but it's extremely slow to return results if you have lots of data in a LogGroup. It has a simple API to get the latest events but it has a 10s delay and its a bit tricky to get all the events from all the LogStreams in the right order.
Kinesis will ingest tons of data at minimal cost, and let you stream it back in real time.
Lambda can subscribe to all the CloudWatch Log or Kinesis Events for parsing, filtering, and forwarding.
AWS ElasticSearch can ingest all the logs and give you a richer query language and visualization tools.
I've been building all this into Convox, an open source AWS toolkit. Take a look at this CloudFormation template for an example of how to configure a CloudWatch Log Group and Lambda filter in your own AWS account:
https://github.com/convox/rack/blob/master/api/dist/kernel.j...
Also, we just use AWS for EC2 mainly.