Splunk, as a company, is a shell of its former self. All they care about is pimping themselves out to maximize profits to an extreme that only Dilbert can relate to, even at the expense of destroying a long term professional relationship over trivial matters. They are more than happy to kill a deal over a 5% disagreement rather than understand the needs of a Fortune 500 customer and negotiate.
They are mad because Cribl is good at transforming data before it ingested by Splunk, so as to reduce the amount of data that is indexed. Period.
Splunk ONLY RECENTLY released “Ingest Actions” to filter data post-ingest (to avoid indexing) for their SaaS product — something that has always been a mainstay of their on-premise “Enterprise” product. Their ONLY suggestion to filter data that we didn’t care to index in early 2021? Cribl. There’s literally no other reason for us to use Cribl.
I’ve been paying for Splunk since 2008 and can’t wait to get away from them. Their sales teams have decayed into unethical slimebags and I am trying everything in my power to not renew our contracts with them. This just sealed the deal.
Source: I cut checks to Splunk for $x,xxx,xxx yearly
I can add from the other side of the fence. I worked for a startup that was acquired by Splunk. They are everything listed here and worse on the inside.
My first few weeks at Splunk were very odd. They try to indoctrinate new hires with a barrage of "A-players" that continuously talked about how awesome Splunk was. Except... When I started Splunk was getting their ass kicked by cloud-first players that had recently come to market. Splunk's monolithic architecture wasn't well suited to be run as SaaS at the time and Splunk was burning cash and losing money on every customer that they suckered into moving away from their perpetual licenses into subscription hell. I left money on the table when I ran out the door less than 6 months later.
I'm curious what Splunk's long game is with this because they just told every F2000 that their bottom line is being chipped away by Cribl and friends. So if I'm an enterprising procurement department I'd be tossing Cribl or Rudderstack or whatever other data transformation preprocessor on the table alongside my renewal. Expand opportunity? If you put your ear to the tracks you can almost hear all of the account managers digging out missed quota excuses.
Splunk isn't innovative and hasn't been for a long time. Most of the employees saw the writing on the wall and went to Snowflake as soon as the opportunity presented itself. Splunk tried to capitalize on the security market by, basically, double charging customers for ES. Instead of delivering value it seems to be Splunk is just looking for ways to squeeze a few last drops of lemonade.
Interesting, I helped manage a splunk install at a fortune 200 about a decade ago. At the time the recommendation was to use syslog-ng to filter incoming logs before indexing. I just heard of cribl 2 weeks ago because the fortune 20 I currently work for is planning on switching to it. I didn't realize it was a massive shift like that, I just thought it was the corporation switching things just because they do that sometimes.
No comment about the company, but want to make clear as a buyer you understand the procurement and legal parts i.e. MFN or MFC.
If they do discount, even 5%, then it ripples across their accounts as a legal matter, esp at your scale. I was a buyer for some big companies, 8 digit, and the procurement office would only do a deal with MFN/MFC clause. They would also audit the supplier from time to time.
I totally understand that ripple effect and am very familiar with Most Favored X when it comes to unit pricing of a tangible good (e.g. xx,xxx physical servers with a particular SKU), but in this case we were talking about a SaaS product where overages were disputed. Nearly every vendor would jump at the chance to discount additional commitments or support at the ‘expense’ of waiving some past overages.
Sounds crazy, but Datadog. I’ve been hammering their product teams for years with specific use cases for the sole purpose of replacing Splunk. They recently migrated search technologies and are rapidly closing the gap. Plus, their exclusion features are instant and fantastic, and their C-suite replies to me when I escalate.
Elasticsearch simply couldn’t handle key collisions. We have hundreds of various apps across 5-10 different languages and frameworks where a key name may be reused as either a string or a hash or an integer or an array. If we can’t freeform search (which Splunk is EXCELLENT at), we just need to be able to transform the data beforehand. Datadog plans to do so with their recent acquisition of Vector.
This is the question. If you’re looking for APM well you’ve got great options but for those using Splunk in the security space (SIEM & SOAR) you’re screwed.
There’s no better SIEM alternative that deals with logs at scale.
Splunk recently screwed a friends Fortune 50 company. They didn't pay a bill on time (renewal negotiations) and Splunk without even contacting them just left all the logs from one of their instances on the floor. They lost everything for literally an entire country.
Ex-splunker here. I just started working at FeatureBase and would say, if your data is in Kafka, FeatureBase might be something to consider. It’s a crazy fast binary index built on Roaring bitmaps.
On the one hand, taking code from your employer and posting it to GitHub with the copyright notices removed is about as clear-cut a case of copyright infringement as you can get -- if they have evidence. Should be easy to confirm or deny by looking at version control history.
On the other hand, the patent claims referenced in the lawsuit seem to me like great examples of software patents that ought to be struck down for being uselessly over-broad. For example, I would love to hear an argument as to how the "'433 Patent" wouldn't be infringed by running Wireshark in a Kubernetes pod. That meets every single one of the claimed elements that Splunk is claiming Cribl is infringing.
From the lawsuit looks like the most clear cut evidence they have is:
- Founder publishing a private protocol definition to help in building for it
- Sales staff sending account and prospect info to their new cribl email addresses before leaving Splunk
- Engineers leaving Splunk with technical specifications, such as their newer S2S protocol versions
The patent stuff is kind of whatever, but all three of those items would be enough to establish some very clear damages. Cribls an exciting new player but they can't take shortcuts like this, if the allegations are founded.
Honest question, where is the line here? Obviously we all retain knowledge from previous jobs so what's the line between that and exactly copying a spec?
I think the line here is pretty straightforward: the contents of your mind are all yours. Anything beyond that (documents, source code, lists of prospects) is not.
Non-compete clauses will try to limit the usefulness of the "in your mind" knowledge by restricting the domains in which you can work post-departure. It's my understanding that such clauses are generally held to be unenforceable except in an acquisition scenario.
I think the line is drawn at actual stealing. In this case, they're not redesigning the protocol from memory or black box testing. Allegedly they took several specs of the protocol from former employees. Even then, apparently the founders were involved in some of the Patent filings from splunk that they are accused of violating. You cant claim IP for a company in the form of a patent and then turn around and re-implement that IP. You clearly believed it was patent-able since you patented it. There's ways of doing this (clean room dev) if you wanted to do that without infringement. (I do feel a lot of the patent claims in the lawsuit are typical generic weak software patents)
Really egregious is taking the sales data. Business analytics around leads, customer satisfaction, pricing, etc are not the same as retaining general knowledge. If you left and remember the point of contact you had at a customer, that's allowed (barring non-solicitation agreements). If you leave and you take a list of customers, data that the business has generated about them, etc, that was never yours and it's not your knowledge. It's clearly the business's and there's usually dozens of people involved in the creation. That's clearly theft, especially since it was never yours to begin with.
honestly, I've never moved from a role in one company to a role in a new company that directly competes with the role I had in my old company.
While I have jumped to competitors, I moved to roles that weren't in any form competition to my former team/role. That makes it easy, even if I would accidentally take things with me, I wouldn't be tempted to look at it, as there would be no point.
So yes, I take all my growth, knowledge and experience, but nothing that is really unique (say trade secrets) to old company would directly apply to my new role, so there has never been any problem. Once one is willing to jump to a competitor in a manner where you trade secret knowledge would benefit your role directly, one is creating a problem.
To me, unless there is a legal document you signed with your employer, there is no line. Even, IMO, IP is not `property` so that it cannot be used against. But that is another discussion.
Splunk is the best at what it does with no close competition.
I've been looking into Cribl and it seems their product has surpassed their competition as well but not in search, more in data summarization and log reduction, possibly before you ship it off to a more proper place like Splunk.
Splunk's cost makes it inaccessible to most people or companies. I mean, I work in infosec and I highly caution against Splunk because it is so amazing you will hate anything else but in security you need tons of otherwise rubbish data collected centrally sometimes and it will force you into a corner where you will say you can't afford to store that log you really should be storing. Better a crappy tool that can be used to find the logs you need than a nice tool that can only retain so much.
Cribl is supposed to help people reduce what they put i Splunk so they can keep using Splunk, it would have been nice if they partnered instead.
Graylog is another nice tool I like that is somewhat but only slightly similar to Cribl that was founded by a former Splunker out of frustration.
It has always been the Cadillac of search, and moreso with unstructured indexing (e.g. key collisions with different data structures. Foo = string vs foo = integer vs foo = array).
Your queries or infrastructure were not optimized. It’s very fast when optimized.
Very very fast, I can do an all time search on terabytes of data in seconds.
But you have to learn to use it, if you don't give it an index and a sourcetype that will slow it down, and like ES leading wildcards slow things down. The fastest searches are simple terms like a word or an IP.
40 minutes sounds exceptionally bad, but 5-10 minutes with splunk was totally common when I worked at Apple almost a decade ago, and I could never figure out why because I only ever used it for O(grep on a log file on disk) level operations. I was probably holding it wrong or maybe the infra team had misconfigured it, idk.
We recently transitioned to it at Notion and it’s been very fast, outperforming the previous log vendor substantially while offering better search and UX. If you used the on-prem version, the cloud version is quite a different experience.
Yes BUT it needs tuning. Splunk is complicated and takes continuous maintenance to optimize speed.
I work as a Splunk integrator and here's what I often see:
1. Customer installs Splunk with a qualified Splunk or third-party architect team. The deployment works well.
2. Customer adds infrastructure to the deployment. Splunk slows down. License costs go up.
3. Customer chooses between outside help or DIY. DIY rarely works.
4. Customer now needs outside help. Now Splunk is very slow and expensive, and now it will cost a lot to tune it.
Splunk, the company, is in a tough spot for several reasons: rotating c-level cast, unpopular changes to license model, bad acquisitions. The product is still best in class but tough to keep optimized.
> Splunk is the best at what it does with no close competition.
I'm with you. Splunk core - the indexing, automatic parsing, HA architecture, is unsurpassed. You can rebuild/duplicate parts of it but it's not going to come close to what Splunk can do, effortlessly, out of the box. I'm frustrated at the crud that Splunk has acquired which doesn't solve their customer's core problems. Splunk isn't well-rep in the network space. In my past I've worked for a huge tech company that was the darling of its day and Splunk business trajectory reminds me of that; we're within the start of the descent.
I read through the complaints in this thread, how it's slow, behemoth, hard to manage, copmlexities grow ... I've never experienced this problem. I've built and managed 3 Splunk clustered installations, in the 10sTB/day, and I will never use anything else. Sadly, that makes me only able to work for people able to afford the license :nervous laugh: So if you're made of money and want black car white glove data service, buy Splunk and hire people like me.
There's also Gravwell (https://www.gravwell.io) that competes head-on with Splunk and doesn't punish you for storing/ indexing more data. I'm on their board and knew the founders before I joined so I'm a bit biased but it's basically what if you wrote Splunk from scratch using modern tech.
Splunk is a great tool but expensive. I like splunk's aggregation feature very much. If it is server logs, it can aggregate and tell me how many http 500 errors I have, how many requests resulted in 404 etc. It can tell me top IP addresses where I am getting requests from, etc.
I want to take a CSV file and provide same functionality. Eg. Give user information on how many times each field occurs. For example, if it is a CSV file with cities, countries, continents, I want to aggregate and tell how many cities are in each country and how many countries are in each continent.
Is there an open source version of splunk I can modify? I tried logstash but it is not straight forward to work with. It still needs me to define schema everytime.
What you're describing sounds like Loki (Grafana's Prometheus inspired logging tool, which is super fast and cheap/easy, even though it sacrifices some flexibility to get there) Metric Queries: https://grafana.com/docs/loki/latest/logql/metric_queries/
We're building Matano (https://github.com/matanolabs/matano), an open source security lake platform. It's a different approach since we normalize logs from JSON, csv, etc, and ingest them into Apache Iceberg tables, but it allows for massive scale and joins, aggregations, etc using SQL.
Dang! Back in early days of AppDynamics, the founder who started AppDynamics after working at CA got hit by CA lawsuit, which lasted for a while but eventually got settled. Similar allegations. it was highly unpleasant and detrimental to the IPO preps. Some of my colleagues from there went to Cribl and sure hope they aren't going to be impacted, but they likely will will.
Our alerting solution, "OpterVics", was bought by Splunk. Since then it's been a shitshow - the service is running, but it's almost impossible to get a response from support.
They sent us an invoice for renewal in early August. I replied back (5 separate times) asking for the original contract (our ops department is tightening up on vendor management, didn't have it on file already); and we've heard nothing. Our service has continued to work despite not having paid (or signed a renewal), but we're switching to opsgenie.
Readit News
They are mad because Cribl is good at transforming data before it ingested by Splunk, so as to reduce the amount of data that is indexed. Period.
Splunk ONLY RECENTLY released “Ingest Actions” to filter data post-ingest (to avoid indexing) for their SaaS product — something that has always been a mainstay of their on-premise “Enterprise” product. Their ONLY suggestion to filter data that we didn’t care to index in early 2021? Cribl. There’s literally no other reason for us to use Cribl.
I’ve been paying for Splunk since 2008 and can’t wait to get away from them. Their sales teams have decayed into unethical slimebags and I am trying everything in my power to not renew our contracts with them. This just sealed the deal.
Source: I cut checks to Splunk for $x,xxx,xxx yearly
My first few weeks at Splunk were very odd. They try to indoctrinate new hires with a barrage of "A-players" that continuously talked about how awesome Splunk was. Except... When I started Splunk was getting their ass kicked by cloud-first players that had recently come to market. Splunk's monolithic architecture wasn't well suited to be run as SaaS at the time and Splunk was burning cash and losing money on every customer that they suckered into moving away from their perpetual licenses into subscription hell. I left money on the table when I ran out the door less than 6 months later.
I'm curious what Splunk's long game is with this because they just told every F2000 that their bottom line is being chipped away by Cribl and friends. So if I'm an enterprising procurement department I'd be tossing Cribl or Rudderstack or whatever other data transformation preprocessor on the table alongside my renewal. Expand opportunity? If you put your ear to the tracks you can almost hear all of the account managers digging out missed quota excuses.
Splunk isn't innovative and hasn't been for a long time. Most of the employees saw the writing on the wall and went to Snowflake as soon as the opportunity presented itself. Splunk tried to capitalize on the security market by, basically, double charging customers for ES. Instead of delivering value it seems to be Splunk is just looking for ways to squeeze a few last drops of lemonade.
Deleted Comment
If they do discount, even 5%, then it ripples across their accounts as a legal matter, esp at your scale. I was a buyer for some big companies, 8 digit, and the procurement office would only do a deal with MFN/MFC clause. They would also audit the supplier from time to time.
Deleted Comment
Elasticsearch simply couldn’t handle key collisions. We have hundreds of various apps across 5-10 different languages and frameworks where a key name may be reused as either a string or a hash or an integer or an array. If we can’t freeform search (which Splunk is EXCELLENT at), we just need to be able to transform the data beforehand. Datadog plans to do so with their recent acquisition of Vector.
This is the question. If you’re looking for APM well you’ve got great options but for those using Splunk in the security space (SIEM & SOAR) you’re screwed.
There’s no better SIEM alternative that deals with logs at scale.
Splunk recently screwed a friends Fortune 50 company. They didn't pay a bill on time (renewal negotiations) and Splunk without even contacting them just left all the logs from one of their instances on the floor. They lost everything for literally an entire country.
I mean EVERYTHING.
(This seems to be the repository in question, but it's been taken down: https://web.archive.org/web/20210104032001/https://github.co...)
On the other hand, the patent claims referenced in the lawsuit seem to me like great examples of software patents that ought to be struck down for being uselessly over-broad. For example, I would love to hear an argument as to how the "'433 Patent" wouldn't be infringed by running Wireshark in a Kubernetes pod. That meets every single one of the claimed elements that Splunk is claiming Cribl is infringing.
Presumably anyone with Wireshark could reverse it, so does it impart a significant advantage? Or is it just about control?
- Founder publishing a private protocol definition to help in building for it
- Sales staff sending account and prospect info to their new cribl email addresses before leaving Splunk
- Engineers leaving Splunk with technical specifications, such as their newer S2S protocol versions
The patent stuff is kind of whatever, but all three of those items would be enough to establish some very clear damages. Cribls an exciting new player but they can't take shortcuts like this, if the allegations are founded.
Non-compete clauses will try to limit the usefulness of the "in your mind" knowledge by restricting the domains in which you can work post-departure. It's my understanding that such clauses are generally held to be unenforceable except in an acquisition scenario.
Really egregious is taking the sales data. Business analytics around leads, customer satisfaction, pricing, etc are not the same as retaining general knowledge. If you left and remember the point of contact you had at a customer, that's allowed (barring non-solicitation agreements). If you leave and you take a list of customers, data that the business has generated about them, etc, that was never yours and it's not your knowledge. It's clearly the business's and there's usually dozens of people involved in the creation. That's clearly theft, especially since it was never yours to begin with.
While I have jumped to competitors, I moved to roles that weren't in any form competition to my former team/role. That makes it easy, even if I would accidentally take things with me, I wouldn't be tempted to look at it, as there would be no point.
So yes, I take all my growth, knowledge and experience, but nothing that is really unique (say trade secrets) to old company would directly apply to my new role, so there has never been any problem. Once one is willing to jump to a competitor in a manner where you trade secret knowledge would benefit your role directly, one is creating a problem.
I've been looking into Cribl and it seems their product has surpassed their competition as well but not in search, more in data summarization and log reduction, possibly before you ship it off to a more proper place like Splunk.
Splunk's cost makes it inaccessible to most people or companies. I mean, I work in infosec and I highly caution against Splunk because it is so amazing you will hate anything else but in security you need tons of otherwise rubbish data collected centrally sometimes and it will force you into a corner where you will say you can't afford to store that log you really should be storing. Better a crappy tool that can be used to find the logs you need than a nice tool that can only retain so much.
Cribl is supposed to help people reduce what they put i Splunk so they can keep using Splunk, it would have been nice if they partnered instead.
Graylog is another nice tool I like that is somewhat but only slightly similar to Cribl that was founded by a former Splunker out of frustration.
Last time I used it was almost a decade ago and it was rubbish, queries took 10-40 minutes to complete.
Your queries or infrastructure were not optimized. It’s very fast when optimized.
But you have to learn to use it, if you don't give it an index and a sourcetype that will slow it down, and like ES leading wildcards slow things down. The fastest searches are simple terms like a word or an IP.
I work as a Splunk integrator and here's what I often see:
1. Customer installs Splunk with a qualified Splunk or third-party architect team. The deployment works well.
2. Customer adds infrastructure to the deployment. Splunk slows down. License costs go up.
3. Customer chooses between outside help or DIY. DIY rarely works.
4. Customer now needs outside help. Now Splunk is very slow and expensive, and now it will cost a lot to tune it.
Splunk, the company, is in a tough spot for several reasons: rotating c-level cast, unpopular changes to license model, bad acquisitions. The product is still best in class but tough to keep optimized.
I'm with you. Splunk core - the indexing, automatic parsing, HA architecture, is unsurpassed. You can rebuild/duplicate parts of it but it's not going to come close to what Splunk can do, effortlessly, out of the box. I'm frustrated at the crud that Splunk has acquired which doesn't solve their customer's core problems. Splunk isn't well-rep in the network space. In my past I've worked for a huge tech company that was the darling of its day and Splunk business trajectory reminds me of that; we're within the start of the descent.
I read through the complaints in this thread, how it's slow, behemoth, hard to manage, copmlexities grow ... I've never experienced this problem. I've built and managed 3 Splunk clustered installations, in the 10sTB/day, and I will never use anything else. Sadly, that makes me only able to work for people able to afford the license :nervous laugh: So if you're made of money and want black car white glove data service, buy Splunk and hire people like me.
As an end user having used both to manage logs on a few dozen distributed applications I would never choose Splunk over Humio.
Deleted Comment
I want to take a CSV file and provide same functionality. Eg. Give user information on how many times each field occurs. For example, if it is a CSV file with cities, countries, continents, I want to aggregate and tell how many cities are in each country and how many countries are in each continent.
Is there an open source version of splunk I can modify? I tried logstash but it is not straight forward to work with. It still needs me to define schema everytime.
Thx!
https://github.com/grafana/loki might work for you. It’s not a drop in replacement for Splunk, FWIW.
They sent us an invoice for renewal in early August. I replied back (5 separate times) asking for the original contract (our ops department is tightening up on vendor management, didn't have it on file already); and we've heard nothing. Our service has continued to work despite not having paid (or signed a renewal), but we're switching to opsgenie.