These are well put together products with a host of great features, such as excellent UIs, sophisticated live searching via web interfaces and sometimes query languages and alerting. They require various levels of installation and they have costs, either through volume-based tiered systems or monthly payments.
For a bootstrapped business, this can be problematic, for instance when a surge of logs - indicating a possible important problem that needs to be solved - pushes volume on to another tier. Should the “log ransom” be paid?
Instead, I recalled from earlier times surely the simplest log watcher: Swatchdog [1]. It is rather venerable software. Its file history from its source download shows dates in 2015, but it was written much earlier - the 90s or possibly 80s by Todd Atkins [2].
We wanted to have alerts in Slack - the blog explains how we did it. In short: *very simply*. The code is available [3].
[1]: https://github.com/ToddAtkins/swatchdog
I like the opposite method - default alert on everything and develop an allowlist that quietens things down you don't want to hear. This is great for alerting you to unexpected things. And once in a while you actually want to know about some of those things :-)
It may sound very noisy but it's not too bad, especially once you're allowlist is setup. Logcheck[0] is a good tool for this and it runs by default at 2 minutes past each hour, emailing in a report of everything that isn't allowed. I think it matches some regex to what it deems higher threat events and those are always alerted on.
I'll conceed that this method isn't stellar for cattle! And we don't bother with it for things like kubernetes clusters or servers with semi-regular turnover for instance.
For pets and long lived servers that need looking after it's a good tool.
[0]https://logcheck.org/
At work we're using a combination of Splunk, Microsoft Sentinel, and Grafana Loki (can really recommend the latter combined with promtail).
There's some curious artefacts in this repo - the systemd service file calling usr/bin/env to a $PYTHON_EXECUTABLE (set in the env file to /usr/bin/python3) but the python script being called starts with '#!/usr/bin/env python' (which on my Debian unstable system launches python2).
Throwing executables under /etc/ is a .. contentious manoeuvre.
And the - so far as I can see - lack of support for handling journald output I suppose is a reflection of this sentiment:
> There’s endless criticism of systemd in the Linux community and I expect it will be properly superseded in the next few years. [...] It’s a curiously un-robust system.
How to grep the logs for errors and send an email
https://docs.fluentd.org/how-to-guides/splunk-like-grep-and-...
The slack plugin you could use instead of email:
https://github.com/sowawa/fluent-plugin-slack
The Python logging framework is quite flexible
Nothing to do with the article, but it would be nice if Slack (or IOS/Android) offered better customization of notifications. Errors and warnings are indistinguishable from regular messages on mobile, which means they often get lost in the shuffle.
For "unfussy" machines, I like to just set up rsyslog to send high priority syslogs right to email. [1]
Also, I used to use emails as notifications for many, many years...but lately have been using my own little matrix cli app/script[1]. I'm such a fan of matrix nowadays. Also, I stopped sending a full log file, and merely send only a snippet...Often, its enough for me to get the gist of what is happening. And, if certain thresholds exceed beyond crazy levels, then the messages become more urgent from my machines. :-)
[1] shameless plug: https://github.com/mxuribe/howler
Free, fast, versatile, adaptable to almost every case
Your company?
Anyway - not a "competitor" to the genuinely free and open-source Swatchdog.
Not sure what "genuinely free" means.