Clickhouse has proven to also be a very capable database for logs and there are stacks that use it for log storage.
VictoriaLogs is very promising in this regard!
My next task is to get centralized logging going with Victoria Logs + Vector, I'll have to check this out once I get everything setup. I believe I can use LogQL with Victoria Logs but I haven't tried it out yet. https://docs.victoriametrics.com/victorialogs/logsql/
https://github.com/nklmilojevic/home/blob/main/kubernetes/ap...
https://github.com/nklmilojevic/home/tree/main/kubernetes/ap...
Here you have Vector in aggregator + agent mode and several sources. VictoriaLogs also recently added Grafana datasource so it is fairly easy to set it up:
https://github.com/nklmilojevic/home/blob/main/kubernetes/ap...
I'm a big fan of VictoriaMetrics as well and we use it extensively in my company at high scale.
{{kanban}}
# Todo
- [ ] item 1
- [ ] item 2
# In Progress
- [ ] item 1
- [ ] item 2
{{/kanban}}
[1] https://www.get-plume.com/
Ingested logs 24h: 428 Mil Ingested bytes 24h: 625GB Inser req/s: 6k/s
8vCPU, 16GB mem. Running standard-rwo PVC on GCP.
We have a couple of projects like this with similar usage and similar machine sizing.
Still running vmlogs-single, and we will until we see a need to move to vmlogs-cluster version.