Prometheus/VictoriaMetrics/Grafana are pretty good, had no issues with it and there's an exporter for damn near anything. They're pretty easy to custom write too.
this post was submitted on 23 May 2024
21 points (92.0% liked)
Selfhosted
40137 readers
593 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 1 year ago
MODERATORS
But these 3 are all about metrics, right? While they’re great to monitor and analyse numbers (ping times, disk space, memory, etc.), they aren’t that great with e.g. plaintext error messages in log files. That’s how I remember it from a few years ago, at least.
Grafana/Loki does logs. Still early days for me but it’s solid so far.
Cheers! I've heard of Prometheus/Grafana but VictoriaMetrics was a new one. Gonna look into it!
Yeah VictoriaMetrics is the new favorite since Influx keeps reinventing their wheels and trying to move everyone to the cloud.
May have to explore this, I still run influxdb and telegraf for a push metrics operation instead of pull like prom. Things have been smooth for a while but a couple months ago disk temps and metrics stopped working with no errors or missing plugins
I’ve been using Zabbix for ages now. It has issues but I got used to it.
Zabbix still remains a good choice imo. It works fine with Grafana et now the Zabbix-grafana plugin is officially supported by Grafana.
Zabbix without Grafana is pretty weak in term of visualization.
Uptime Kuma is great for simple up/down and web checks. Librenms is worth looking at too for other metrics.
I'll have a look! Cheers!
I've used Zabbix for that before. I hope you like SNMP, though!
I've used SNMP a lot together with nagios so I should be able to handle it. :D
I'm using CheckMk for pretty much all of that. Personally I found zabbix to have too much overhead.
For me it’s the other way around. In Check_MK I was constantly writing new custom checks and it was all manual code and overall felt like Nagios on steroids (what it was back then) - just not in a good way.
In Zabbix you can do everything in the UI without messing around in the file system. And things like translating SNMP results to readable text works throughout the system without having to include a Python file and then call it from within your various other checks. All the alerting logic can be clicked together and easily amended in the UI. It’s so much more comfortable once you’ve figured it out.
I used to use MQTT, static_status and Healthchecks.io, and have that data passed through to Home Assistant, but it started to get pretty cumbersome as the amount of machines I had grew.
I now use just Zabbix and HealthchecksIO. I did need to spend some time writing new templates for some additional data I wanted to collect (like SMART data for SSDs that provide health metrics in non-standard attributes, and HealthchecksIO so I could see the status of various checks on my zabbix dashboard)
Zabbix also has some additional features I found appealing, like proxies that can continue recording data when the main server is down, and built in encryption. Some checks like open ports/icmp responses etc can be checked using either the local agent, the remote server, or both, which helps quickly diagnose things like firewall config issues.
I did look at some other solutions, but I wanted something integrated to hit the ground running. Mobile apps are very limited, and there is no official one to my knowledge. I use Moobix which I don't believe is FOSS - but I could be wrong there
Try each solution out and see what works best for you!
You know you can basically implement Healthchecks.io completely in Zabbix using zabbix-sender or any compatible implementation of it? (Or find a better way, e.g. querying the timestamp of a logfile or even check the logfile for "OK" or "ERROR" lines... lots of ways possible.)
I use netdata (the FOSS agent only, not the cloud offering) on all my servers (physical, VMs...) and stream all metrics to a parent netdata instance. It works extremely well for me.
Other solutions are too cumbersome and heavy on maintenance for me. You can query netdata from prometheus/grafana [1] if you really need custom dashboards.
I guess you wouldn't be able to install it on the router/switch but there is a SNMP collector which should be able to query bandwidth info from the network appliances.
Gonna check it out!
Is it easy to setup automatic responses to the alerts, f.e. restarting a service if it isn't answering requests in a timely manner?
Have you used it together with Windows Servers too?
Windows Servers
No
setup automatic responses to the alerts
It should be possible using script to execute on alarm = /your/custom/remediation-script https://learn.netdata.cloud/docs/alerts-&-notifications/notifications/agent-dispatched-notifications/agent-notifications-reference. I have not experimented with this yet, but soon will (implementing a custom notification channel for specific alarms)
restarting a service if it isn’t answering requests
I'd rather find the root cause of the downtime/malfunction instead of blindly restarting the service, just my 2 cents.
Also uptime kuma for fast and easy up/down, web services, etc.