Agreed. It was a mix of too ambitious standards for up-to-date data and poor configuration on my side.
Follow up on a previous post: [DISCUSS] Recent momentary outages
I've been working on a simple opt-in solution, primarily for Lemmy end users like me (but also helpful for admins), to easily check the status/health of their favourite instance.
๐ lemmy-meter.info
You can find the details of the implementation in 
lemmy-meter github repo.
โ @admins: would you be interested in adding your instance to lemmy-meter?
You don't need to do anything except confirming - I'll handle the rest. It should only take a few minutes for your instance to show up in lemmy-meter.
Out of the box it will send only 4 HTTP GET requests per minute to your instance. However that is totally configurable if it sounds too much or too little.
PS: I wasn't sure how to reach out to the admins short of messaging them individually.
sane defaults and a timeout period
I agree. This makes more sense.
Your name will be associated with abuse forevermore.
I was going to ignore your reply as a ๐ง given it's an opt-in service for HTTP monitoring. But then you had a good point on the next line!
Let's use such important labels where they actually make sense ๐
beyond acceptable use
Since literally every aspect of lemmy-meter is configurable per instance, I'm not worried about that ๐ The admins can tell me what's the frequency/number they're comfortable w/ and I can reconfigure the solution.
You can hit the endpoint /api/v3/site for information about an instance including the admins list.
Exactly what I was looking for. Thanks very much ๐
Thanks for the link. Had no idea about that.
I got to admit that your point about the presentation skills of the author are all correct! Perhaps the reason that I was able to relate to the material and ignore those flaws was that it's a topic that I've been actively struggling w/ in the past few years ๐
That said, I'm still happy that this wasn't a YouTube video or we'd be having this conversation in the comments section (if ever!) ๐
To your point and @krnpnk@feddit.de's RE embedded systems:
That's absolutely true that such a mindset is probably not going to work in an embedded environment. The author, w/o explicitly mentioning it anywhere, is explicitly talking about distributed systems where you've got plenty of resources, stable network connectivity and a log/trace ingestion solution (like Sumo or Datadog) alongside your setup.
That's indeed an expensive setup, esp for embedded software.
The narrow scope and the stylistic problem aside, I believe the author's view is correct, if a bit radical.
One of major pain points of troubleshooting distributed systems is sifting through the logs produced by different services and teams w/ different takes of what are the important bits of information in a log message.
It get extremely hairy when you've got a non-linear lifeline for a request (ie branches of execution.) And even worse when you need to keep your logs free of any type of information which could potentially identify a customer.
The article and the conversation here got me thinking that may be a combo of tracing and structured logging can help simplify investigations.
Thanks for sharing your insights.
Thinking out loud here...
In my experience with traditional logging and distributed systems, timestamps and request IDs do store the information required to partially reconstruct a timeline:
- In the case of a linear (single branch) timeline you can always "query" by a request ID and order by the timestamps and that's pretty much what tracing will do too.
- Things, however, get complicated when you've a timeline w/ multiple branches.
For example, consider the following relatively simple diagram.
Reconstructing the causality and join/fork relations between the executions nodes is almost impossible using traditional logs whereas a tracing solution will turn this into a nice visual w/ all the spans and sub-spans.
That said, logs do shine when things go wrong; when you start your investigation by using a stacktrace in the logs as a clue. That (stacktrace) is something that I'm not sure a tracing solution will be able to provide.
they should complement each other
Yes! You nailed it ๐ฏ
Logs are indispensable for troubleshooting (and potentially nothing else) while tracers are great for, well, tracing the data/request throughout the system and analyse the mutations.
I'm not sure how this got cross-posted! I most certainly didn't do it ๐คทโโ๏ธ
I cross-posted the same questions on Matrix and got the answer there.
The hook I'm using is invoked before the minor modes are setup - that's why it's being overridden. The suggestion was to have a hook function for each minor mode that I want to control. It's not clean but gets the job done.
Here's the working snippet:
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defun bahman/helm-major-mode.hook ()
(display-line-numbers-mode -1)
(puni-mode -1))
(add-hook 'helm-major-mode-hook
#'bahman/helm-major-mode.hook)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defvar bahman/display-line-numbers-mode.disabled-modes
'(vterm-mode erlang-shell-mode)
"Disable `display-line-numbers' for the specified modes.")
(defun bahman/display-line-numbers-mode.hook ()
(when (seq-contains-p bahman/display-line-numbers-mode.disabled-modes
major-mode)
(display-line-numbers-mode -1)))
(add-hook 'display-line-numbers-mode-hook
#'bahman/display-line-numbers-mode.hook)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defvar bahman/puni-mode.disabled-modes
'(vterm-mode)
"Disable `puni-mode' for the specificied modes.")
(defun bahman/puni-mode.hook ()
(when (seq-contains-p bahman/puni-mode.disabled-modes
major-mode)
(puni-mode -1)))
(add-hook 'puni-mode-hook
#'bahman/puni-mode.hook)
This is quite intriguing. But DHH has left so many details out (at least in that post) as pointed out by @breadsmasher@lemmy.world - it makes it difficult to relate to.
On the other hand, like DHH said, one's mileage may vary: it's, in many ways, a case-by-case analysis that companies should do.
I know many businesses shrink the OPs team and hire less experienced OPs people to save $$$. But just to forward those saved $$$ to cloud providers. I can only assume DDH's team is comprised of a bunch of experienced well-payed OPs people who can pull such feats off.
Nonetheless, looking forward to, hopefully, a follow up post that lays out some more details. Pray share if you come across it ๐
I think I understand where RMS was coming from RE "recursive variables". As I wrote in my blog:
Recursive variables are quite powerful as they introduce a pinch of imperative programming into the otherwise totally declarative nature of a Makefile.
They extend the capabilities of Make quite substantially. But like any other powerful tool, one needs to use them sparsely and responsibly or end up w/ a complex and hard to debug Makefile.
In my experience, most of the times I can avoid using recursive variables and instead lay out the rules and prerequisites in a way that does the same. However, occasionally, I'd have to resort to them and I'm thankful that RMS didn't win and they exist in GNU Make today ๐ IMO purist solutions have a tendency to turn out impractical.
Uh, I'm not sure I understand what you mean.
Update 1
Thanks all for your feedback ๐ I think everybody made a valid point that the OOTB configuration of 33 requests/min was quite useless and we can do better than that.
I reconfigured timeouts and probes and tuned it down to 4 HTTP GET requests/minute out of the box - see the configuration for details.
๐ A pre-release version is available at lemmy-meter.info.
For the moment, it only probes the test instances
I'd very much appreciate your further thoughts and feedback.