Between 19:45 UTC and 19:50 UTC, there was a mistake in how information was stored temporarily (cached) on Beehaw. This mistake could have allowed some people to see and use other people's accounts without permission.

If you were using the website during that time, please check that your account settings and email address are still correct. Also, make sure that any posts or actions you made during that time are still connected to your account.

It's important to note that we don't have any proof that this error was actually used by anyone to do anything bad during the short time it happened.

top 16 comments

sorted by: hot top controversial new old

[–] admin@beehaw.org 16 points 1 year ago (1 children)

I can verify that @Penguincoder@beehaw.org is one of our volunteer sysadmins and this incident was witnessed by me.

[–] Helix@beehaw.org 8 points 1 year ago (1 children)

same, same :p

[–] admin@beehaw.org 9 points 1 year ago

I can, also, verify that @Helix@beehaw.org is another volunteer sysadmin here at Beehaw.

[–] argv_minus_one@beehaw.org 15 points 1 year ago

That's one of the two hard problems of programming: cache invalidation, naming things, and off-by-one errors.

[–] fishy_2_0@beehaw.org 8 points 1 year ago

thanks for alerting everyone to this aswell as reacring quickly

[–] Cougar@beehaw.org 6 points 1 year ago (1 children)

Do we know what could have caused this cache error?

[–] Penguincoder@beehaw.org 10 points 1 year ago (1 children)

Yes; configuration settings for the web server involving improving performance. Those settings have been changed back to the previous, non-issue ones. So this should not occur at this time, or again.

[–] Cougar@beehaw.org 10 points 1 year ago (1 children)

Is this a mistake that's easy to do for an inexperienced instance admin or just a consequence of too much fiddling and shouldn't be an issue for other instances?

[–] Penguincoder@beehaw.org 14 points 1 year ago (1 children)

Was a result of too much fiddling. Attempting to gain even better performance from a bottleneck issue due to recent user influx. It was not an error in the Lemmy instance or Lemmy-UI but rather the web server front-end misconfiguration.

[–] karce@beehaw.org 6 points 1 year ago* (last edited 1 year ago) (2 children)

Thanks for working through these issues and improving performance of the website! Very appreciated. I've been tempted recently to create my own Lemmy instance, was this a problem with an nginx configuration option? How much does Beehaw deviate from a standard Lemmy deployment?

Feel free to answer vaguely if you don't feel comfortable with giving away the details : )

[–] Helix@feddit.de 6 points 1 year ago (2 children)

was this a problem with an nginx configuration option?

Basically, this was proxy_cache_key being configured incorrectly. If you don't use the proxy_cache you should be fine.

The only thing we changed from the norm is ulimits and some nginx settings. If we figure out what works well, we'll probably create a post about how to host lemmy. If you stick to the defaults, you'll be mostly fine if your instance isn't as big as Beehaw's.

[–] karce@beehaw.org 3 points 1 year ago

Awesome! Thanks for the response. I'd love a post sometime on hosting Lemmy. I'd find it very interesting and useful!

[–] nutomic@lemmy.ml 2 points 1 year ago (1 children)

Im also considering to setup nginx caching for lemmy.ml. Did you find a configuration which works?

[–] Helix@feddit.de 3 points 1 year ago (1 children)

Not yet. Session tracking in Lemmy is pretty hard to proxy, I'll have to dive into the code to figure out why.

[–] nutomic@lemmy.ml 3 points 1 year ago

Have a look at this: https://github.com/LemmyNet/lemmy-ansible/pull/75

Sending proper cache-control headers from Lemmy will require some big code changes though.

[–] Penguincoder@beehaw.org 3 points 1 year ago

Yes.

Deviation is pretty minimal, related to configuration and customization.

load more comments