this post was submitted on 05 Jul 2023
3350 points (99.4% liked)

Lemmy.World Announcements

29079 readers
223 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world/

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS
 

Another day, another update.

More troubleshooting was done today. What did we do:

  • Yesterday evening @phiresky@phiresky@lemmy.world did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
  • @cetra3@lemmy.ml created a docker image containing 3PR's: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
  • We started using this image, and saw a big drop in CPU usage and disk load.
  • We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws.
  • We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
  • We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set ~~proxy_next_upstream timeout;~~ max_fails=5 in nginx.

Currently we're running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the ~~proxy_next_upstream timeout;~~ max_fails=5 workaround but for now it seems to hold with 1.

Thanks to @phiresky@lemmy.world , @cetra3@lemmy.ml , @stanford@discuss.as200950.com, @db0@lemmy.dbzer0.com , @jelloeater85@lemmy.world , @TragicNotCute@lemmy.world for their help!

And not to forget, thanks to @nutomic@lemmy.ml and @dessalines@lemmy.ml for their continuing hard work on Lemmy!

And thank you all for your patience, we'll keep working on it!

Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.

Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that's now started, and I noticed the proxy_next_upstream timeout setting didn't work (or I didn't set it properly) so I used max_fails=5 for each upstream, that does actually work.

(page 3) 50 comments
sorted by: hot top controversial new old
[–] slashzero@hakbox.social 10 points 1 year ago* (last edited 1 year ago)

As a Performance Engineer myself, these are the kind of performance improvements I like to see. Those graphs look wonderful. Nice job to all.

[–] bathorygod@lemmy.world 10 points 1 year ago

You guys did an awesome job.

[–] EvilCartyen@lemmy.world 10 points 1 year ago (1 children)

Things have been super smooth lately, thanks for all the work!

load more comments (1 replies)
[–] CriticalMiss@lemmy.world 9 points 1 year ago

I like that the post goes in detail and allows us tech nerds to get hard watching this stuff instead of the regular corpo jumbo change log that consists of:

  1. we uhh fixed some stuff so yeah good?
[–] ef9357@lemmy.one 9 points 1 year ago

Love the transparency. Thanks to the entire team!

[–] IanM32@lemmy.world 9 points 1 year ago

A lot of this stuff is pretty opaque to me, but I still read through it because I love how detailed they are in sharing what's going on under the hood and how it relates to problems users experience. Kudos to you guys!

[–] BURN@lemmy.world 9 points 1 year ago

I love reading these! Thanks for all the work

I hope to start on some small contributions sometime next week. Stability has been noticeably better the last few days and I imagine it’s only going to get better.

thankyouthankyouthankyouthankyouthankyou it's working very well today...

[–] anal_enjoyer@reddthat.com 9 points 1 year ago

Love these updates! Love the transparency!

[–] KonQuesting@lemmy.sdf.org 8 points 1 year ago

Thanks for the updates! Seeing the details of how you work through these early issues is valuable to those of us thinking of starting an instance.

[–] Olap@lemmy.world 8 points 1 year ago

Proving that orgs do need DBAs!

[–] DharkStare@lemmy.world 8 points 1 year ago

Great job. Everything seems to be working smoothly for me now. The past several days have been a bit rough but now it's all working.

[–] Hynek29@lemmy.world 8 points 1 year ago

Great job guys! It really feels more responsive today

[–] lichkain@lemmy.world 7 points 1 year ago* (last edited 1 year ago) (1 children)

Would HAProxy work better as a load balancer? For work we switched due to some issues with NGINX; so far, the service has been much more consistent with pretty much no downtime, even when restarting server hosts.

load more comments (1 replies)
[–] Boiglenoight@lemmy.world 7 points 1 year ago

Compared to days prior, things are running much better today. Page load speed, reliable post/reply/upvoting. I dunno if it's just happenstance but whatever knobs, levers and keystrokes you're manipulating, keep going the things! Thanks so much for a home absent of corporate BS.

Definitely noticeable!

Also, the amount of communication here is AWESOME! Thank you for keeping us updated.

[–] atretador@lemmy.world 7 points 1 year ago

Ty for the hard work <3

[–] Ipodjockey@lemmy.world 7 points 1 year ago

Sweet. Yeah it is definitely snappier today. Feels great.

[–] legroom@lemmy.world 7 points 1 year ago

Great work !!

[–] Techie@lemmy.world 7 points 1 year ago

Awesome, thanks!

[–] NatoBoram@lemmy.world 7 points 1 year ago

It works so well, that's very refreshing

[–] throwsbooks@lemmy.world 6 points 1 year ago

Now that's a satisfying couple of charts!

[–] Drunemeton@lemmy.world 6 points 1 year ago

Thanks for the update and thanks for the amazing collaboration you’re fostering!

I feel like Lemmy.World is leading the way for other instances to follow, and that’s a good thing to do. Even if it’s painful for everyone involved.

[–] oryx@lemmy.world 6 points 1 year ago

Things have been very noticably improving every day since the big update! I've had very, very few problems today browsing on Jerboa.

Amazing work, thank you so much!

load more comments
view more: β€Ή prev next β€Ί