Yesterday evening @phiresky@phiresky@lemmy.world did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
@cetra3@lemmy.ml created a docker image containing 3PR's: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
We started using this image, and saw a big drop in CPU usage and disk load.
We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws.
We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set ~~proxy_next_upstream timeout;~~ max_fails=5 in nginx.

Currently we're running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the ~~proxy_next_upstream timeout;~~ max_fails=5 workaround but for now it seems to hold with 1.

Thanks to @phiresky@lemmy.world , @cetra3@lemmy.ml , @stanford@discuss.as200950.com, @db0@lemmy.dbzer0.com , @jelloeater85@lemmy.world , @TragicNotCute@lemmy.world for their help!

And not to forget, thanks to @nutomic@lemmy.ml and @dessalines@lemmy.ml for their continuing hard work on Lemmy!

And thank you all for your patience, we'll keep working on it!

Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.

Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that's now started, and I noticed the proxy_next_upstream timeout setting didn't work (or I didn't set it properly) so I used max_fails=5 for each upstream, that does actually work.

top 50 comments

sorted by: hot top controversial new old

[–] phiresky@lemmy.world 890 points 1 year ago* (last edited 1 year ago) (9 children)

server load is too low, everyone upvote more stuff so i can optimize more

edit: guess there is some more work to be done 😁

[–] woelkchen@lemmy.world 144 points 1 year ago (2 children)

Upvote causes an endless spinner on Liftoff. 😁

[–] TurnItOff_OnAgain@lemmy.world 44 points 1 year ago (2 children)

I'm getting 504 gateway time outs when I try to upvote

load more comments (2 replies)

load more comments (1 replies)

[–] marsokod@lemmy.world 117 points 1 year ago (8 children)

I don't understand your graph. It says you are measuring gigabit/sec but shouldn't the true performance rating be gigabeans/sec for a Lemmy instance?

load more comments (7 replies)

[–] Snipe_AT@lemmy.atay.dev 54 points 1 year ago

aye aye sir, to the upvote machine!

[–] PatFussy@lemm.ee 27 points 1 year ago (1 children)

Double the image upload size and you will see more shitposts

load more comments (1 replies)

load more comments (5 replies)

[–] Rootiest@lemmy.world 436 points 1 year ago (10 children)

Test:

Upvote if you can see this comment. 👍

load more comments (10 replies)

[–] isaachernandez@lemmy.world 253 points 1 year ago (2 children)

The change is noticeable. Good job guys.

Thanks for the updates.

[–] Carnelian@lemmy.world 94 points 1 year ago (1 children)

I agree. Felt it immediately when I started browsing. Everything is faster and more responsive, on top of the error messages disappearing

load more comments (1 replies)

[–] mintiefresh@lemmy.world 124 points 1 year ago (2 children)

Wow. So much smoother today.

Great work.

You dropped this 👑

load more comments (2 replies)

[–] GnothiSeauton@lemmy.world 124 points 1 year ago (4 children)

This is why having a big popular instance isn't all bad. It helps detect and fix the scaling problems and inefficiencies for all the other 1000s of instances out there!

[–] AlmightySnoo@lemmy.world 59 points 1 year ago (1 children)

This, if everyone kept just spreading out to smaller instances as suggested in the beginning, while still a sensible thing to do, no one would have noticed these performance issues. We need to think a few years out, assuming Lemmy succeeds and Reddit dies, and expect that "small instance" will mean 50k users.

load more comments (1 replies)

load more comments (3 replies)

[–] kionay@lemmy.world 115 points 1 year ago

You guys had better quit it with all this amazing transparency or it's going to completely ruin every other service for me. Seriously though amazing work and amazing communication.

[–] dreadedsemi@lemmy.world 99 points 1 year ago

My upvote can go through fast now

Good work

[–] wilberfan@lemm.ee 97 points 1 year ago

I love the smell of updates in the morning.

[–] sv1sjp@lemmy.world 91 points 1 year ago (6 children)

Thank you guys for your awesome work!

Also to other people: DONATE TO FOSS PROJECTS. If 50.000 people donate only 0.5€, we have 25.000€ for funding the servers, coding, motivating/ people etc. Just don't take a cup of coffee for 1 day. We are already 2 millions in Lemmy instances. We can build a decentralized world together!!

[–] wmrch@lemmy.world 24 points 1 year ago

You can pry my cup of coffee from my my cold, dead hands.

Will donate anyway, I really want this project to keep going.

load more comments (5 replies)

[–] Marxine@lemmy.world 80 points 1 year ago

Boy does it feel good to have those reports and understand the work you guys do. It's really inspiring! Thanks for your hard work, everything has been silk smooth! This instance is really great, Lemmy and its devs are really amazing and I feel at home in a nice, cozy community.

[–] Spectator@lemmy.world 80 points 1 year ago

I'm not sure wtf you just said, but lemmy.world feels very smooth today, so thank you for your continued hard work!

[–] sirnak@lemmy.world 78 points 1 year ago (4 children)

Am I getting this correct: the whole lemmy.world instance run in one single container on one single host?

[–] cley_faye@lemmy.world 46 points 1 year ago (1 children)

You'd be surprised at how much performance this kind of setup can squeeze off. Often the limitation is more on the DB/storage than network handling and processing power.

[–] eek2121@lemmy.world 24 points 1 year ago

This. Most of the time, the bottleneck will be the database backend.

Curious if lemmy.world uses separate reader/writer instances.

load more comments (3 replies)

[–] Timou@lemmy.world 67 points 1 year ago (1 children)

So that's why it was so smooth today... Great work!

load more comments (1 replies)

[–] ericjmorey@lemmy.world 65 points 1 year ago* (last edited 1 year ago)

Submitting PRs is literally the most effective response that helps everyone who uses Lemmy. Thanks to you all.

[–] MR_GABARISE@lemmy.world 63 points 1 year ago (3 children)

This is better optimization than most enterprise devs will see in their lifetimes.

[–] clutchmatic@lemmy.world 24 points 1 year ago

Some managers of the devs are not that interested in significant optimizations... Depends on what incentives and company culture drives them

load more comments (2 replies)

[–] xandertron@lemmy.world 61 points 1 year ago

upvoting posts is so much more stable now, we might actually see more bean posts as a result

[–] MindfuckRocketship@lemmy.world 58 points 1 year ago* (last edited 1 year ago) (4 children)

Upvotes are still getting rejected. Replies hang so I cancel out and it turns out they did post.

That said, browsing is pretty snappy and smooth. I know the kinks will get worked out eventually. Thanks for the update.

Edit: This now appears resolved minutes later. All smooth on my end.

[–] ruud@lemmy.world 94 points 1 year ago (3 children)

Hmm. Seems to work for me.. (Yes this is a test reply)

[–] G_Wash1776@lemmy.world 31 points 1 year ago (2 children)

Everyone it’s a test reply, deploy the upvotes

load more comments (2 replies)

load more comments (3 replies)

[–] Molecular0079@lemmy.world 53 points 1 year ago

This is why I love open source. The fact that a community can directly debug the code that's it's being hosted on and directly contribute the improvements back is just wild. Thanks for all the hard work @ruud@lemmy.world and the rest of the lemmy.world team! The site already feels much more responsive.

[–] teraflopsweat@lemmy.world 52 points 1 year ago

Literally a night and day difference in performance and stability! Thank you all for the hard work. To other users like me, consider reducing or replacing one of your lesser used subscriptions and directing that money to Lemmy. It’s much better served here if you ask me.

[–] MrPoopyButthole@lemmy.world 49 points 1 year ago

The server is absofuckinglutely flying today! It feels smooth and bug free!!! You guys are legends.

[–] _Rho_@lemmy.world 45 points 1 year ago* (last edited 1 year ago) (5 children)

As a data engineer, I'd be interested in hearing more about the SQL troubleshooting.

EDIT: It looks like !lemmyperformance@lemmy.ml is a good place to subscribe to for more technical info on some of these performance improvements.

Also the Lemmy GitHub of course contains more information on bugs/enhancements/etc.

load more comments (5 replies)

[–] BitOneZero@lemmy.world 40 points 1 year ago* (last edited 1 year ago)

Good to see a heavy production server taking on the scaling issues. Thank you! To discuss Lemmy performance issues, there is a dedicated community: !lemmyperformance@lemmy.ml

[–] 0235@lemmy.world 38 points 1 year ago (4 children)

Appreciate that these updates use the yyyy-mm-dd format :D

load more comments (4 replies)

[–] asamson23@lemmy.world 36 points 1 year ago (1 children)

It now feels pretty good to browse and it now makes the experience of using Lemmy much more enjoyable. Having to spam the vote buttons was really annoying.

load more comments (1 replies)

[–] nostalgicgamerz@lemmy.world 34 points 1 year ago* (last edited 1 year ago) (7 children)

Can we have an update on the status of Lemmy.world and how close ties we are going to have with Meta's threads? Threads is going to support ActivityPub, but time has shown that this is an attempt to try to kill this open platform and eventually replace it with theirs once they get everyone in their ecosystem. (Embrace, Extend...extinguish) Mastodon has said today that they don't mind sleeping with vipers when their demise / dissolution is in Meta's best interest.

Please tell me we are defederating from Meta....or let us know what to expect

EDIT: I originally stated that Mastodon told them to fuck off, but I got confused with Fosstodon (who did that). Mastodon doesn't mind being in bed with Meta

load more comments (7 replies)

[–] xavier666@lemm.ee 34 points 1 year ago (3 children)

Even though i'm not from this instance, this is such a nice way of keeping the users posted about changes. I wish more companies (I know this is not a company) went straight to the point, instead of using vague terms like "improved stability, fixed few issues with an update" when things are changed. I hope all instance owners follow this trend.

load more comments (3 replies)

[–] radfordhound@programming.dev 34 points 1 year ago

It's so smooth now; the speed difference is insane! You all are doing excellent work!

[–] JoeKrogan@lemmy.world 32 points 1 year ago

Thanks to all involved across the board. Great work all around 👏👏

[–] simple@lemm.ee 30 points 1 year ago

Huge props to everyone working on the project. It's awesome seeing everyone work together and resolving issues so quickly!

[–] helios@social.ggbox.fr 30 points 1 year ago

Thanks for this very nice report.

[–] ekZepp@lemmy.world 28 points 1 year ago

[–] solidgrue@lemmy.world 28 points 1 year ago

Gadzooks! These are huge fixes. Compliments to the team, you guys pulled off a small miracle today.

[–] yonerboner@lemmy.world 28 points 1 year ago

Thanks for the detailed update and all the hard work you guys are doing!

[–] Marxine@lemmy.world 25 points 1 year ago (2 children)

Lemmy's devs and the .world admins have done in a month what Reddit hasn't done in it's whole existence: having a smooth and almost bug-free experience.

Jerboa feels so damn FRESH to use now!

load more comments (2 replies)

[–] Oxff@lemmy.world 25 points 1 year ago

The site is running so much better now, thanks to all. BTW: Love these updates!

[–] MetricExpansion@lemmy.world 24 points 1 year ago (6 children)

I'm very curious: does single Lemmy instance have the ability to horizontally scale to multiple machines? You can only get so big of a machine. You did mention a second container, so that would suggest that the Lemmy software is able to do so, but I'm curious if I'm reading that right.

load more comments (6 replies)

load more comments