this post was submitted on 01 Jul 2023
3840 points (97.3% liked)

Lemmy.World Announcements

29084 readers
220 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news ๐Ÿ˜

Outages ๐Ÿ”ฅ

https://status.lemmy.world/

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations ๐Ÿ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS
 

Looks like it works.

Edit still see some performance issues. Needs more troubleshooting

Update: Registrations re-opened We encountered a bug where people could not log in, see https://github.com/LemmyNet/lemmy/issues/3422#issuecomment-1616112264 . As a workaround we opened registrations.

Thanks

First of all, I would like to thank the Lemmy.world team and the 2 admins of other servers @stanford@discuss.as200950.com and @sunaurus@lemm.ee for their help! We did some thorough troubleshooting to get this working!

The upgrade

The upgrade itself isn't too hard. Create a backup, and then change the image names in the docker-compose.yml and restart.

But, like the first 2 tries, after a few minutes the site started getting slow until it stopped responding. Then the troubleshooting started.

The solutions

What I had noticed previously, is that the lemmy container could reach around 1500% CPU usage, above that the site got slow. Which is weird, because the server has 64 threads, so 6400% should be the max. So we tried what @sunaurus@lemm.ee had suggested before: we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.

Et voilร . That seems to work.

Also, as suggested by him, we start the lemmy containers with the scheduler disabled, and have 1 extra lemmy running with the scheduler enabled, unused for other stuff.

There will be room for improvement, and probably new bugs, but we're very happy lemmy.world is now at 0.18.1-rc. This fixes a lot of bugs.

(page 8) 50 comments
sorted by: hot top controversial new old

Does browsing with Incognito/Private mode opens up new bugs, or does the refreshing thing kept the same principle? I should be stayed as logged on, but for some reason - after this update - whenever I open a new private tab from the tab I'm logged on I am indicated as not logged in.

[โ€“] Sigmatics@lemmy.world 2 points 1 year ago

Still seeing a lot of 504s, and this is at a time where the site shouldn't be super busy

[โ€“] bappity@lemmy.world 2 points 1 year ago

thank you for upgrading! I much prefer the new URL structure of this version

[โ€“] hymenopterror@lemmy.world 2 points 1 year ago

YEAH BABEYYY

[โ€“] KD_14@lemmy.world 2 points 1 year ago

Are you hosting on AWS?

[โ€“] darkbaron202@lemmy.world 2 points 1 year ago

thank u for letting us know what happening behind the scene. Me myself is a sysadmin and really love to read story about scaling up servers and it actually works! Once again. Thank you.

[โ€“] Djangofett@lemmy.ca 2 points 1 year ago

Showing us the true value of horizontal instead of vertical scaling! Kubernetes when?

[โ€“] kratoz29@lemmy.world 2 points 1 year ago

Seems that sometimes the hot algorithm gets messed up :/

[โ€“] reblochon@lemmy.world 1 points 1 year ago

Thanks for your work!

[โ€“] Someguy89@lemmy.world 1 points 1 year ago

I'm a reddit migration and love what I saw so far! Keep it up!

[โ€“] evil_opossum@lemmy.world 1 points 1 year ago

Congratulations on getting it up and running!

[โ€“] Zoness@lemmy.world 1 points 1 year ago

Congrats on figuring it out! I'm just wading into docker in a professional capacity so I admit some of it feels like magic to my traditional developer brain but glad it worked out.

[โ€“] CataclysmZA@lemmy.world 1 points 1 year ago

Posting this from Jerboa. I am grateful for the hard work!

[โ€“] danielton@lemmy.world 1 points 1 year ago

Thank you for all your hard work running this instance and for keeping us posted!

[โ€“] devopspalmer@lemmy.world 1 points 1 year ago

Great work! ๐Ÿ‘ glad to be on the newer release with lemmy.world

[โ€“] evil_opossum@lemmy.world 1 points 1 year ago

Congratulations on getting it up and running!

[โ€“] Yokimari@lemmy.world 1 points 1 year ago

Seems to be working great so far.

load more comments
view more: โ€น prev next โ€บ