lodion

joined 1 year ago
MODERATOR OF
[–] lodion@aussie.zone 1 points 8 hours ago

Yeah I like it too... but it was generating an abnormal amount of traffic. I'll see if I can figure out what is going on... but mlmym is pretty basic, doesn't seem to have much in the way of configuration options.

[–] lodion@aussie.zone 3 points 3 days ago

I especially like the look on this guy's face...

[–] lodion@aussie.zone 3 points 3 days ago (1 children)

Was told there would be puppies. A++ would view again.

[–] lodion@aussie.zone 1 points 4 days ago

Yeah... I understood outgoing federation of activities was linear FIFO. The new feature in 0.19.6 allows N instances of linear FIFO, with related activities federated in sequence.

Our outgoing activities go out very quickly, aussie.zone has a very low activity rate so remote instances don't tend to get behind at all.

How a LW user replied to a current AZ post and had it show up here in a decent timeframe is a mystery to me, seeing as they're still running their custom version of 0.19.3

[–] lodion@aussie.zone 1 points 4 days ago

I can only guess the order activities are federated in is random? Which doesn't make much sense.

[–] lodion@aussie.zone 3 points 6 days ago (2 children)

It's configurable... defaults to 1 thread. Given we're slowly catching up with LW during low activity parts of the day, I think 2 should be enough.

[–] lodion@aussie.zone 3 points 6 days ago (1 children)

Not sure about the whole picture, but credit to Keating for bringing in compulsory superannuation.

[–] lodion@aussie.zone 14 points 6 days ago* (last edited 6 days ago) (9 children)

0.19.6 includes many fixes and new features... the biggest for me is "Parallel federation sending". This allows instance admins to send multiple federated activities in parrallel, rather than singly in series.

Not that its a real issue for other admins with activities from AZ, but I have set this to 2 for our outgoing activities.

Hopefully the lemmy.world admins will upgrade to 0.19.6 soon and do the same, this should allow AZ to catch up with activities from LW, and stay up to date. Over time as activities from LW have decreased we've "caught up" to within ~5 days of live. Doubling the rate that LW sends activities to AZ will help massively.

[–] lodion@aussie.zone 6 points 6 days ago (1 children)

Ok... DB migration was an offline my migration, my bad. Took longer I expected... but we're back in action 🤓

33
submitted 6 days ago* (last edited 6 days ago) by lodion@aussie.zone to c/meta@aussie.zone
 

I'll be working on upgrading aussie.zone to lemmy 0.19.6 today. All going well disruption will be brief, but there may be some performance issues related to back end DB changes required as part of the upgrade.

I'll unpin this once complete.

[–] lodion@aussie.zone 3 points 1 week ago

Maybe AskAnAussie? Or something like that. Want to keep the communities here Australia focused.

Federation issues with LW have improved, and I expect to be eliminated with a future version of the lemmy server software.

[–] lodion@aussie.zone 8 points 1 week ago (1 children)

Good, fwiw you're banned here too now.

11
submitted 2 weeks ago* (last edited 3 days ago) by lodion@aussie.zone to c/meta@aussie.zone
 

I've spun this up for fun, to see how it compares to the base lemmy UI. Give it a whirl, and post any feedback in this thread. Enjoy!

It could go down at any time, as it looks as though the dev is no longer maintining it...

edit: using this https://github.com/rystaf/mlmym

UPDATE Tuesday 12/11: I've killed this off for now. Unclear of why, but was seeing a huge number of requests from this frontend to the lemmy server back end. Today it alone sent ~40% more requests than all clients and federation messages combined.

 

Its been 6 months or so... figure its time for another of these. Keep in mind there have been some major config changes in the last week, which has resulted in the oddities below.

Graphs below cover 2 months, except Cloudflare which only goes to 30 days on free accounts.

CPU:

Memory:

Network:

Storage:

Cloudflare caching:

Comments: The server is still happily chugging along. Looking even happier now that I've properly migrated pict-rs to its integrated object storage config, rather than the bodged up setup.

RAM/CPU are all fine. Storage use is growing slowly as various databases grow. Still a long way from needing to purge old posts, if ever.

Cloudflare is saving less traffic these days, since Lemmy added support to proxy all images. Not a concern, well under the bandwidth cap for the server.

As usual feel free to ask any questions.

37
Pictures are broken (aussie.zone)
submitted 3 weeks ago* (last edited 3 weeks ago) by lodion@aussie.zone to c/meta@aussie.zone
 

I'm in the process of migrating images to a properly configured object storage setup. This involves an offline migration of files. Once complete, I'll start up pict-rs again. Until then, most images will be broken.

All going well this will finish by morning Perth time, and once up and running again may help with the ongoing issues we've had with images.

 

After some users have had issues recently, I've finally gotten around to putting in place a better solution for outbound email from this instance. It now sends out via Amazon SES, rather than directly from our OVH VPS.

The result is emails should actually get to more people now, rather than being blocked by over-enthusiastic spam filters... looking at you Outlook and Gmail.

11
REBOOTING (aussie.zone)
 

About to reboot the server, hold onto your hats.

 

Hey all, following the work over the weekend we're now running Lemmy 0.19.4. please post any comments, questions, feedback or issues in this thread.

One of the major features added has been the ability to proxy third party images, which I've enabled. I'll be keeping a closer eye on our server utilisation to see how this goes...

71
Maintenance (aussie.zone)
submitted 5 months ago* (last edited 5 months ago) by lodion@aussie.zone to c/meta@aussie.zone
 

This weekend I'll be working to upgrade AZ to lemmy 0.19.4, which requires changes to some other back end supporting systems.

Expect occasional errors/slowdowns, broken images etc.

Once complete, I'll be making further changes to enable/tweak some of the new features.

UPDATE: one of the back end component upgrades requires dumping and reimporting the entire lemmy database. This will require ~1 hour of total downtime for the site. I expect this to kick off tonight ~9pm Perth time.

UPDATE2: DB dump/re-import going to happen ~6pm Perth time, ie about 10 minutes from this edit.

UPDATGE3: we're back after the postgres upgrade. Next will be a brief outage for the lemmy upgrade itself... after I've had dinner 🙂

UPDATE34: We're on lemmy 0.19.4 now. I'll be looking at new features/settings and playing around with them.

 

Its been a little while since I posted stuff :)

CPU:

Memory:

Network:

Storage:

Cloudflare caching:

Comments:
Not much has changed in quite a while. I still have a cron-job running to restart Lemmy every day due to memory leaks, hopefully this improves with future updates. Outside of that, CPU, memory and network usage are fine.
Object storage usage is growing steadily, but we're a long way from paying more than the monthly minimum Wasabi fee.

51
submitted 10 months ago* (last edited 10 months ago) by lodion@aussie.zone to c/meta@aussie.zone
 

The upgrade to lemmy 0.19 has introduced some issues that are being investigated.. but we currently have no fixes for:

  • thumbnails break after a time. Caused by memory exhaustion killing object storage processes.
  • messages to/from lemmy instance not yet running 0.19 are not federating. I believe it requires bugfixes by the devs.

~~I've re-enabled 2 hourly lemmy restarts. Hopefully this will help with both issues, though it will result in a disruption to the site around every couple of hours.

When the hourly restarts are disabled I'll unpin this post. As any other issues are identified I'll post them here too. ~~

Update: I've disabled the 2 hourly restart after upgrading to 0.19.2... lets see how this goes...

Update2: no issues seen since the upgrade, looks to have resolved both the memory leak and the federation issues. Hooray :)

60
Lemmy 0.19 Upgrade (aussie.zone)
submitted 11 months ago* (last edited 11 months ago) by lodion@aussie.zone to c/meta@aussie.zone
 

~~I'll be restarting AZ today for an update to lemmy 0.19.~~

Upgrade complete.

This is a major upgrade, so I expect there to be some issues. Strap in, enjoy the ride.

Expect:

  • further restarts
  • bugs
  • slowdowns
  • logouts
  • 2FA being disabled
  • possibly issues with images, upgrading pictrs to 0.4 at the same time
 

Hi all,

Looks like I'll be visiting Melbourne in winter 2024 for a wedding with my family. Any must-see tourist attractions you can suggest?
We'll definitely be hitting up Melbourne zoo, and maybe Werribee.

Thinking I'll have to hire a car.. though I'd prefer to stay central and use public transport and Uber around when required.

view more: next ›