this post was submitted on 23 May 2023

28 points (100.0% liked)

Beehaw Support

2796 readers

1 users here now

Support and meta community for Beehaw. Ask your questions about the community, technical issues, and other such things here.

A brief FAQ for lurkers and new users can be found here.

Our September 2024 financial update is here.

For a refresher on our philosophy, see also What is Beehaw?, The spirit of the rules, and Beehaw is a Community

This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

if you can see this, it's up

founded 2 years ago

MODERATORS

Penguincoder@beehaw.org

remington@beehaw.org

why 7 days of new posts are gone (beehaw.org)

submitted 1 year ago by alyaza@beehaw.org to c/support@beehaw.org

21 comments fedilink hide all child comments

in short: our server crashed the other day and we're not really sure why, so we had to restore a backup

it went down at about 1:40 pm yesterday and we gave it until early this morning to diagnose what the problem was ourselves and get feedback from our server host (Digital Ocean) on what the problem might be. unfortunately we couldn't do that and none of their suggestions helped, so this morning we pulled the trigger on restoring the backup. as a consequence a few days of posts and registrations are now lost.

so: if you registered some time in the past 7 days, you will need to re-register.

(as a registration aside, we approve registrations very quickly so if you have registered your account should be approved within 24 hours of joining the site)

it's unfortunate to lose all the posts and registrations of a week but we don't pretend to have 100% uptime (nor do we really have the funding to guarantee that, lol) so it is what it is. thanks for using the site, folks, hopefully this won't happen again anytime soon

top 18 comments

sorted by: hot top controversial new old

[–] admin@beehaw.org 15 points 1 year ago (3 children)

I can add what the Digital Ocean support team told me. Namely, the server suffered from some type of "kernel panic" that is very rare. The Digital Ocean support team do not have access to our back-end user interface nor can they access our server otherwise.

They guided me, step by step, through several attempts to restore the server from this "kernel panic". We made eight different attempts which all failed and we don't know what caused this issue in the first place.

Luckily, we have server backups enabled and this was the last resort.

[–] Hirom@beehaw.org 12 points 1 year ago* (last edited 1 year ago) (1 children)

A kernel panic may prevent a VM from booting but shouldn't prevent accessing the disk image for recovery purpose.

Maybe the kernel panic corrupted the disk image, or maybe Digital Ocean doesn't have a recovery mechanism that work when the VM doesn't boot.

[–] admin@beehaw.org 3 points 1 year ago

Maybe the kernel panic corrupted the disk image...

More than likely, this is what happened.

[–] possiblylinux127@beehaw.org 3 points 1 year ago

Was the filesystem corrupted?

[–] admin@beehaw.org 3 points 1 year ago (1 children)

More than likely.

load more comments (1 replies)

load more comments (-1 replies)

[–] anji@lemmy.anji.nl 13 points 1 year ago

Curiously, because I access Beehaw through my own instance, I didn't notice. Those posts and replies are still there in my database. Makes me wish ActivityPub applications were truly distributed/decentralized and it could recover from this.

Thanks for running Beehaw btw.

[–] jherazob@beehaw.org 11 points 1 year ago

As a sysadmin i feel your pain

[–] gloating_swann@beehaw.org 11 points 1 year ago

Very glad to have you back.

[–] dax@beehaw.org 10 points 1 year ago (1 children)

https://media.tenor.com/bVUh2pUkmZMAAAAC/futurama-were-back-baby.gif

[–] admin@beehaw.org 13 points 1 year ago

[–] tardigrada@beehaw.org 9 points 1 year ago (1 children)

Does it make sense to repost?

[–] alyaza@beehaw.org 13 points 1 year ago

if there are articles that got eaten by the past 7 days that you think are worth reposting yeah, i'd repost them

[–] Lost_Wanderer@beehaw.org 9 points 1 year ago (1 children)

Thanks for the updates!

[–] Troy@beehaw.org 7 points 1 year ago

... yeah, and keep up the good work 😅

[–] nachtigall@feddit.de 9 points 1 year ago (1 children)

Damn that’s weird. feddit.de recently had a crash too which resulted in the loss of a few posts. Do we already know the reason behind the crash @wintermute@feddit.de ? (Sorry I can’t check it on Matrix. It’s annoying that Element supports only one account and I hate other clients too much, and I am too lazy to login to the web client :D)

[–] wintermute@feddit.de 10 points 1 year ago* (last edited 1 year ago)

The pictrs container filled up the disk with temp files.

The issue was closed, but the problem still persists.

[–] Fortyseven@beehaw.org 8 points 1 year ago

Oof, that's rough. Welcome back.

load more comments