I get the impression that we're headed for the same issues that pop up when we put all our eggs in one basket with Reddit/FB/whatever. People flock to the largest instance, and someday that instance could go down due to cost or the host losing interest.

I'm wondering whether it would be technically achievable to have servers/instances and federation where the communities are essentially mirrored or have broadly distributed existence - maybe even with user storage a la torrents.

If there's a large blargh@lemmy.here community and a small blargh@lemmy.there community, all of the discussion, images, contributions to lemmy.here die if the server goes down for good. Yes, the users can relocate to lemmmy.there - even under the same community name - but it's not the same as having full continuity of a completely mirrored community.

I realize this concept has technical hurdles and would involve a reimagining of how the fediverse works, but I worry we're just setting up for another blowup at some TBD date when individual sysadmins decide they've had enough. If it's not truly distributed and just functions as a series of interconnected fiefdoms, communities and their information won't survive outages, deaths, and power struggles.

you are viewing a single comment's thread
view the rest of the comments

[–] mo_ztt@lemmy.world 28 points 1 year ago (1 children)

Dude I am psyched to hear you say this, I am literally spending time today working on trying to get the next little update working for exactly this. The destination of "it can actually be used as a shared backing store for a federated community" is a little bit in the ambitious far future, but some of the earlier destinations can still be pretty useful as far as making the whole thing work better, I think.

Link to a little project community

LMK what you think / if you want to try setting up a proxy instance once it reaches that point or contributing in other ways.

[–] rexxit@lemmy.world 6 points 1 year ago (1 children)

I just typed out a whole reply and it got eaten after I clicked reply - no surprise there are still going to be bugs in this implementation, nevermind getting more complex with it.

I'm not technical enough to make anything happen if it involves code, but I'm way psyched you're working on something along these lines. The way I see the problem there are some distinct issues

sharing the load of hosting (i.e. users are also hosts, as with torrents)
fault-tolerance / outage-tolerance
community/data immortality (i.e. discrete communities that are somehow not reliant on any instance)

[–] mo_ztt@lemmy.world 7 points 1 year ago

Yah absolutely. A lot of the problems in #1 and #2 are already solved, with technologies like Bittorrent as you mentioned, although doing the work to make it work isn't trivial. #3 can potentially be complex. So at present I'm imagining that someone will need to "pin" on some machine under their control all the data that operates a given community, in order for it to be guaranteed that all that data is available. Someone has to guarantee the storage in order for it to be guaranteed that the appropriate data won't be deleted, and I think it's better to just be explicit with the requirement and let people decide how to configure their software. Then the issue is just that the technology works well enough that that data can be found quickly if it's needed, and cached on other servers well enough that you can run that "pin" on your own home internet or on like a $20/mo Digital Ocean account, as long as you have the disk space and a machine that can be available almost all the time.

That sounds simple, but there are weird little things to worry about -- e.g. what if a stored-in-the-shared-store community is abandoned, and part of it passes out of the shared memory of the system, but not all of it? At what point can someone else create a new community to replace it? What happens to little pieces of shared content that still do exist within the system when that happens? What if at some point the old community reappears? I feel like that can be dealt with reasonably well, but it's clear that there are questions like that that are uncharted territory which always has some surprises.