this post was submitted on 13 Jun 2023
666 points (100.0% liked)

Lemmy.World Announcements

29079 readers
246 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages 🔥

https://status.lemmy.world/

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations 💗

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS
ruud@lemmy.world
lwadmin@lemmy.world
lwCET@lemmy.world
jelloeater85@lemmy.world
Serinus@lemmy.world
lw_mod_notification@lemmy.world
MrKaplan@lemmy.world
666
Addressing the Exponential Growth of Communities (lemmynsfw.com)
submitted 1 year ago* (last edited 1 year ago) by OptimusPrime@lemmynsfw.com to c/lemmyworld@lemmy.world
218 comments fedilink hide all child comments
 

Over the past few days, I've witnessed a remarkable surge in the number of communities on browse.feddit.de. What started with 2k communities quickly grew to 4k, and now it has reached an astonishing 8k. While this exponential growth signifies a thriving platform, it also brings forth challenges such as increased fragmentation and the emergence of echo chambers. To tackle these issues, I propose the implementation of a Cross-Instance Automatic Multireddit feature within Lemmy. This feature aims to consolidate posts from communities with similar topics across all federated instances into a centralized location. By doing so, we can mitigate community fragmentation, counter the formation of echo chambers, and ultimately foster stronger community engagement. I welcome any insights or recommendations regarding the optimal implementation of this feature to ensure its effectiveness and success.

you are viewing a single comment's thread
view the rest of the comments
[–] PriorProject@lemmy.world 1 points 1 year ago (1 children)

There’s an advantage to splitting the load amongst multiple instances.

I don't have links handy, but the devs have said multiple times that federation replication load is not a limiting factor. It's the browsing load that matters, which is already spread across the instances that subscribers hail from. There isn't a performance issue here to fix, at least around the current network size.

If community@host1.com has issues, the subscribers to community can still access community@host2.com.

Again, user-account federation already provides this. If the host where the community resides is down, I can still read existing posts, and post and comment with users on my local instance. I don't see significant further benefits in splitting the community hosting.

Voluntary syncing of communities seems to me a natural extension of the federated philosophy.

I don't disagree with this, the challenge is that the federated philosphy makes everything it touches very complicated. A sprinkle of federation in the core of an ecosystem brings enormous resilience against the ecosystem getting coopted against the will of its users. A heaping spoonful of federation makes it an unusable mess. The complexity of federated design needs to bring very big benefits to "pull its weight" in complexity. The tradeoff looks positive for me for federated replication even though the cost in complexity is large, but I don't for meta-communities. They seem like cosmetic improvements over the existing capabilities.

But fair enough, I see where you're coming from. It's not madness, on-balance it's just not something I see providing value in proportion to its conceptual complexity.

[–] MentallyExhausted@reddthat.com 2 points 1 year ago (1 children)

If the host where the community resides is down, I can still read existing posts, and post and comment with users on my local instance.

Can you help me understand how that all works? I ran into this today where an instance wouldn’t let me submit due to performance issues, and I jumped instances to be able to provide some (potentially) important breaking news to “community.” If they were fully in-sync and one instance didn’t “own” the community, when things stabilized, my content would have replicated instead of being stranded on a lonely instance. As it stands, that content will never be part of the “real” community.

But fair enough, I see where you’re coming from. It’s not madness, on-balance it’s just not something I see providing value in proportion to its conceptual complexity.

Right on. I’m still new to the fediverse and am probably missing some foundational concepts still. Love where it’s headed though.

[–] PriorProject@lemmy.world 2 points 1 year ago (1 children)

If the host where the community resides is down, I can still read existing posts, and post and comment with users on my local instance.

Can you help me understand how that all works?

I'll preface this by saying I haven't read the code and it's certainly possible there are errors in my mental model, but my belief is:

  1. When the community-instance is up and it gets a post from a user local to the community... It write that data to its db and asynchronously sends federated messages to instances with users that sub the community. Those instances write it to their local DB.
  2. When the community-instance goes down and someone on a user-instence tries to read the post, it gets read from the local-db cached copy.
  3. While the community-instance is still down and user-a on a user-instance posts or comments, again it goes into the local DB and async sends a federation message to the community-instance... the message doesn't immediately get through though.
  4. While the community-instance is down, if user-b on the same user-instance browses the community, they see user-a's content from step 3 in the user-instances-db. If user-c on a different user-instance checks, they won't see user-a's content until the community-instance comes back up to process the federation messages from step 3.
  5. When the community-instance comes back up, it accepts the federation messages from step 3 and updates all subscribed instances so user-a's content is viewable everywhere.

It's possible I'm mistaken about 3-5, but if you had trouble posting... my guess would be that your own user-instance was struggling and slow. But if you've rules that out maybe the update behavior when the community-instance is down works differently than I think it does. The reading in step 2 I'm quite certain about though.

[–] MentallyExhausted@reddthat.com 3 points 1 year ago (1 children)

I'm pretty sure you can't post to community-instance if they're down/having issues to the local DB. The instance I'm on is very stable and every time I think it may be an issue, it turns out to be the other instance.

It sounds like a lot of the "sync" behavior I was thinking of is already built-in if we can just expand on it a bit.

[–] PriorProject@lemmy.world 2 points 1 year ago (1 children)

I'm pretty sure you can't post to community-instance if they're down/having issues to the local DB. The instance I'm on is very stable and every time I think it may be an issue, it turns out to be the other instance.

I can't disprove this. My experience when lemmy.ml was struggling was that I could comment-reply and vote consistently even though many of the communities I'm active in are on that struggling server. But maybe I was just lucky. I can't explain the behavior you observed, and it may be that certain operations are more directly passed through to the community-hosting instance than I previously believed.

But irrespective of the details, plain federation does provide some significant resilience to transient instance failure, and there's probably room to tune it further in at least some ways.

Interesting discussion, though.

[–] MentallyExhausted@reddthat.com 1 points 1 year ago (1 children)

Ah, yeah I should clarify that I was able to comment on existing posts, but could not submit a new post. Fun discussion, agreed!

[–] PriorProject@lemmy.world 2 points 1 year ago

That's the difference then, I do A LOT more commenting than posting. Perhaps there's some essential reason they're handled differently or hopefully an accidental one and posts could be made to behave more like comments someday.