I'm going to set up a general purpose instance tomorrow with the intention of handling a relatively large number of users. The main problem is choosing a domain!
Lemmy
Everything about Lemmy; bugs, gripes, praises, and advocacy.
For discussion about the lemmy.ml instance, go to !meta@lemmy.ml.
Naming things is one of the two most difficult issues in IT, alongside cache validation and off-by-one errors.
I was also contemplating setting up a new instance for this. I have 100s of gigs of unused ram, CPUs on idle and a 10gbit connection looking for something to do. The only issue I couldn't figure out was the name. I own itjust.works was thinking of something clever subdomain to use with it. I'm glad I'm not the only one with this issue
Sadly, I feel like the Fediverse, based on ActivityPub, was fundamentally designed wrong for scaling potential. I do like Fedi and I like ActivityPub, but I think instances should not have to be responsible for all of this:
- Owning user accounts
- Exclusively host communities
- Serving local and remote users webpages and media
- Never going down, as this results in users and content becoming unavailable
Because servers "own" the user accounts and communities it's not trivial for users to switch to a different instance, and as instances scale their costs go up slightly exponentially.
I wish the Fediverse from the beginning was a truly distributed content replication platform, usenet-style or Matrix-style, and every instance would add additional capacity to the network instead of hosting specific communities or users.
I guess it's a bit too late for a redesign now... Perhaps decentralized identifiers will take us there in some form in the future.
Not sure why you reference Matrix, which has even worse scaling issues as it indeed tries to replicate nearly the entire network on every server.
The Fediverse is really just working how the general web does, just with some standardized API for websites to interact. It's not perfect, but it works and has proven to be relatively scalable.
It sounds a bit like you had a bit too much of the Bluesky cool-aid, which indeed replicates nearly all of the mistakes of Matrix and makes it impossible to scale via small community owned servers instead of big company owned data-centers (which might be by design?).
Well yeah, point taken that replicating everything everywhere and forever might be impossible. But I do believe at a minimum my identity should be portable and accessing Fedi (ie. in microblogging: posting and viewing a feed of the latest posts of my follows) should be decoupled from which instance I pick to access the Fediverse.
I don't particularly like how owners of instances which grew are now essentially locked in to having to spend 100s or 1000s of dollars a month keeping their now expensive instances running and providing service. This is a bad place to be for a platform ran by volunteers. Letting instance owners scale their service down as well as up would be ideal. But this requires at least decentralized identity, and at best some form of content hosting redundancy...
It's easy to say the current architecture of Fedi works when it's still small. Your instance has 139 users.. That's not intended as a slight. Hosting instances is good and I applaud you for it! But I wish it were easier to more equally share the load once the platform becomes more popular.
@nutomic@lemmy.ml It might be a good idea to default the Communities page to All instead of Local, to help push users into discovering other instances and promote them.
Lemmy.world is a new server, accepting signups. You're welcome there.
Point us to where the coin slot is. E.g. Patreon. We insert coin 🪙, you upgrade.
I have been wondering how cumbersome the Lemmy design will become for some. I love the idea that it is federated and decentralized however these are also major drawbacks for most average
users (i.e not multi account users.
Multiple accounts needed for maximum uptime
on different instances. What if I really like my username and its taken on another instance? If one instance is down and i comment with my other account will i then need to manage replies etc through different profiles? What happens if something spins up another instance of a similar domain so that they can get a username of someone to imitate them? I am sure these can be blocked after the fact or will other federated instances be automatically blocked.
What happens when someone gets bored of their instance and stops it, or it gets blocked, or they start getting unwanted attention. Does this mean all that content then goes into the ether?
Will this go down the route of whomever provides the instance with the most resources, best load balancing becoming the one
, blocking other instances and controlling it as if it were private and independent?
There are a lot wait and see
things, but I am excited to help and see what this great project becomes.
I've experienced a taste of this already. I checked the instance list a couple days ago, and didn't see one that stood out for my interests, so I created an account on the main lemmy.ml instance.
I just registered the same username on another but as far as I can tell, there is no way to merge or link these two accounts. So all the setup I've done and all the communities I've subscribed to, I have to do over again.
------
Another "issue" (a bug or feature?) I'm seeing is there are a lot of duplicate communities between the instances. I guess one will eventually "prevail" and become the defacto instance for that community.
Over at https://join-lemmy.org/ , when someone clicked on "Join a Server", they are presented with a list of instances, it's not that obvious that these are cross-accessible (yes, the homepage mentioned it, but not here), and people are bound to look for one with the most users.
Perhaps, add a simple TLI5 explanation/diagram explaining how Lemmy works on https://join-lemmy.org/instances .
(The documents are also too wordy for most people to care.)
I think lemmy will be bitten in the ass by not having considered clustering/horizontal scaling from the start. Federation alone as a scaling mechanism is only feasible for "nerds". But if the network wants to grow, we will need a few scale-able large hosted instances. And if their only choice is to scale vertically, there will be a hard limit (unless we put a good old Mainframe somewhere ^^).
Another downside of this design is: you can't run it with high availability. If there's only one process per instance, updating it will mean the whole instance is down. Sure, if all goes well this downtime is under a second. But if it doesn't go well or if a migration is needed, this might quickly become hours.
I think you probably underestimate how far one can get with "vertical" scaling. Here's the dockerfile: https://raw.githubusercontent.com/LemmyNet/lemmy/release/v0.17/docker/prod/docker-compose.yml
- It includes 4 different containers... so there's a way to scale out to 4 machines right away. Maybe not every container is doing an equal amount of work... but there's some amount of immediately available machine-splitting.
- I'm no expert, but I believe that at least the lemmy and lemmy-ui containers are stateless. If so, they're horizontally scalable already.
- Postgres then would likely be the main bottleneck. But postgres offers read-replicas, so again the write-load and the read-load can be hosted on separate machines. And if there's enough read-load, you can have many replicas.
Other comments from the admins have shown that lemmy.ml today is running on a single eight-core box and it's currently hosting 30k registered users and over 1k active. So how much more compute capacity can we throw at "vertical" scaling on the current software architecture?
- Just by going to a bigger single box, we can get 128 cores with no problem, a 16x bump in capacity. Does that get us to at least to 300k registered + 10k active?
- Splitting the containers onto 4 separate machines. Does that get us 2x more?
- Adding PG read-replicas and additional lemmy/lemm-ui containers would allow us to expand our instance footprint to maybe 6 physical machines should get us another 2x or more in performance.
Conservatively, that's 100x the computing capacity of the current hardware and could potentially support 1m registered users and 50k active. Now, I don't REALLY expect this to be possible today, there will be many software bottlenecks found along the way to scaling a single instance this large. But my point is that there's already a medium amount of horizontal scalability built into lemmy, and if the software doesn't fall over for algorithmic reasons (which is will at first), the current infrastructure architecture allows quite a lot of growth. There's plenty of time between now and a federation of million user instances to adopt a truly distributed storage backend if needed.
I tried like 4 or 5 instances before coming to lemmy.ml, but none of them were taking applications anymore. Finding even those was a hassle, since all I got was a list of domains without any details as to what the instance is about or if they allowed newcomers.
Now that I've setup everything, Lemmy does seem like nice alternative to Reddit, but as someone from the outside, all of this is daunting.
Another thing:
We do need more site admins to help us handle the applications and moderation.
For obvious reasons, we prefer ppl who have been here for a long time, and post / comment consistently. If you'd like to help us out, so that nutomic and I can focus on coding, that would be splendid.
If now is struggling then on June 12 will be a nightmare.
Reddit will go dark in protest, many messages to join Lemmy, most instances will be overloaded or even DDoS with so many users, like what happen with Mastodon.
I sent my registration yesterday, because I signed in another instance, one from my country, but I couldn't see all the post and no comments from lemmy.ml even thought is supposedly linked, so thank you for approving my account.
Even if I'm a tech savvy person I found the whole experience of joining lemmy pretty bad, I like the concept of federation, but I think it's too confusing to normal people, it really needs to be more seamless if you want to grow, how? idk, I was thinking some sort of replication, when you sign up, you are registered to the main instance (this) and given the choice to select other instances, automatically selecting let's say another 3 based on your location, then your account is synced in all the registered and linked instances, when you login if an instance is experiencing overload then it switches to another one. I don't know if this is realistic or out of the scope of Lemmy, or maybe against the philosophy of it. I'm just rambling.
I'm just glad that there is an open alternative for anonymous social interaction in this day of walled internet services such as discord, twitter, facebook etc. and I wish you all the success.
Agreed, someone needs to create an easy "sign up here" with a default option (maybe just randomize across various instances, not sure)
I wonder if a longer term solution would be to auto rotate the server list to bump less popular ones.
Users are likely going to see this as it's the "official" Lemmy instance when trying to join for the first time.
Any admins of instances that are accepting people, give your best elevator pitch!
Crashing with 994 users online?
that's not a good look
Well when you host your own website with your own funds, you try to not pay for unneeded perfs. But then a few hundreds people join, register, create content, upload images and videos, and suddenly your small VPS can't handle the load. And you're hesitant to scale up because it costs money, sometimes it costs time too because you need to migrate stuff, and maybe in a few days (or a few hours?) that load will disappear and now you're on a more expensive tier for nothing.
It's good to be popular. Congratulations on the growth!
If server crashes again the devs should just stop all registrations to this instance, people always follow path of least resistance and the first dev instance is just too easy to be stuck onto
Is there any way to migrate instance or do I make a new account entirely?
Got my instance running earlier today thanks to some helpful people in the Matrix chat. Intention at the time was to have some friends of mine join but I'm not opposed to having signups as well. Long live open source, federated software!
I guess every bigger instance needs to constantly promote a monthly support fee to keep the servers runnin. Something that most people already use and don't have to register first. Like PayPal or Patreon.
Edit: And I support Lemmy now on Patreon.
First post for me!
Sorry, I applied and got approved here. Still waiting to hear back from beehaw…
I’m really digging this UI compared to Reddit, but I am 99.9% a mobile user via the native Reddit app (don’t @ me!)
I am very tempted to setup my own instance. Wondering what resource usage looks like for an instance.
Is there any way to transfer an account to a different instance?
There is an issue open for this at https://github.com/LemmyNet/lemmy/issues/506, but it's not addressed yet. To migrate, you'd have to:
- Manually create a new account.
- Manually copy your settings.
- Create a post, comment, or set your profile description to point to your new account so people can manually find you at your new account.
- Your existing comments and posts would stay associated with your current/old account.
I wouldn't sweat it though. It's not currently "broken", and the lemmy.ml admins can shut off new signups if it becomes a major problem. My take is this is more about directing new signups elsewhere. The one instance can't hold every new user that's signing up right now.
I applied for a few other instances but this one came through first. Your downfall is being too good compared to the competition.
This is one of the biggest hurdles to get into Lemmy. I consider myself quite tech savvy but I am at a stage of my life that I cannot read hundreds of page of documentation just to use a forum.
There need to be a way to seamlessly move people from instance to another without them having to do it themselves or at the least a way way shorter documentation that goes to the point in one page.
well, what can be done to help with upgrades to this server? what does that entail?
I'm a noob. I created an account on beehaw and on lemmy.ml. That's because I see communities on one instance that I'm interested in and a different community on another instance. So if there's a technology community on both, how do I get to see all the technology posts without having to have two accounts?
This is really confusing for noobs like me. I'd just like to see one community to technology, one for Science, one for nintendo etc. I don't care it it's spread out amongst different servers to divvy up the load, but from the user side, it needs to be seamlessly integrated.
I'm still learning how all this works though. But I don't know how many folks that are more casual than me will be willing to figure it out. I hope they do though! It'll be worth it to leave reddit in the rearview mirror!
Edit: lawdy, I just figured it out. Local vs all on the communities list. It was right in front of my face. good grief!
Is there a feature to send a read-only/static link to Lemmy pages?
I’m envisioning a pre-cached version of the page that is updated hourly or so, rather than querying the database live for every comment on the thread. In a perfect world, these could also be offloaded to a CDN as static pages…