this post was submitted on 03 Jul 2023
266 points (98.5% liked)

Selfhosted

40157 readers
588 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
 

Just started self hosting this instance. Nothing on the docs mentioned anything about storage considerations.

you are viewing a single comment's thread
view the rest of the comments
[–] ruud@lemmy.world 280 points 1 year ago (7 children)

This is lemmy.world after 4 weeks:

58G	pictrs
34G	postgres
[–] i_lost_my_bagel@seriously.iamincredibly.gay 79 points 1 year ago (2 children)

Considering this is going to be around a 5 user instance at most I think I'll be good for awhile. Thanks!

[–] manitcor@lemmy.intai.tech 56 points 1 year ago* (last edited 1 year ago) (1 children)

im running 50 users right now, subbed to A LOT of communities, seeing db growth of about 100mb per day.

[–] Pleonasm@programming.dev 18 points 1 year ago (2 children)

That seems high when you extrapolate that to 10000 users, like a larger instance might have.

[–] terribleplan@lemmy.nrd.li 67 points 1 year ago* (last edited 1 year ago) (2 children)

It's all about how many communities your user(s) subscribe to since your instance basically acts as a mirror for those.

My instance has been running for 23 days, and I am pretty much the only active local user:

7.3G    pictrs
5.3G    postgres

edit: I may have a slight ~~Reddit~~ Lemmy problem

[–] Pleonasm@programming.dev 9 points 1 year ago (1 children)

So if you're the only user (let's assume for ease) then, that represents all the updates (posts, comments, votes) from each community that you are subscribed to?

[–] terribleplan@lemmy.nrd.li 16 points 1 year ago (2 children)

Yeah, and I purposely subscribe to (or sometimes have a dedicated "federation helper bot" account I run subscribe to) most of the most popular communities on the most popular instances so I can get a decent sampling of what's going on in the fediverse on the "All" feed. So I assume my storage usage is maybe a bit higher than what an "average" single-user instance may be...

[–] bdonvr@thelemmy.club 4 points 1 year ago

lmao same here. I have a spare account that I use to sub to everything worth subbing to. I haven't automated it yet though.

[–] ipkpjersi@lemmy.one 3 points 1 year ago (1 children)

Ooh, that's a really good idea, I need a federation helper bot/account when I start self-hosting a Lemmy instance!

[–] terribleplan@lemmy.nrd.li 3 points 1 year ago

Yeah it's not automated or anything, I just pop an incognito window and use it when there is a communitI think is worth seeing sometimes in "All" (or just for archiving purposes) but don't want to clutter "Subscribed". I may make something to auto-subscribe to communities meeting some criteria or something at some point in the future...

[–] alexanderkehr@lemmy.alexware.systems 3 points 1 year ago (2 children)

Do you also post stuff? I mean my instance is only about an hour old, but I've subscribed to some communities, yet I don't see the picture service consuming the S3 storage I've configured

[–] terribleplan@lemmy.nrd.li 13 points 1 year ago

Lemmy caches every thumbnail of every post for like a month or something using Pictrs, so that storage will eventually hit a sort of equilibrium and start growing much more slowly (only reflecting post/thumbnail volume during the cache time).

Between profile images, community banners/icons, post images etc. there are probably a few dozen images that will be sticking around for the long haul at the moment.

[–] bdonvr@thelemmy.club 5 points 1 year ago

Your instance only caches thumbnails, so it won't take much space. The full images are served from the remote instance. So you basically only store whatever your users upload.

[–] Stubborn9867@lemmy.jnks.xyz 23 points 1 year ago (1 children)

It won't scale linearly. A lot of those users will be subscribed to subs the instance is already replicating. It would only be new subs that would add to the growth.

[–] bdonvr@thelemmy.club 4 points 1 year ago

And only active subs. And even then, it's just text and tiny thumbnails.

[–] Dran_Arcana@lemmy.world 15 points 1 year ago (1 children)

Question if you know: does a lemmy instance have to be publically accessable to work? Like, if I make an instance on my homelab can the instance "fetch" content and serve it faster locally? Could I reply to a post and have others see it? Etc

[–] msinfo32@sh.itjust.works 1 points 1 year ago

wondering this also! wouldnt it require a domain for your account though?

[–] HappyHam@lemm.ee 18 points 1 year ago (1 children)

Now I wonder how viable it would be to support video hosting. The answer is almost certainly "God no!"

[–] GatoB@lemmy.world 1 points 1 year ago

It is viable through other hostings

[–] Grimr0c@lemmy.world 16 points 1 year ago

Honestly, Less than I thought!

[–] mruczek@lemm.ee 14 points 1 year ago (1 children)

Interesting, I thought it would be waaayyy more

[–] BigWigglyStyle@lemmynsfw.com 17 points 1 year ago

At the end of the day the vast majority of what needs to be saved is text. If media content is embedded, the the server just has to save the path to the file not the file itself.

[–] myersguy@lemmy.simpl.website 11 points 1 year ago

Wow, that is surprisingly not bad given the size of the instance!

[–] bdonvr@thelemmy.club 11 points 1 year ago* (last edited 1 year ago)

Yeah lemmy seems to use just about nothing for data storage.

[–] lightrush@lemmy.ca 5 points 1 year ago* (last edited 1 year ago)

Feels like this will benefit from some sort of fuzzy deduplication in the pictrs storage. I bet there are a lot of similar pics in there. E.g. if one pic or a gif is very similar to another, say just different quality or size, or compression, it should keep only one copy. It might already do this for the same files uploaded by different people as those can be compared trivially via hashing, but I doubt it does similarity based deduplication.