this post was submitted on 30 Sep 2024

10 points (100.0% liked)

The Linux Lugcast Podcast

171 readers

4 users here now

website: https://www.linuxlugcast.com/

mumble chat: lugcast.minnix.dev in the lugcast room

email: feedback@linuxlugcast.com

matrix room: https://matrix.to/#/#lugcast:minnix.dev

youtube: https://www.youtube.com/@thelinuxlugcast/videos

peertube: https://nightshift.minnix.dev/c/linux_lugcast/videos

founded 1 year ago

MODERATORS

minnix@lemux.minnix.dev

Honkeymagoo@lemux.minnix.dev

10

Exploring Llamafile: Mozilla's Attempt in the World of Open Source AI (itsfoss.com)

submitted 2 weeks ago by minnix@lemux.minnix.dev to c/linux_lugcast@lemux.minnix.dev

4 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] muntedcrocodile@lemm.ee 2 points 1 week ago (1 children)

The issue isnt if its easy to use the issue is a matter of compute. A majority of people on the internet only have mobile access. We need a way to let the masses utilise distributed compute in a secure and private way.

[–] JackGreenEarth@lemm.ee 1 points 1 week ago (1 children)

You can run many 7B models on phones with 8GB RAM

[–] muntedcrocodile@lemm.ee 3 points 1 week ago (1 children)

I find the 7b models dont have the capabilities id like and i cant image the tokens per second would be very good.

[–] minnix@lemux.minnix.dev 2 points 1 week ago

Yeah running it and it actually being useful are two very different things.