this post was submitted on 17 Jul 2023
16 points (100.0% liked)

Free Open-Source Artificial Intelligence

2806 readers
1 users here now

Welcome to Free Open-Source Artificial Intelligence!

We are a community dedicated to forwarding the availability and access to:

Free Open Source Artificial Intelligence (F.O.S.A.I.)

More AI Communities

LLM Leaderboards

Developer Resources

GitHub Projects

FOSAI Time Capsule

founded 1 year ago
MODERATORS
Blaed@lemmy.world
fosai@lemmy.world
16
Any good self-hosted LLM for software development? (lemmy.fakeplastictrees.ee)
submitted 1 year ago by ghostwolf@lemmy.fakeplastictrees.ee to c/fosai@lemmy.world
5 comments fedilink hide all child comments
 

Hi,

I'm looking for something that could generate code and provide technical assistance on a level similar to ChatGPT4 or at least 3.5. I'm generally satisfied with it, but for privacy and security reasons I can't share some details and code listings with OpenAI. Hence, I'm looking for a self-hosted alternative.

Any recommendations? If nothing specific comes to mind, what parameters should I look at in my search? I've never worked with LLMs yet and there are so many of them. I just know that I could use oobabooga/text-generation-webui to access a model in a friendly way.

Thanks in advance.

you are viewing a single comment's thread
view the rest of the comments
[–] toxuin@lemmy.ca 1 points 1 year ago

There is a bit of a conundrum here: in order to run a model that is any good in coding you want it to have a lot of parameters (the more the better) but also since it’s code and not some spoken language - precision matters here. Home hardware like 3090 is able to run ~30b models, but there is a catch - it just fits and only in quantized form = with 4x worse precision typically. Unless we see some breakthrough here that makes inference of huge models possible at full precision - the hosted AI will always be better for coding. Not saying such breakthrough is impossible though - quite the opposite in my opinion.