this post was submitted on 30 Jun 2023
13 points (100.0% liked)

LocalLLaMA

2221 readers
1 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 1 year ago
MODERATORS
pax@sh.itjust.works
SkySyrup@sh.itjust.works
noneabove1182@sh.itjust.works
13
Vicuna-33B-1-3-SuperHOT-8K-GPTQ (huggingface.co)
submitted 1 year ago by notfromhere@lemmy.one to c/localllama@sh.itjust.works
9 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] actuallyacat@sh.itjust.works 3 points 1 year ago

That's what llama.cpp and kobold.cpp do, the KV cache is the last thing that gets offloaded so you can offload weights and keep the cache in RAM. Although neither support SuperHOT right now.

MQA models like Falcon-40B or MPT are going to be better for large context lengths. They have a tiny KV cache so even blown up 16x it's not a problem.