this post was submitted on 14 Jun 2023
1 points (100.0% liked)

Machine Learning

478 readers
2 users here now

A community for posting things related to machine learning

Icon base by Lorc under CC BY 3.0 with modifications to add a gradient

founded 1 year ago
MODERATORS
Ategon@programming.dev
Akisamb@programming.dev
ericjmorey@programming.dev
1
Has anybody replaced attention with Hyena Hierarchy (programming.dev)
submitted 1 year ago by Akisamb@programming.dev to c/machine_learning@programming.dev
0 comments fedilink hide all child comments
 

Hyena Hierarchy seems to aim to be a drop-in replacement for attention : https://arxiv.org/pdf/2302.10866.pdf

It looks good on paper, but I haven't been able to find anybody using it in a model. Does anyone have an example of a code or implementation ? Is there really a big improvement on long context lengths ?

no comments (yet)
sorted by: hot top controversial new old
there doesn't seem to be anything here