Hey everyone!

I think it's time we had a fosai model on HuggingFace. I'd like to start collecting ideas, strategies, and approaches for fine-tuning our first community model.

I'm open to hearing what you think we should do. We will release more in time. This is just the beginning.

For now, I say let's pick a current open-source foundation model and fine-tune on datasets we all curate together, built around a loose concept of using a fine-tuned LLM to teach ourselves more bleeding-edge technologies (and how to build them using technical tools and concepts).

FOSAI is a non-profit movement. You own everything fosai as much as I do. It is synonymous with the concept of FOSS. It is for everyone to champion as they see fit. Anyone is welcome to join me in training or tuning using the workflows I share along the way.

You are encouraged to leverage fosai tools to create and express ideas of your own. All fosai models will be licensed under Apache 2.0. I am open to hearing thoughts if other licenses should be considered.

We're Building FOSAI Models! 🤖

Our goal is to fine-tune a foundation model and open-source it. We're going to start with one foundation family with smaller parameters (7B/13B) then work our way up to 40B (or other sizes), moving to the next as we vote on what foundation we should fine-tune as a community.

Fine-Tuned Use Case ☑️

Technical

• FOSAI Model Idea #1 - Research & Development Assistant
• FOSAI Model Idea #2 - Technical Project Manager
• FOSAI Model Idea #3 - Personal Software Developer
• FOSAI Model Idea #4 - Life Coach / Teacher / Mentor
• FOSAI Model Idea #5 - FOSAI OS / System Assistant

Non-Technical

• FOSAI Model Idea #6 - Dungeon Master / Lore Master
• FOSAI Model Idea #7 - Sentient Robot Character
• FOSAI Model Idea #8 - Friendly Companion Character
• FOSAI Model Idea #9 - General RPG or Sci-Fi Character
• FOSAI Model Idea #10 - Philosophical Character

OR

FOSAI Foundation Model ☑️

Foundation Model ☑️

(Pick one)

• Mistral
• Llama 2
• Falcon
• ..(Your Submission Here)

Model Name & Convention

• snake_case_example
• CamelCaseExample
• kebab-case-example

0.) FOSAI ☑️

• fosai-7B
• fosai-13B

1.) FOSAI Assistant ☑️

• fosai-assitant-7B
• fosai-assistant-13B

2.) FOSAI Atlas ☑️

• fosai-atlas-7B
• fosai-atlas-13B

3.) FOSAI Navigator ☑️

• fosai-navigator-7B
• fosai-navigator-13B

4.) ?

Datasets ☑️

• TBD!
• What datasets do you think we should fine-tune on?

Alignment ☑️

To embody open-source mentalities, I think it's worth releasing both censored and uncensored versions of our models. This is something I will consider as we train and fine-tune over time. Like any tool, you are responsible for your usage and how you choose to incorporate into your business and/or personal life.

License ☑️

All fosai models will be licensed under Apache 2.0. I am open to hearing thoughts if other licenses should be considered.

This will be a fine-tuned model, so it may inherit some of the permissions and license agreements as its foundation model and have other implications depending on your country or local law.

Generally speaking, you can expect that all fosai models will be commercially viable through the selection process of its foundation family and the post-processing steps that are fine-tuning the model.

Costs

I will be personally covering all training and deployment costs. This may change if I choose to put together some sort of patronage, but for now - don't worry about this. I will be using something like RunPod or some other custom deployed solution for training.

Cast Your Votes! ☑️

Share Your Ideas & Vote in the Comments Below! ✅

What do you want to see out of this first community model? What are some of the fine-tuning ideas you've wanted to try, but never had the time or chance to test? Let me know in the comments and we'll brainstorm together.

I am in no rush to get this out, so I will leave this up for everyone to see and interact with until I feel we have a solid direction we can all agree upon. There will be plenty of more opportunities to create, curate, and customize more fosai models I plan to release in the future.

Update [10/25/23]: I may have found a fine-tuning workflow for both Llama (2) and Mistral, but I haven't had any time to validate the first test run. Once I have a chance to do this and test some inference I'll be updating this post with the workflow, the models, and some sample output with example datasets. Unfortunately, I have ran out of personal funds to allocate to training, so it is unsure when I will have a chance to make another attempt at this if this first attempt doesn't pan out. Will keep everyone posted as we approach the end of 2023.

🤖 Happy FOSAI Friday! 🚀

Friday, September 29, 2023

HyperTech News Report #0002

Hello Everyone!

Welcome back to the HyperTech News Report! This week we're seeing some really exciting developments in futuristic technologies. With more tools and methods releasing by the day, I feel we're in for a renaissance in software. I hope hardware is soon to follow.. but I am here for it! So are you. Brace yourselves. Change is coming! This next year will be very interesting to watch unfold.

Table of Contents

• New Foundation Model!
• Metaverse Developments
• NVIDIA NeMo Guardrails
• Tutorial Highlights

Community Changelog

• Cleaned up some old content (let me know if you notice something that should be archived or updated)

Image of the Week

This image of the week comes from a DALL-E 3 demonstration by Will Depue. This depicts a popular image for diffusion models benchmarks - the astronaut riding a horse in space. Apparently this was hard to get right, and others have had trouble replicating it - but it seems to have been generated by DALL-E 3 nevertheless. Curious to see how it stacks up against other diffusers when its more widely available.

New Foundation Model!

There have been many new models hitting HuggingFace on the daily. The recent influx has made it hard to benchmark and keep up with these models - so I will be highlighting a hand select curated week-by-week, exploring these with more focus (a few at a time).

If you have any model favorites (or showcase suggestions) let me know what they are in the comments below and I'll add them to the growing catalog!

This week we're taking a look at Mistral - a new foundation model with a sliding attention mechanism that gives it advantages over other models. Better yet - the mistral.ai team released this new model under the Apache 2.0 license. Massive shoutout to this team, this is huge for anyone who wants more options (commercially) outside of Llama 2 and Falcon families.

From Mistralai:

The best 7B, Apache 2.0.. Mistral-7B-v0.1 is a small, yet powerful model adaptable to many use-cases. Mistral 7B is better than Llama 2 13B on all benchmarks, has natural coding abilities, and 8k sequence length. It’s released under Apache 2.0 licence, and we made it easy to deploy on any cloud.

Learn More

Mistralai

• https://huggingface.co/mistralai/Mistral-7B-v0.1
• https://mistral.ai/news/announcing-mistral-7b/
• https://docs.mistral.ai/quickstart/

TheBloke (Quantized)

• https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF https://huggingface.co/TheBloke/Mistral-7B-v0.1-GPT

More About GPTQ

• https://github.com/ggerganov/llama.cpp/pull/1827

More About GGUF

• https://github.com/ggerganov/llama.cpp/pull/2398#issuecomment-1682837610

Metaverse Developments

Mark Zuckerberg had his third round interview on the Lex Fridman podcast - but this time, in the updated Metaverse. This is pretty wild. We seem to have officially left uncanny valley territory. There are still clearly bugs and improvements to be made - but imagine the possibilities of this mixed reality technology (paired with VR LLM applications).

The type of experiences we can begin to explore in these digital realms are going to evolve into things of true sci-fi in our near future. This is all very exciting stuff to look forward to as AI proliferates markets and drives innovation.

What do you think? Zuck looks more human in the metaverse than in real life.. mission.. success?

Click here for the podcast episode.

NVIDIA NeMo Guardrails

If you haven't heard about NeMo Guardrails, you should check it out. It is a new library and approach for aligning models and completing functions for LLMs. It is similar to LangChain and LlamaIndex, but uses an in-house developed language from NVIDIA called 'colang' for configuration, with NeMo Guardrail libraries in python friendly syntax.

This is still a new and unexplored tool, but could provide some interesting results with some creative applications. It is also particularly powerful if you need to align enterprise LLMs for clients or stakeholders.

Learn More

Tutorial Highlights

Mistral 7B - Small But Mighty 🚀 🚀

• https://www.youtube.com/watch?v=z4wPiallZcI&ab_channel=PromptEngineering

Chatbots with RAG: LangChain Full Walkthrough

• https://www.youtube.com/watch?v=LhnCsygAvzY&ab_channel=JamesBriggs

NVIDIA NeMo Guardrails: Full Walkthrough for Chatbots / AI

• https://www.youtube.com/watch?v=SwqusllMCnE&t=1s&ab_channel=JamesBriggs

Author's Note

This post was authored by the moderator of !fosai@lemmy.world - Blaed. I make games, produce music, write about tech, and develop free open-source artificial intelligence (FOSAI) for fun. I do most of this through a company called HyperionTechnologies a.k.a. HyperTech or HYPERION - a sci-fi company.

Thanks for Reading!

If you found anything about this post interesting, consider subscribing to !fosai@lemmy.world where I do my best to keep you informed about free open-source artificial intelligence as it emerges in real-time.

Our community is quickly becoming a living time capsule thanks to the rapid innovation of this field. If you've gotten this far, I cordially invite you to join us and dance along the path to AGI and the great unknown.

Come on in, the water is fine, the gates are wide open! You're still early to the party, so there is still plenty of wonder and discussion yet to be had in our little corner of the digiverse.

This post was written by a human. For other humans. About machines. Who work for humans for other machines. At least for now...

Until next time!

Blaed