Technology

But we know exactly what they’re doing conceptually, and individually, and in aggregate.

Can you define and give examples of what you mean at each level here? Maybe we're just not understanding each other and mean the same thing.

Read your own sources from your previous post, that’s what they’re telling you.

The Anthropic one is saying they think they have a way to figure it out, but it hasn't been tested on large models. This is their last paragraph:

Our next challenge is to scale this approach up from the small model we demonstrate success on to frontier models which are many times larger and substantially more complicated. For the first time, we feel that the next primary obstacle to interpreting large language models is engineering rather than science.

They are literally only able to do this on a small one layer transformer model. GPT 3 has 96 layers and 175 billion parameters.

Also, in their linked paper:

A key challenge to our agenda of reverse engineering neural networks is the curse of dimensionality: as we study ever-larger models, the volume of the latent space representing the model's internal state that we need to interpret grows exponentially. We do not currently see a way to understand, search or enumerate such a space unless it can be decomposed into independent components, each of which we can understand on its own.

Under the Future Work heading:

Scaling the application of sparse autoencoders to frontier models strikes us as one of the most important questions going forward. We're quite hopeful that these or similar methods will work – Cunningham et al.'s work [17] seems to suggest this approach can work on somewhat larger models, and we have preliminary results that point in the same direction. However, there are significant computational challenges to be overcome.

How are you getting from that that this is a solved problem?

Concepts are indeed abstract but LLMs have no concepts in them, simply vectors. The vectors do not represent concepts in anything close to the same way that your thoughts do. They are not 1:1 with objects, they are not a “thought,” and anyway there is nothing to “think” them. They are literally only word weights, transformed to text at the end of the generation process.

Again, you aren't making sense here. Word/sentence vectors are literally a way to represent the concept of those words/sentences. That's what they were built for. That's how they are described. Let's take a step back to try to understand each other.

Are you trying to say that only human minds can understand concepts? I don't buy the human brains are magic bit, and neither does our current understanding of physics. Are you assuming I'm saying that LLMs are sentient, conscious, have thoughts or similar? I'm not. Jury's out on the thought thing, but I certainly don't believe the other two things. There's no magic with them, same with human brains. We just don't fully understand what happens inside either. Anthropic in the work I quoted is making good progress at that, and I think they may be pretty close, but in terms of LLMs (and not Small LMs), they are still a black box. We know the math behind them, the software, etc. We have some theories. We still do not understand. If you can prove otherwise, please provide me with a source. Stuff is happening really fast in AI, and maybe I blinked and missed something.

I think you're maybe having a hard time with using numbers to represent concepts. While a lot less abstract, we do this all the time in geometry. ((0, 0), (10, 0), (10, 10), (0, 10), (0, 0)) What's that? It's a square. Word vectors work differently but have the same outcome (albeit in a more abstract way).

the vectors do not represent single words, but tokens

I was talking word vectors where the vectors DO represent words. It's in the name. LLMs don't specifically use word vectors, but the embeddings they do use work similarly.

Tokens do not represent the meaning of a word/partial word/phrase, just the statistical use of that word given the data the word was found in.

You are correct tokens don't represent the meaning of a word. However, tokens are scalars. You are conflating tokens and embeddings / word vectors here. Tokens are used to simplify converting a string into a format a neural network can understand (a vector). If we used each ascii character in the input/output string as a vector input to the network, we'd have to have a lot more parameters than if we combine the characters in some way (i.e. tokens). As you said, they can be a word or a part of a word. There's no statistics embedded with the tokens (there are some methods of using statistics to choose what tokens to use, but that's decided before even training the model and can not ever change [with our current approach]). You can read here for more information on tokens. Or you can play around with the gpt3 tokenizer.

Your concept of a chair is an abstract thought representation of a chair. An LLM has vectors that combine or decompose in some way to turn into the word “chair,” but are not a concept of a chair or an abstract representation of a chair. It is simply vectors and weights, unrelated to anything that actually exists.

If you know Python, you should grab nltk and experiment with gensim, their word vectors.

model.most_similar(positive=['woman','king'], negative=['man'], topn = 1) [('queen', 0.71181...)]

king + woman - man = queen

Seems like an abstract representation of those things as concepts using math. For the record, word vectors are actually pretty understandable/understood by people because you can visualize them easily. When you do, you find similar concepts clustered together (this is how vector search works except with text embeddings). Anyway, it just really seems like linking numbers to concepts is not clicking with you, or you somehow think it's not possible. Reading up on computational linguistics might help.

That is obviously totally different in kind to human thought and abstract concepts. It is just not that, and not even remotely similar.

Yes, neural networks (although initially built thinking they were a computer version of a neuron), are a lot different from how actual brains work as we've learned in however many decades since they were invented. If you're saying that intelligence and understanding is limited to the human mind, then please point to some non-religious literature that backs up your assertion.

You say you are familiar with neural networks and AI but these are really basic underpinnings of those concepts that you are misunderstanding. Maybe you need to do more research here before asserting your experience?

I'm pretty confident in my understanding, though I'm always open to new ideas that are backed with peer reviewed research. I'm not going to get into a dick waving contest here, so I guess we'll have to agree to disagree.

As a side note, going back to your definition of intelligence. That was for psychology. I'll note that the Wikipedia page for Intelligence has this to say:

The definition of intelligence is controversial, varying in what its abilities are and whether or not it is quantifiable.

And so I'll reiterate that we don't have a good definition of intelligence.

Your concept of a chair is an abstract thought representation of a chair. An LLM has vectors that combine or decompose in some way to turn into the word “chair,” but are not a concept of a chair or an abstract representation of a chair. It is simply vectors and weights, unrelated to anything that actually exists.

Just so incredibly wrong. Fortunately, I'll have save myself time arguing with such a misunderstanding. GPT-4 is here to help:

This reads like a misunderstanding of how LLMs (like GPT) work. Saying an LLM's understanding is "simply vectors and weights" is like saying our brain's understanding is just "neurons and synapses". Both systems are trying to capture patterns in data. The LLM does have a representation of a chair, but it's in its own encoded form, much like our neurons have encoded representations of concepts. Oversimplifying and saying it's unrelated to anything that actually exists misses the point of how pattern recognition and information encoding works in both machines and humans.