but if you’re going to criticize them, please do it accurately
You should take your own advice.
They do know the meaning of words, but only in relation to other words.
That’s only one part to meaning and it’s the only one LLMs have. It’s facinating what this one part can do, but we don’t operate this way. LLM have no world model, no logic model to associate a word to. It doesn’t think, it’s still just and input - output machine.
It’s not a statistical thing like word frequency pattern.
Instead, they’re doing math on words in a several hundred-thousand dimensional array where placement on this grid indicates the meaning of the word
I’m sorry, how is this not statistics?
The training is by it’s very nature statistical. We give millions of text inputs with expected outputs and tune the model until they match. How is this anything but statistics??
It developed this array via training on terabytes of text, but it’s not storing a copy of that text, nor looking it up, nor copying anything from it
Yes and no? Yes - it’s not storing a copy of the training data in the text form. No - it most definetly can “memorize” text, if that’s not a copy I don’t know what is.
I could memorize foreign script text without understanding it and then I could recreate it. Did I make a copy? no. Can I make a copy? yes.
You should take your own advice.
That’s only one part to meaning and it’s the only one LLMs have. It’s facinating what this one part can do, but we don’t operate this way. LLM have no world model, no logic model to associate a word to. It doesn’t think, it’s still just and input - output machine.
I’m sorry, how is this not statistics?
The training is by it’s very nature statistical. We give millions of text inputs with expected outputs and tune the model until they match. How is this anything but statistics??
Yes and no? Yes - it’s not storing a copy of the training data in the text form. No - it most definetly can “memorize” text, if that’s not a copy I don’t know what is.
I could memorize foreign script text without understanding it and then I could recreate it. Did I make a copy? no. Can I make a copy? yes.