That’s what is wild about it. At any given point in time, the model is wholly consumed only with the very next token. Maybe that token is a running narrative of ‘reasoning’ or directly in the output, either way, the AI does not have anything to model anything beyond the very next token. It doesn’t have a destination in mind and is just finding the words to get there, it’s building it up word by word. The overall ‘meaning’ is an emergent property of just picking the very next token and seeing what happens.
Honestly, it’s shocking it works as well as it does. More shockingly, there are AI enthusiasts that argue that’s how the human brain works, which I can’t imagine someone going through life with every thought rooted in building it up word by word.
Well that description of a car is actually fairly close to the fundamentals, add an engine or motor and a steering wheel, and you’ve got it. Yes, a lot of engineering goes into the best possible realization of those basics, efficiency, suspension, safety, maintenance, and just a ton of more stuff, and it is a very valued execution above and beyond what, say, the Model T delivered. Automotive engineers have done hard and valuable work and complicated work, but no one is surprised that Model T led to faster, more comfortable, safer, more convenient vehicles that move around. It is a bit more surprising that LLM architecture works as well as it does while always focusing on the next token without ability to go further, best case running through and messing up and regenerating until you have operator appropriate output.
The ‘seahorse emoji’ was a pretty fun example of this at work, as it didn’t have a seahorse emoji, but since it wasn’t trying to generate the emoji, it started by building up the words to confirm and introduce the emoji since obviously there will be one, then putting up a wrong thing, then the words that would go after the wrong thing, but the weight still suggested there should be a correct answer and to start generating words for another try, and so on. “Reasoning” does the job of incurring this hit out of sight a lot of the time. Looking at the reasoning chains you’ll see this behavior a fair amount, that the model suggests words that build toward an answer but fails on the key word and retries until something tests right or it models that it tried enough and it can’t find the key word that would have been expected. It can of course digest it’s own output and summarize the result without showing the operator spinning out, but it at all times is operating on the fundamental principle of model+very cleverly managed context influencing an answer one token at a time and ideally discarding the first run.
I assumed that but everything I have seen as I dug deeper has been that at some level that is what is happening. If it is ‘reasoning’, it’s generating a ‘reasoning chain’ next token by next token and using that to influence the final output tokens. The reasoning chain is discarded and since the actual output is a continuation of the reasoning chain it may conceptually be described as allowing the model to ‘rethink’ things, but even as the generation of a ‘reasoning chain’ has results that more closely resemble reasoning, it is still a scenario where it’s building it one token at a time and we get to see meaning as an emergent property, rather than trying to find words to build to a more abstract concept like humans do. It just gets to throw away the intermediate work and the extra tokens manage to improve the ‘accuracy’ of the preserved final output.
The interesting bits are when it derives the likely hood of something being correct and does more passes, or splits the data apart in the first pass and opens up new context processes with specialized instructions to handle it. The code stuff goes full on ororborus on some models, writes out the code on one pass, checks for issues on another pass, runs the code looking for errors on a third pass and goes back to step 1 if it fails.
They’re getting a lot out of it for it primarily just being a weighted token generator wrapped in an orchestrator.
Particularly software development with very good and very quick tests allow rerolling and that is very appealing in those scenarios. Problems being that very good tests are rarer than people like to think and sometimes it just gets stuck in a loop. At work the other week someone set it at the task of fixing a bunch of build warnings that had accumulated over the years. It succeeded after burning through tokens to take 30 tries at it. It’s solution after all that hard thinking? It put // @ts-nocheck at the top of every file and called it a day.
But superficially, someone handed it a chore and didn’t have to think about it, and if no one looked deeper, then it was able to get to the desired behavior simply by rerunning the given task over and over without human intervention until it worked. Which is also broadly relatable as there’s a lot of humans in the industry acting broadly the same, but I’ve always been frustrated by those folks anyway.
Wide scoping like that rarely ends well. Give it the list, tell it to spin a new agent for every single issue and operate on them separately. If any of those agents run out of context memory, it will just shit the bed. Clarify that you need the test to actually work and not be commented out, and that the tests themselves still need to be viable and indicate a healthy working app, not just report that it’s healthy. Trying to get tests happy in AI is like making wishes on a monkey’s paw. Then assign someone to look after it because that bastard will chew through 10’s of thousands of dollars to fix a laundry list of tests, which in my dev experience will be all broken and ignored in 2-3 minor releases.
You can generally get it to do things, even somewhat complicated things, but it’s not easy, and the shit it gives you back while you’re trying to convince it to do the job is the same stuff you’d fire an employee over.
My favorite thing is to make it make ansible tools, They’re easy to read, easy to lock down, keys and passwords go in a vault that it’s forbidden to read. Then I have it write runbooks on how to operate the playbooks. I then make an index of runbooks, and ask it to do them. It’s an ansible job, so I can read it easily and make sure it’s not able to delete my drive or my production database.
If I lose access to cheap tokens, everything I’ve making is still usable, and for now, I can spend my time working on stuff that’s not simple enough that it can handle it well.
Its just a ball rolling down a hill trying to find the lowest point, but its like a super fancy ball with a ton of rules, and the hills are really detailed with like little walls and bridges and stuff.
Well yes and no. it is steered by a buffer of context as well that sorts/ranks/informs what the next word should be. That context differentiates if you are talking about apple the fruit or apple the company or apple the device. Heres a great overview if anyone is interested. And no, its not my video. Its a youtube intro to how AI works. Best watched with duckduckgo browser which trims out youtubes overly frequent ad interruptions.
https://www.youtube.com/watch?v=OYvlznJ4IZQ
But that context is a mix of model output and other sources. The model output portion was generated token by token, and is combined in interesting ways with things like human response, search results, software output. It’s still a backward looking mechanism, rather than having established a concept as a goal and then trying to build the words to reach that concept like we do.
Size and strategy for managing the context has been critical for improved subjective results, but it still doesn’t exhibit the behavior of the words as a tool to address some concept, everything about the model is about the words themselves. So we end up with something very good at generating what seems right and there’s a super high chance of what seems to be right actually being right. Especially when the software can automatically execute commands and the good or bad results reach into the context window, enabling it to effectively get automatically second guessed. The potential for automatic verification in some scenarios automatically feeding the context window is what makes it particularly appealing for software folks, though not universal.
That’s what is wild about it. At any given point in time, the model is wholly consumed only with the very next token. Maybe that token is a running narrative of ‘reasoning’ or directly in the output, either way, the AI does not have anything to model anything beyond the very next token. It doesn’t have a destination in mind and is just finding the words to get there, it’s building it up word by word. The overall ‘meaning’ is an emergent property of just picking the very next token and seeing what happens.
Honestly, it’s shocking it works as well as it does. More shockingly, there are AI enthusiasts that argue that’s how the human brain works, which I can’t imagine someone going through life with every thought rooted in building it up word by word.
My masters degree in intelligent systems wept in the corner after seeing your explanation.
But I guess cars are just four wheels and a fancy basket people sit in.
Well that description of a car is actually fairly close to the fundamentals, add an engine or motor and a steering wheel, and you’ve got it. Yes, a lot of engineering goes into the best possible realization of those basics, efficiency, suspension, safety, maintenance, and just a ton of more stuff, and it is a very valued execution above and beyond what, say, the Model T delivered. Automotive engineers have done hard and valuable work and complicated work, but no one is surprised that Model T led to faster, more comfortable, safer, more convenient vehicles that move around. It is a bit more surprising that LLM architecture works as well as it does while always focusing on the next token without ability to go further, best case running through and messing up and regenerating until you have operator appropriate output.
The ‘seahorse emoji’ was a pretty fun example of this at work, as it didn’t have a seahorse emoji, but since it wasn’t trying to generate the emoji, it started by building up the words to confirm and introduce the emoji since obviously there will be one, then putting up a wrong thing, then the words that would go after the wrong thing, but the weight still suggested there should be a correct answer and to start generating words for another try, and so on. “Reasoning” does the job of incurring this hit out of sight a lot of the time. Looking at the reasoning chains you’ll see this behavior a fair amount, that the model suggests words that build toward an answer but fails on the key word and retries until something tests right or it models that it tried enough and it can’t find the key word that would have been expected. It can of course digest it’s own output and summarize the result without showing the operator spinning out, but it at all times is operating on the fundamental principle of model+very cleverly managed context influencing an answer one token at a time and ideally discarding the first run.
its not that simple. whatever opinion you might on llms have you have to agree this is oversimplifying at best.
I assumed that but everything I have seen as I dug deeper has been that at some level that is what is happening. If it is ‘reasoning’, it’s generating a ‘reasoning chain’ next token by next token and using that to influence the final output tokens. The reasoning chain is discarded and since the actual output is a continuation of the reasoning chain it may conceptually be described as allowing the model to ‘rethink’ things, but even as the generation of a ‘reasoning chain’ has results that more closely resemble reasoning, it is still a scenario where it’s building it one token at a time and we get to see meaning as an emergent property, rather than trying to find words to build to a more abstract concept like humans do. It just gets to throw away the intermediate work and the extra tokens manage to improve the ‘accuracy’ of the preserved final output.
The interesting bits are when it derives the likely hood of something being correct and does more passes, or splits the data apart in the first pass and opens up new context processes with specialized instructions to handle it. The code stuff goes full on ororborus on some models, writes out the code on one pass, checks for issues on another pass, runs the code looking for errors on a third pass and goes back to step 1 if it fails.
They’re getting a lot out of it for it primarily just being a weighted token generator wrapped in an orchestrator.
Particularly software development with very good and very quick tests allow rerolling and that is very appealing in those scenarios. Problems being that very good tests are rarer than people like to think and sometimes it just gets stuck in a loop. At work the other week someone set it at the task of fixing a bunch of build warnings that had accumulated over the years. It succeeded after burning through tokens to take 30 tries at it. It’s solution after all that hard thinking? It put // @ts-nocheck at the top of every file and called it a day.
But superficially, someone handed it a chore and didn’t have to think about it, and if no one looked deeper, then it was able to get to the desired behavior simply by rerunning the given task over and over without human intervention until it worked. Which is also broadly relatable as there’s a lot of humans in the industry acting broadly the same, but I’ve always been frustrated by those folks anyway.
Wide scoping like that rarely ends well. Give it the list, tell it to spin a new agent for every single issue and operate on them separately. If any of those agents run out of context memory, it will just shit the bed. Clarify that you need the test to actually work and not be commented out, and that the tests themselves still need to be viable and indicate a healthy working app, not just report that it’s healthy. Trying to get tests happy in AI is like making wishes on a monkey’s paw. Then assign someone to look after it because that bastard will chew through 10’s of thousands of dollars to fix a laundry list of tests, which in my dev experience will be all broken and ignored in 2-3 minor releases.
You can generally get it to do things, even somewhat complicated things, but it’s not easy, and the shit it gives you back while you’re trying to convince it to do the job is the same stuff you’d fire an employee over.
My favorite thing is to make it make ansible tools, They’re easy to read, easy to lock down, keys and passwords go in a vault that it’s forbidden to read. Then I have it write runbooks on how to operate the playbooks. I then make an index of runbooks, and ask it to do them. It’s an ansible job, so I can read it easily and make sure it’s not able to delete my drive or my production database.
If I lose access to cheap tokens, everything I’ve making is still usable, and for now, I can spend my time working on stuff that’s not simple enough that it can handle it well.
Its just a ball rolling down a hill trying to find the lowest point, but its like a super fancy ball with a ton of rules, and the hills are really detailed with like little walls and bridges and stuff.
Well yes and no. it is steered by a buffer of context as well that sorts/ranks/informs what the next word should be. That context differentiates if you are talking about apple the fruit or apple the company or apple the device. Heres a great overview if anyone is interested. And no, its not my video. Its a youtube intro to how AI works. Best watched with duckduckgo browser which trims out youtubes overly frequent ad interruptions. https://www.youtube.com/watch?v=OYvlznJ4IZQ
But that context is a mix of model output and other sources. The model output portion was generated token by token, and is combined in interesting ways with things like human response, search results, software output. It’s still a backward looking mechanism, rather than having established a concept as a goal and then trying to build the words to reach that concept like we do.
Size and strategy for managing the context has been critical for improved subjective results, but it still doesn’t exhibit the behavior of the words as a tool to address some concept, everything about the model is about the words themselves. So we end up with something very good at generating what seems right and there’s a super high chance of what seems to be right actually being right. Especially when the software can automatically execute commands and the good or bad results reach into the context window, enabling it to effectively get automatically second guessed. The potential for automatic verification in some scenarios automatically feeding the context window is what makes it particularly appealing for software folks, though not universal.