C is a garrulous and compulsive liar and can't wait to spin you this incredible yarn about where they went last weekend. But C doesn't like to be caught in an obvious inconsistency.
Cleverly, you tricked C into starting their side of the conversation by revealing they were at home all last weekend. And now it's going to be a lot harder for C to spin their yarn. Never mind, they say the best way to lie is to tell the truth and nobody will notice when C starts sneaking in the fibs.
This is exactly what LLMs are. They're devices to spew out random text with correlations between different parts (rather than just being independent draws of tokens). But if you get them to start with a "prompt" they'll generate random text that's appropriately correlated with that prompt.
@dpiponi It doesn't stop with association though. For some tasks it has pretty strong causal model. I.e. you can test interventions and counterfactuals on it.
@rms80 @flaviusb @dpiponi Does it matter, though? There's no "fake addition" that brings you the same results in arithmetic, but without the "real math" behind it. The addition that works is just... addition. Ditto for every other task.
If the next token batch is merely predicted to be some bytes from Metasploit payload your system would be pwned.
@dpwiz @rms80 @dpiponi It's 'fake addition' because there is no addition behind it. Eg if you give it some addition task that is not in its training set, it gets the wrong answer. Like, testing LLMs - especially the ones being popularised like ChatGPT that have ingested ~ most of the internet - on 'addition problems' to prove what we might call 'math ability' or 'model building' suffers from both 'testing on the training set' and 'construct validity' issues.
@flaviusb @dpwiz @rms80 I'm not sure what you mean by "not in its training set" because I can give ChatGPT additions it probably hasn't seen before and get correct results.
(For my tests I used pseudorandomly generated integers because a neural network could, in principle, learn the kinds of biases humans have when generating "random" numbers.)
@dpiponi @dpwiz @rms80 Interesting. I have seen people generate pseudorandom test vectors like that and get reasonable rates of failure. I wonder what the difference in test vector generation or etc was. (From my perspective this is quite frustrating as the 'LLM' parts of eg ChatGPT can't train to produce arithmetic machines, but we have very little clarity on what eg 'ChatGPT' *actually* is, due to things like secrecy around parts of the architecture + the pervasive use of ghost labour).
@dpiponi @dpwiz @rms80 Like, to be clear, some nontrivial percentage of the things 'attached' to ChatGPT et al are more like the 'human fallback' of 'Brenda' (obligatory nplusone link: https://www.nplusonemag.com/issue-44/essays/human_fallback/ ), rather than like the integration of 'theories' you get with SMT, and their interventions only have continued effect through the rest of the conversation because of the same 'simulate memory via smuggling state in the input' technique that ChatGPT already uses for conversations.
@dpwiz @flaviusb @dpiponi fake addition: https://www.mathsisfun.com/numbers/addition-table.html