🌐
Videos Blog About Series πŸ—ΊοΈ
❓
πŸ”‘

Lifting LLMs to new heights with PCP πŸ”—
1786647031  

🏷️ blog

No, not that PCP. Nor "Praise, Correct Praise". Perl::Critic::Policy.

The core issue when dealing with LLM tooling to generate programs is the same problem working with other people. Unstated preferences means you get output which is unacceptable, but understandable because mind-reading is beyond both. Encoding this in things like skills is still not enough. Just like with your own internal wiki documenting how you "do things around here", you can be sure it will be routinely ignored.

No, you need something tighter in the OODA loop. This is the same reason that LSPs and IDEs are so popular with people too. Programming languages have so much to remember that "teacher, my brain is full" is a serious problem, only remediated by near-instant feedback. Given LLM agents are dumber and more forgetful than the any fauna you have ever worked with (it is a computer after all), the rule applies.

Sometimes even this isn't enough, and you need hooks. The earlier in the process, the better (as always). This also will save you tokens, as it doesn't waste work on things that won't pass muster.

The process that works (which I tentatively call "rudra" - "he who eliminates problems by their roots") is like this:

  1. Ask LLM to do thing, with existing LSP/Critic/Tidy hooked up.
  2. Review diff. If there is any issue, no matter how trivial, the change-set and LLM session gets the death penalty.
  3. Ask LLM to write critic policy or a hook which will prevent it from ever making such errors in judgement again.
  4. Repeat process from step 1
This is essentially simulated annealing at a layer of abstraction you don't normally encounter it. To be fair, these "agents" are actually a declarative data retrieval language, much like SQL. They simply have a query planner which is a Markov model. Most do not recognize this because it is at a higher level of abstraction than they are accustomed to.

The usage model here has similar limitations to SA. You will quickly find there are points at which you reach diminishing returns as regards encoding your preferences. This is no problem for most people, because "good enough" is.

25 most recent posts older than 1786647031
Size:
Jump to:
POTZREBIE
© 2020-2023 Troglodyne LLC