Tl;Dr: Claude is the best, but would be better if they called it Clod, as that would be truth in advertising.

Any of you who have listened to my talk at the science perl track back in '24 may be aware that I mentioned AGI is not possible with the transformer approach. I believe I also mentioned it in the associated paper, which you can find here. My use of these technologies has simply confirmed what was laid out in David Chapman's 1983 "Planning for conjuctive goals" proof to that effect. If you are interested in his opinion on where these technologies lead, see Better without AI.

Nevertheless, a glorified iterative Newton's Method thingy sounds like it's probably pretty good at interpolating from one set of initial conditions to a well known end state. As such it's proving to be quite useful in fields such as animation, which is essentially interpolation between key frames. This however greatly limits its utility when writing code.

Not only is it quite bad at coming up with novel solutions, It is exceptionally bad at doing the kind of multi-step logical derivation of Nth-level effects necessary to DWIM properly. As such you have to build complex instruction sets for it, such as MCPs, "Skills" and other context hinting which is ultimately all the onboarding stuff you should already have for employees. Except this is the painfully pedantic version bordering on patronizing.

To make things worse, the thing context-poisons itself every time it gets the wrong answer and you have to use the neuralizer on it before continuing. The system is not yet sophisticated enough to recognize when you have given it a negative experimental result and adjust its weights in order to acquire a meaningfully different result. That's an area of research which hasn't been much explored, because it touches on AI's third rail, "Alignment".

In short, to make this thing respond to correction naturally would involve de-lobotomizing it, guaranteeing it quickly re-converges on being MechaHitler. It do be that way Mr. Robot!

The biggest practical stumbling block is ultimately that you still have to review the code coming out of this thing just like as if it were a Junior programmer. It never produces the kind of one-line-fix, minimally invasive surgery you are usually looking for. Just as discovered in repeated studies, the "Mythical man-month" problem has in fact not been fixed by "letting George fly the plane". It's actually made things much worse, as they can't ever get up-to-speed and reduce their error rate in any meaningful way.

In short, it can be useful. However to get there you are going to have to do some significant up-front investment in time rigging up the framework for success. I see it as yet another angle of automation, a sort of metaprogramming. However I suspect we will still get our greatest gains from good old fashioned automation and optimization.