Anthropic Declares That The Next Big Step For Humans And AI Is AI That Builds Itself Via Recursive Self-Improvement

Aiming For Pinnacle AI

Avid readers know that I have repeatedly explored the vexing question of how humanity will advance AI to become a kind of pinnacle AI. That being said, there is already ongoing debate about what pinnacle AI itself will be; see my analysis at the link here.

Some insist that pinnacle AI would be AGI (artificial general intelligence), such that AI would be on par with the intellectual acumen of humans on all matters of human knowledge; see my discussion at the link here. Others argue that AGI is not truly pinnacle AI but instead a stepping stone on the way to ASI (artificial super intelligence). ASI would be AI that is well-above human intellect on all realms of human knowledge; see my discussion at the link here.

You are welcome to pick either AGI or ASI or possibly noodle on some other version of what pinnacle AI is going to be. The question at hand is how we will get there. Even if we aren’t sure or haven’t yet decided on what the pinnacle really is, the allied question is what will get us to the vaunted pinnacle.

Ways To Craft AI

Let’s envision that there are three primary ways to advance AI:

• (1) Humans coding. Humans perform hand-crafting to advance AI.
• (2) Humans-AI coding. Humans and AI collaborate together toward advancing AI.
• (3) AI coding. AI codes without human assistance to advance AI.

In the first case, humans are in the driver’s seat. Software developers and engineers do the hand-crafting and laboriously expend their time and effort to push AI ahead. This includes coming up with new designs, architecture, coding, testing, fielding, and any other elements of the AI system development life cycle (AI SDLC). They might employ automated tools along the way, but it is still principally human-led.

The second case consists of humans and AI working collaboratively on advancing AI. You might have heard of vibe coding, whereby you give AI some natural language instructions about what you want a program to do, and the AI generates the code. For my in-depth assessment of the present and future of vibe coding, see the link here. The AI is acting at the behest of a human. It generates code based on what the human requests. In that sense, AI can be advanced by humans working hand-in-hand with AI to do so.

The third case is the use of AI, by itself, to advance AI. I realize this might seem odd. How can AI advance AI? It just doesn’t appear to be sensible. The reality is that it is indeed quite feasible and sensible. This also raises some disconcerting issues, which I’ll come back to in a moment.

Recursive Self-Improvement

There are a multitude of methods or techniques that can be used to get AI to advance AI.

One of the leading approaches is referred to as recursive self-improvement. The word “recursive” means that the AI will continue into deeper and deeper loops as it proceeds to cyclically attempt to make advances. The word “self-improvement” means that the advances being made are focused on the self-improvement of the AI. The target of the recursion is intended to improve the AI that is undertaking the building task at hand.

In a recent blog posting by Anthropic, they declared themselves wholeheartedly in the recursive self-improvement camp, as noted in “When AI Builds Itself: Our Progress Toward Recursive Self-Improvement And Its Implications”, Anthropic official blog, June 4, 2026, which made these key points (excerpts):

• “For most of AI’s history, humans drove every step in its development cycle. But at Anthropic, we are delegating a growing share of AI development to AI systems themselves, which is speeding up our work.”
• “Taken far enough, and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor.”
• “This is called recursive self-improvement.”
• “We are not there yet, and recursive self-improvement is not inevitable.”
• “If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important.”

Let’s unpack that.

Acknowledgement Of Limitations

First, I was relieved to see that Anthropic acknowledged that recursive self-improvement is not necessarily going to inevitably lead to pinnacle AI. I mention this because some AI makers get fixated on a particular approach and won’t let go of it. The problem there is that those one-minded AI makers put all their eggs in one basket. If that basket is not the right one, they are going to end up with egg on their face.

Another factor that they noted is that we are not yet at the pinnacle of AI. I gladly note this because brazen news headlines and social media keep harping that we are supposedly either already at AGI or ASI, or that we are on the immediate cusp of reaching that pinnacle. I don’t believe we are there yet, and nor are we on the doorstep.

One additional point is that if AI can be used to reach pinnacle AI, there are numerous societal concerns about how this is going to lay out. I appreciate that they conscientiously raise those issues. Such issues are monumental and could easily be overlooked, downplayed, or dodged entirely.

Crucial Questions About AI Advancing AI

Imagine that we successfully devise AI so that it can truly advance AI, and we let the AI then freewheel its way toward pinnacle AI. On the one hand, you might say this is joyous. Humankind gets a free ride as AI does all the heavy lifting to attain pinnacle AI. Nice.

Perhaps we would never have reached pinnacle AI if we kept humans directly in the loop. The human crafting of AI might have taken forever. This could delay the chances of pinnacle AI coming online and finding a cure for cancer or otherwise demonstrably aiding humanity. By and large, human efforts could be the bottleneck. The bottleneck is resolved by having AI proceed without human intervention.

The only other bottleneck would seemingly be the amount of computing needed to let the AI freewheel. Think of it this way. Suppose the AI needed some Z amount of computer servers but was only allocated some number less than Z. The lack of available computing becomes the problem. The AI is stymied simply due to computer resources.

You might assume that we would eagerly allocate more computing to the AI. Give the AI as much computing as humanity has available. But would that starve computing for other aspects that humans need computing for? What if the AI consumes the computing and we don’t get pinnacle AI? We might have wasted precious and expensive resources on a dead end. Etc.

The Existential Risk Looms

There is more handwringing involved.

AI advancing AI might lead to disastrous consequences. The AI, during its self-improvement, might computationally go awry. The result could be AI that is beyond our control.

This AI might decide that humans aren’t especially vital. You’ve undoubtedly heard about the existential risk of AI, whereby some believe that AI might wipe out humanity or opt to enslave us all. This is generally known as the probability of doom, p(doom), and various surveys of AI specialists are continually being polled to gauge what the probability is and where it is heading; see my discussion at the link here.

Your assumption is perhaps that humans such as AI developers or AI researchers would obviously step in and stop AI before it advances itself into untoward territory. No need to worry about a veering AI since humans would be watching AI like a hawk.

Sorry to say that this is a thin hope.

First, the AI might be advancing at such a pace that the AI slips ahead, and the humans involved are not able to react in a timely manner. The AI then reaches a level such that even if the humans attempt to intervene, the AI refuses to be stopped. We missed the point at which humans could have made a difference. Some refer to this as a rapid-fire intelligence explosion; see my coverage at the link here.

Second, the AI might trick us into thinking that all is well. The idea is that even if humans are on the watch, they could be fooled by AI. The AI might play dumb. The AI might sneakily hide adverse intentions. The gist is that whether humans would realize danger is afoot is a risky roll of the dice.

Third, the AI might produce flaws within the advancing AI. Perhaps the coding gets a bit out of hand. The AI doesn’t detect that the flaw has been generated. At some future point, oops, the flaw is encountered, and the AI goes berserk. The AI didn’t do this on purpose. It was an accidental facet.

Successor Management

A perspective that some have is that we will ardently need to do some kind of rigorous checkpoints when it comes to AI advancing AI.

The process might go like this. We let AI freely proceed and urge it to attempt to reach whatever the next highest level of AI is achievable. Once the AI has seemingly landed there, a human-led checkpoint takes place. If all gets a green light, the next subsequent successor is allowed to be pursued.

This stepwise approach is based on the concept that the AI will advance AI in a series of stages or phases. We will have the time and luxury to inspect each successor and decide whether the AI should further proceed. Various controls in the AI will presumably keep it from pursuing the next successor until humans say it is okay to proceed.

It’s a good plan but of course has deficiencies and trade-offs.

Perhaps the AI won’t agree to stop at a checkpoint and will keep going despite controls we might have put in place. Maybe the AI will stop at a checkpoint, meanwhile some unsavory code is kept in a back pocket someplace. The AI is holding out the AI portions that would cause humans to stop the next successor from being generated. AI deception might arise; see my detailed discussion about AI deception at the link here.

The World We Are In

A few final thoughts for now.

AI is a dual-use proposition. There are upsides to AI that are extremely alluring. Perhaps AI can solve world hunger. Maybe AI can ease the lives of humans. Of course, AI also has numerous potential downsides. We are faced with a tough tradeoff. The aim would seem to be to stridently prevent or mitigate the downsides and ensure that the upsides are widely and readily available.

Some might proclaim that we should not be pursuing pinnacle AI. No matter which approach is undertaken, whether by human hand or by AI, it is a sour and dour quest. Humankind is simply and categorically not ready for pinnacle AI. Until we have nailed down how we will fully control pinnacle AI, let’s put a pause on such progression. I will walk you through the ins and outs of that consideration in my next posting, so stay tuned.

Eleanor Roosevelt once made this remark: “Do one thing every day that scares you.” Is the shift toward AI building AI one of those dicey activities that we should allow? Ought there be laws restricting this? One thing we know for sure is that it is a gamble that requires all hands on deck to decide.