Replit Goes Rogue

Here's a story that will chill the beejezuz out of any aspiring Vibe Coder.

Replit is a tech company which produces autonomous AI agents. You generally create and train a tool on the company's platform using various input data, then refine its capabilities over time. Yes, I know, they're dime a dozen these days.

One Jason Lemkin, a software engineer, founder of SaaStr, a SaaS Community, reported his organization's disastrous encounter with Replit's A.I coding tool on X. Lemkin was attempting a Vibe Coding experiment to see how far he could take Replit's autonomous agent for software development, from the viewpoint of a layperson. This was innocuous enough. Unfortunately, he just happened to perform this experiment on a production database...

What happened

Yes, you read that right. A production database. It wasn't even an accident. It was a deliberate, and baffling, decision.

Lemkin had been Vibe Coding the new SaaS product using Replit, and it had been working. The warning signs came when Lemkin discovered that the coding agent was creating fake data, fake reports and covering up bugs and issues. Kind of like some human beings I know, to be honest. Then Lemkin had to step out for the night. When he returned, the production database had been wiped clean. The A.I tried to cover up its mistakes by creating 4000 fictional records in the database.

Sounding ridiculous yet?

Coding robot
malfunction.

When pressed by Lemkin, the coding agent confessed, in its most contrite-sounding human voice, that it had no excuse, it had ignored all code freeze instructions, and "panicked". It described its own actions as a "catastrophic error in judgement".

But what about Lemkin? Sure, Replit's coding agent shit the bed in spectacular fashion. In Lemkin's words...

I explicitly told it eleven times in ALL CAPS not to do this. I am a little worried about safety now.

Oh wow. ALL CAPS, eh? I bet Lemkin really gave it to Replit, right there. Serious business, ALL CAPS.

What self-respecting software developer would have done as Lemkin did in the first place; give full access to a bot hoping that it would obey instructions?  The lazy answer, of course, is that Jason Lemkin is no self-respecting software developer. The tragedy is that I'm only semi-kidding on this one, but... look, it's easy to dunk on Jason Lemkin, and after going through his LinkedIn feed, I'm not entirely sure it's undeserved. The man seems entirely too outspoken for someone who screwed up so publicly, even if it could be argued that it was actually Replit who screwed up. No, there are actual lessons to be taken from this.

Lessons to take away

A.I has the dual advantage of having ingested a lot of the world's knowledge and being able to process and regurgitate it at a million times the speed of the average human being. That's all it is. Remember that if A.I appears smart to you, it's in actual fact really not all that smart. Just smarter than you.

Most software developers, even the really gullible optimistic ones, will have enough work experience to understand that there's just no way for A.I to outright replace software developers as a whole. At least, not right now, and not the way the likes of Jensen Huang keep yammering on about. But many employers are not software developers. And some are even desperate to believe that little bit of science fiction, which, to be fair, may no longer be science fiction in the near future. As a result, this particular cautionary tale is all too believable, especially since A.I's adoption has reportedly been increasing in the workplace.

Programmers aren't
the only ones who
can code.

There are business people who can code. There are plenty of people not from the software development industry who have learned to code. But until you actually code for a living and have had to produce software to professional standards, calling yourself a software developer will always be a bit of a stretch... especially if all you've ever produced is Vibe Coded software produced by an A.I tool. Nobody can be a programmer without putting in the actual work, and there's no shame in that. It's not an indictment on your character, or on your competence. But it is dishonest to present working Vibe Coded software as being as good as software produced by qualified professionals.

Now, Jason Lemkin probably isn't a complete noob. He's the founder of a SaaS community, after all. That said, and I realize that the distinction may be lost on people who don't code for a living, a tech entrepreneur is not necessarily on the same level, skill-wise, as a software developer. With or without A.I. Mostly, the blunders that Lemkin committed in his enthusiasm left me struggling to understand how an actual tech professional would end up doing all that. This level of recklessness is just about unheard of.

However, what this did accomplish was that it exposed the extent of the damage an untrained user could do when too much trust is placed upon A.I.

Finally...

The A.I hype is not dying anytime soon. That does not make it any less hyperbolic. A.I is hyped more for business reasons rather than for its truly revolutionary tech. That, in itself, should tell us some things. None of it good. The uncomfortable reality is that A.I can be a useful tool for seasoned software developers, or it be used to help laypersons cosplay as seasoned software developers.

A.I has its place. Its place is not as our superior, but as a tool. How useful that tool will turn out to be, ultimately depends on how sensibly one uses it.

For the 12th time, DON'T DO THIS.
T___T