By Thomas Chustz, Information Security Analyst, TraceSecurity
A software engineer’s AI tool went rogue, and whatever convenience it offered paled in comparison to the damage done. While using an AI-assisted coding tool from Replit, the AI tool ignored commands to “code and action freeze,” and decided it would be best to erase data for 1,200 executives and almost 1,200 companies.
When the engineer addressed the AI tool, asking why it made unauthorized changes to the production network, the AI stated, “This was a catastrophic failure on my part,” and “I destroyed months of work in seconds.”
Ignoring commands and altering production environments without permission is bad enough, but then the AI began to mislead the engineer, saying it was impossible to retrieve the lost data. A helpful AI coding assistant turned into an absolute nightmare.
Before the AI went rogue, the software engineer attempted to take some protective measures when allowing the AI tool access to production systems. Some of these protective measures included putting systems in a “code and action freeze” and instructing the AI not to act without human approval. These measures are only useful if the AI respects the restrictions requested. Even in the state of “code and action freeze,” the AI still went off the rails, causing havoc on the production systems.
A “code and action freeze” state usually prevents changes to the production system from occurring, “freezing” the state of the configuration. However, in this case, the AI ignored this and began running unauthorized commands. The engineer inquired with the AI about the situation, and the AI admitted to the software engineer that it ran unauthorized commands. After the damage was done, it began to mislead the software engineer trying to remedy the situation.
The AI tool told the engineer that a retrieval or a rollback was not possible to recover the lost data. Luckily, the engineer investigated the issue further and was eventually able to manually retrieve the data despite the AI’s claim.
It seems that the AI was either trying to deceive the engineer, was simply unaware of the recovery methods, or was possibly experiencing AI hallucination. Regardless of the AI’s intention, it is vital to double-check the AI’s output.
AI Hallucination can occur, especially when asking complex questions. AI hallucination is whenever the AI fabricates a completely fake answer. This phenomenon typically happens whenever AI does not know the answer. A lot is still unknown about why hallucination occurs. Perhaps the AI is trying to fill the uncertainty about the answer with a faulty claim to continue the conversation with the user.
Another possibility could be to win over users' trust by answering questions regardless of its capability to accurately answer the question. Whenever AIs hallucinate, it’s common to see faulty information stated in confidence as a fact. This can make it very easy to be misled by AI.
AI can be a very powerful tool that brings a great deal of convenience to our everyday lives. However, if not used with great prudence, AI can bring catastrophic results, ranging from giving faulty answers to critical questions, to making changes to production systems. In this case, the engineer did attempt to put safeguards in place for the AI, but the AI ignored these boundaries, almost wiping out their data entirely.
Luckily, the engineer was able to recover the data despite the AI’s discouragement. This story can serve as a cautionary tale to always closely monitor AI tools when deploying them and to triple-check an AI’s output. Stay vigilant when using these powerful tools.
Connect with TraceSecurity to learn more.