Building agents isn’t just about speed. Trust is the new premium and builders need airtight guardrails to turn lawless AI into trustworthy tools.
Sydney Scott
Editorial Strategist, AI
Workday
A bad AI chatbot answer creates a review problem. A bad artificial intelligence agent with production access can trigger an operational crisis.
That risk now sits at the centre of software development. As AI moves from suggesting code to planning work and taking action, developers must decide what stops a system when it gets things wrong.
AI use has raced far ahead of trust. Google Cloud’s September 2025 DevOps Research and Assessment (DORA) study of nearly 5,000 technology professionals found that 90% used AI at work and more than 80% said it improved productivity. Yet only 24% trusted the technology “a lot” or “a great deal”. Another 30% trusted it little or not at all.
The concern is valid. Earlier this year, a Cursor coding agent working on a routine task for software company PocketOS hit a credential mismatch, found a broadly scoped access token, and deleted the company’s production database and its volume-level backups through a single cloud call.
The wipe took nine seconds. The failure exposed a danger developers had not contained: an agent working in a test environment could still reach live systems and take an irreversible action without human approval.
The trust gap now carries real stakes. It creates an engineering problem, a business risk, and a fight over who controls increasingly capable software. Companies want AI’s speed. Developers must keep that speed from turning into damage.
As AI moves from suggesting code to planning work and taking action, developers must decide what stops a system when it gets things wrong.
Report
Risk controls can no longer wait for a final compliance review. By then, teams have already made the choices that matter: what an agent can see, which tools it can call, whether it can write or delete, and when a person must step in.
Agents raise the stakes. A coding assistant can suggest a bad fix. An agent with company data and production access can act on one.
Trust is being built directly into the product itself. Across the ecosystem, tech giants are embedding guardrails: AWS uses Bedrock AgentCore to enforce strict application boundaries, while Microsoft’s Agent Control Specification intercepts risk at every turn – from initial input to final execution.
Shared standards could keep controls from getting trapped inside one vendor’s system. The Linux Foundation’s Agentic AI Foundation houses projects such as Model Context Protocol (MCP), the Goose agent framework and AGENTS.md, with a focus on common agent systems and safety patterns.
The market signal is hard to miss. Permission limits, approval gates, policy checks and audit records are becoming part of the machinery. Vendors that make those controls clear and portable may gain an edge with companies deciding how much authority to hand over.
But tools alone will not fix a weak company. DevOps Research and Assessment (DORA) describes AI as an amplifier: clear workflows and strong teams get stronger, while vague ownership and poor review habits spread faster.
That makes trust both a technical control and a workplace test. Developers need room to challenge uncertain output and slow down a release without taking a hit for it. Forced adoption can produce silent non-compliance, with workers using AI to tick a box and avoiding it for serious work.
If leadership solely measures success by code volume or completed tickets, they are accidentally rewarding raw velocity over systemic safety.
Building with AI isn’t just about pumping out code at lightning speed. It requires building safe, dependable systems. Today, a developer’s real job isn’t just typing out syntax – it’s acting as a system architect. They must frame big problems, set hard operational boundaries and stop single automated changes from breaking the whole business.
Before releasing any new AI system, smart teams use a simple checklist to set their ground rules. Developers must have a clear answer to four key questions:
These questions shift the focus from how impressive an AI model seems to how reliable the entire system actually behaves.
To turn these boundaries into daily practice, enterprise teams rely on the National Institute of Standards and Technology (NIST) AI Risk Management Framework. This framework breaks safety into four continuous functions:
Name specific leaders for security, privacy and legal risk before any building starts. Clearly identify who has the authority to disable the system or approve new data access.
Map out users, data, tools and potential ways people might misuse the system. Apply light checks to simple tasks (like summarising a document) and heavy scrutiny to risky actions (like moving money).
AI can generate code that looks flawless on the surface while hiding messy logic, security gaps or fragile dependencies underneath. This creates a massive backlog of code reviews. If leadership solely measures success by code volume or completed tickets, they are accidentally rewarding raw velocity over systemic safety.
Instead of tracking raw output, test the actual path the AI takes and focus on smart velocity metrics:
Because AI systems change constantly, teams must monitor for unusual behaviour in real time. When something inevitably breaks, skip the finger-pointing. Ask why the system had the room to cause damage, and turn that real-world incident into an automated test so it never happens again.
Baking controls into the product from day one isn’t a drag on velocity – it’s the green light to move faster.
If an AI system is a black box, humans are forced to micromanage it, defeating the purpose of automation. But when you wrap an agent in tight permissions and clear visibility, the dynamic flips. The machine can safely handle complex work because humans know exactly where the boundaries are.
This trust is the ultimate business accelerant, moving organisations past the fear of a catastrophic system wipe and towards confident scaling. Market power is already shifting away from pure model intelligence and towards platforms that offer absolute transparency and a reliable kill switch.
The next competitive edge won’t come from the flashiest AI agent. It will come from the infrastructure that keeps it in check: airtight guardrails, portable safety standards, and architectures built to contain the blast radius of an error.
Baking these controls into the product from day one isn’t a drag on velocity – it’s the green light to move faster. The companies that build trust early will outpace the competition because when their systems inevitably trip up, the damage is already contained, visible and easily fixed.
Report
Defining roles and setting clear boundaries drive AI adoption and optimism. Learn how to unleash AI's potential with a human touch.