What Is an AI Agent?

Software once executed predefined instructions. Now it reasons, plans, and acts on its own. Here is how that actually works under the hood.

This document is a technical primer for engineers building AI agent systems. It covers why AI agents matter, what an AI agent is, multi-agent architectures, the protocol stack that connects applications, agents, and tools, and the failure modes that matter in production.

Figure 1. High-level architecture of an AI agent system

As a roadmap, the whole document builds toward a three-layer model of the agent stack: the application talks to the agent (over REST, WebSocket, or gRPC), agents talk to each other (A2A), and agents talk to tools (MCP). The following sections develop each layer in turn.

Why AI Agents?

Traditional software follows predefined workflows. LLM applications generate responses but typically do not take actions. AI agents bridge this gap by combining reasoning, planning, memory, and tool use to autonomously accomplish goals that require multiple steps and adaptation.

Traditional Software	LLM Application	AI Agent
Executes predefined logic	Generates responses	Reasons and takes actions
Deterministic	Probabilistic text generation	Goal-driven planning
No tool orchestration	Optional tool use	Dynamic tool orchestration
No autonomy	Limited autonomy	High autonomy

What Is an AI Agent?

An AI agent is a software program that uses an AI model to reason about goals, plan actions, use external tools, observe results, and iteratively work toward completing a task. Unlike traditional software, an agent can adapt its behavior based on context rather than following only predefined execution paths.

• Give it a goal

• Figure out the steps needed using an AI model (typically a large language model, although the same architecture can incorporate multimodal models)

• Use tools such as web search, code execution, and APIs to complete each step

• Continue to run until the goal is completed and adjust as necessary

Multi-agent Systems

The industry is increasingly favoring smaller, specialized agents for the same reasons microservices replaced monoliths: reduced prompt complexity, improved reliability, independent scaling, easier testing, and clear separation of responsibilities. Three common multi-agent coordination patterns are: orchestrator-worker, peer-to-peer, and pipeline.

• In the orchestrator-worker pattern, a top-level agent breaks down the goal and delegates to specialist agents.

• In a peer-to-peer pattern, agents communicate as equals — negotiating, challenging each other, or collaborating — more like a team than a hierarchy. Because there is no central coordinator, this pattern suits open-ended problems where the answer emerges from debate; the tradeoff is that it requires explicit conventions for reaching consensus and for terminating, since agents can otherwise negotiate indefinitely.

• In a pipeline pattern, each agent completes its task and passes output to the next — simple, predictable, and easy to debug.

Agent Mesh

Agent meshes are an emerging architectural pattern rather than a formal architecture. In an agent mesh, agents dynamically discover and collaborate with one another without relying on a fixed hierarchy. The mesh self-organizes around the task, allowing any agent to discover and call any other.

AI Agent vs. Microservice

Although AI agents are often implemented as microservices, they are not the same thing. They share many characteristics with traditional software services, but differ fundamentally in how they make decisions.

Similarities:

• Defined input/output contracts

• Composable into larger workflows

• Independently deployable, scalable, and versioned

• Can expose REST, gRPC, or other APIs

• Can be containerized and orchestrated with Kubernetes

Key Differences:

Microservice	AI Agent
Deterministic business logic	Probabilistic reasoning using an AI model
Business logic lives in application code	Business logic is partly delegated to the LLM’s reasoning
Explicit workflow defined in code	Dynamically plans the next action
Executes predefined business rules	Chooses tools and actions based on context
Same input generally produces the same output	Same input may produce different but valid outputs
Easy to unit test	Requires evaluation-based testing
Usually, stateless request processing	Maintains conversational, working, or long-term memory
Calls predefined APIs	Dynamically selects and invokes tools
Failures are handled by programmed logic	Can recover by replanning or trying another strategy

Autonomy Under Uncertainty

Unlike a traditional microservice, an AI agent can operate when the exact execution path is unknown.

An agent can:

• Handle inputs it was never explicitly programmed for.

• Decide which tools to use based on context.

• Recover from unexpected failures by trying another approach.

• Ask clarifying questions when information is ambiguous.

• Revise its plan as new information becomes available.

Agent Memory

Because an agent's context window is finite and does not persist between runs, production agents divide memory by how long it must survive and where it lives:

• Short-term memory — the recent conversation and immediate context (volatile, held in the context window).

• Working memory — the current task's scratchpad: the active goal, plan, and intermediate results (volatile, in-process).

• Long-term memory — knowledge that must outlive a single session: semantic retrieval, a knowledge base, entity/relationship tracking, and continuity for long-running tasks (persisted in vector databases and knowledge stores).

• External systems of record — durable stores the agent reads and writes through tools: files, databases, CRMs, calendars (reached over the tool/service layer, e.g. MCP).

Agent Planning

Planning distinguishes an AI agent from a simple LLM call. A typical reasoning loop is:

Where Do Agents Run?

Because an AI agent is ultimately just software, it can be deployed anywhere modern applications run (containers, VMs, serverless functions, edge devices, or local machines). The underlying LLM may be hosted by OpenAI, Anthropic, Azure OpenAI, or run locally through frameworks such as Ollama or llama.cpp.

How Are Agents Implemented?

An AI agent is fundamentally software that combines application code, one or more AI models, and external tools. There are many ways to package and run the agent. Most production agents run in containers because:

• Reproducibility: Containers provide reproducible execution environments, ensuring that every deployment behaves consistently.

• Scalability: Containers allow agents to scale horizontally by adding or removing instances based on demand.

• Isolation: Containers isolate execution environments so one agent cannot interfere with another.

• Orchestration: Containers let multiple agents run in a system where each agent type is its own deployment and can scale independently.

How Do Agents Communicate?

Like traditional distributed systems, AI agents can communicate using several mechanisms such as REST APIs, direct function calls, shared message queues, and more. An emerging standardized protocol for agent-tool communication is MCP, originally introduced by Anthropic. MCP (Model Context Protocol) standardizes communication between an MCP client (typically an agent) and an MCP server. The server exposes Tools, Resources, and Prompts through a common protocol — so you do not have to build a custom integration for every single thing an agent might need to access.

MCP Server for Tools and Services

An MCP server provides a standardized interface that allows agents to discover and interact with external tools, data sources, and prompt templates. An MCP server exposes three core primitives:

PRIMITIVE	What it is	Example
Tools	Actions the agent can call	send_email(), query_db()
Resources	Data the agent can read	a file, a database row, calendar
Prompts	Pre-built prompt template	A reusable summarization prompt

How It Works Step by Step

1. The MCP server starts and advertises what tools/resources it offers

2. The agent connects and discovers available capabilities

3. The agent decides (via LLM reasoning) which tool to call

4. The agent sends a structured JSON-RPC request to the server

5. The server executes the action and returns a result

6. The agent incorporates the result and continues reasoning

Transport Options

MCP supports two communication transports (refer to this):

• stdio — local process communication

• Streamable HTTP — single HTTP endpoint (POST/GET) for remote servers; can optionally upgrade to an SSE stream for streaming. Replaces the deprecated HTTP + SSE transport.

Agent to Tools Authentication

An agent requires credentials to access tools and services. Agents run as remote processes and must access these credentials without hardcoding them. Credentials can be managed in several ways:

1. Environment Variables (service-to-service access)

When the agent accesses services such as an LLM, GitHub, or a database, where credentials do not change per user — environment variables are the standard. Credentials are injected into the agent’s runtime environment at startup. The agent never sees them in code – it just reads from the environment. In production, these are set via:

• Kubernetes secrets mounted as env vars

• Docker --env-file

• Cloud provider secret injection (AWS Secrets Manager, GCP Secret Manager)

2. OAuth 2.0 (for user-delegated access)

The standard authentication protocol for accessing services is OAuth 2.0. OAuth 2.0 is not AI-specific; agents simply reuse the same authorization protocol that traditional applications have used for years.

The OAuth 2.0 flow consists of the following steps:

1. The client requests authorization from the resource owner directly or indirectly through the authorization server.

2. The resource owner grants authorization to the client. Several grant types are available, depending on the method used in the client’s request and the types supported by the server.

3. The client requests the access token by authenticating with the authorization server using the authorization grant.

4. The authorization server authenticates and validates the client’s authorization grant and issues an access token

5. The client requests the protected resource from the resource server using the access token

6. The resource server validates the access token, and if valid, returns the protected resource

Application to Agents Communication

Unlike agent-to-tool communication, where MCP is a standardized, purpose-built protocol, there is currently no universal standard for application-to-agent communication. Today, applications communicate with agents using whatever means necessary such as REST, gRPC, WebSocket, or native SDKs.

1. Direct Communication: The application calls the LLM API directly, with the agent logic embedded within the application. There is no separate agent service—the application and agent run in the same process.

2. REST API: When the agent runs as a separate microservice, applications communicate with it over HTTP. REST uses the same protocols and patterns as any other microservice; there is nothing agent-specific about the communication layer.

3. WebSocket: When real-time streaming of events, progress updates, or agent responses is required, WebSocket is used to maintain a persistent connection between the application and the agent.

4. gRPC: When high performance is required, gRPC provides strongly typed interfaces, bi-directional streaming, and efficient binary serialization.

5. Message Queue: For asynchronous workloads that do not require an immediate response, a message queue decouples producers and consumers, enabling reliable delivery, retries, and scalable background processing.

   In addition to REST, gRPC, and WebSocket, many frameworks expose SDKs that abstract away the underlying transport, allowing applications to invoke agents through native programming interfaces.

   Some emerging standards that are attempting to fill the gaps are:

1. Agent Protocol by AI Engineer Foundation: An open standard specifically for application-to-agent communication. It defines a clean, vendor-neutral REST interface, but is still early and not yet widely adopted.

2. Realtime API by OpenAI: WebSocket-based protocol for real-time agent interaction such as streaming audio, text, and tool calls bidirectionally.

Agent to Agent Communication

MCP is designed for agent-to-tool/service communication — short request-and-response interactions with a passive server — not for agent-to-agent communication. Agent-to-agent communication requires capability exchange and support for asynchronous, parallel, and long-running interactions. A2A (Agent-to-Agent Protocol), created by Google and now governed by the Linux Foundation, is the emerging standard to address these capabilities. Unlike MCP, both endpoints in A2A are autonomous agents capable of reasoning.

A2A introduces concepts that MCP does not support:

• Agent Cards – agents advertise their identity and capabilities

• Task lifecycle – A2A tracks long-running collaborative tasks

• Multi-turn negotiation – agents can push back, ask questions, and refine

• Streaming partial results – A2A agents can stream progress as they work on the results

End-to-End AI Stack

The full end-to-end AI stack consists of the application, AI agents, and tools/services. Each layer has a different communication pattern that warrants a different protocol. The layer of application to agent is the only layer that reuses existing communication protocols that were designed for application-to-cloud-service communication such as REST, WebSocket, or gRPC. The other layers that are new have emerging purpose-built protocols such as A2A for agent-to-agent communication and MCP for agent-to-tool/service communication.

Full Agent Protocol Stack
Application to Agent	REST / WebSocket / gRPC (existing cloud-service protocols)
Agent to Agent	A2A Protocol (peer collaboration, negotiation, long-running async)
Agent to Tool	MCP Protocol (tool/service discovery and invocation)

An example of an AI stack is shown below. In such an AI stack, a user can ask the AI agent to do the following tasks:

1. Every morning, look for emails asking about meetings. If they do not already have a calendar invite, find an available 30-minute slot this week and draft a reply proposing three available times.

2. When a flight confirmation email arrives, automatically add it to my calendar with departure time and airport.

3. Look for utility or credit card bills due within 7 days and add reminders to my calendar.

4. When OpenTable sends a reservation confirmation, create a calendar event and remind me two hours before.

5. If I receive an email saying one of my flights is delayed, text my wife.

6. Thirty minutes before every meeting, text me the meeting title and location.

7. Every morning at 7 AM, text me today’s calendar schedule.

Potential Issues with AI Agents

Some of the hard problems in an AI system are trust, hallucination, context fragmentation, accountability, and auditability.

1. Trust between agents and LLMs:

Should agent A blindly trust what agent B tells it? Each agent needs to validate inputs independently and treat every message as a potential threat.

2. Infinite loops between agents:

Agent A asks for a response from agent B, then asks for clarification and gets the same response. The agent needs to implement loop detection, maximum revision cycles, and a human escalation threshold.

3. Context fragmentation:

Each agent has a limited context window, and information is lost along the way. Long-running pipelines lose information: the longer the pipeline, the more context is lost. The agent needs to implement a vector store, structured state passing, and explicit context preservation.

4. Accountability and auditability:

If the final output is wrong, which agent caused it? The agent needs to implement logging and an immutable audit trail per agent step and boundary.

5. Hallucination:

AI agents may generate plausible but incorrect information or make incorrect assumptions when reasoning or invoking tools. Production systems should validate critical outputs, verify retrieved information, and incorporate human approval for high-impact decisions.

Additional Production Challenges

Besides hallucinations and context fragmentation, production systems should also address:

• Non-deterministic behavior

• Tool misuse and parameter validation

• Permission boundaries (least privilege)

• Cost management for recursive reasoning

• Human approval for high-impact actions

When Should You Use an AI Agent?

AI agents represent a shift from software that executes predefined instructions to software that reasons, plans, and acts. As the ecosystem matures, protocols such as MCP and A2A are becoming the foundational standards for interoperable agent ecosystems, much as HTTP standardized communication for the web.

Use AI agents when solving open-ended problems requiring reasoning, planning, or dynamic tool selection. Traditional software remains the better choice for deterministic workflows, CRUD operations, fixed ETL pipelines, and straightforward business logic.

Key Takeaways

• AI agents extend LLMs with reasoning, planning, memory, and tool use.

• Multi-agent systems improve scalability through specialization.

• MCP standardizes agent-to-tool communication.

• A2A standardizes agent-to-agent collaboration.

• Production systems require careful attention to security, observability, and reliability.

Further Reading

• Anthropic MCP Specification

• Google A2A Specification

• OpenAI Responses API

• OpenAI Realtime API

• LangGraph

• AutoGen