Dissecting Agent Frameworks: Four Layers, Thirty Lines of Truth

In the last post, I built a tool calling loop in 30 lines of raw Python. A common reaction: "That's all an Agent framework does under the hood?"

Yes — but a framework does more engineering around it.

This post goes one level up: decompose an Agent framework into four layers, mapping each from hand-written code to qwen-agent. By the end, every concept you see in framework documentation will have a concrete counterpart in code you can write yourself.

📦 Links

• Project repo: Building Agent from Scratch
• Lesson code: 02-explore-agentic-frameworks/code_samples/
• qwen-agent: QwenLM/Qwen-Agent

🎯 The Question: What Does a Framework Actually Do?

The core loop from last time:

while True:
    response = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS_SCHEMA)
    msg = response.choices[0].message
    if not msg.tool_calls:
        return msg.content
    for tc in msg.tool_calls:
        result = TOOL_MAP[tc.function.name](**args)
        messages.append({"role": "tool", "content": result})

This works — but everything is tangled together: model connection, tool definitions, message management, loop control.

Every real Agent framework separates these concerns into distinct layers. Different frameworks use different names and APIs, but the separation logic is identical.

🧱 The Four Layers: Client → Agent → Tools → Session

After reading the source code of Microsoft Agent Framework, LangChain, and qwen-agent, I found four layers common to all of them:

  Client ──→ Agent ──→ Tools
  Connection  Packaging  Capability
                │
                └──→ Session
                     Memory

Let's break each one down.

Layer 1: Client — Connecting to the Model

Responsibility: establish a connection to the AI model, handling authentication, request formatting, and response parsing.

Raw SDK:

from openai import OpenAI
 
client = OpenAI(
    api_key=os.getenv("DASHSCOPE_API_KEY"),
    base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
)

qwen-agent:

llm_cfg = {
    "model": "qwen-flash",
    "model_type": "qwen_dashscope",
    "api_key": api_key,
}

Both do exactly the same thing: specify model + credentials + endpoint. The framework version adds model_type to tell the framework which protocol to use — that's it.

Key insight: the Client layer contains zero Agent logic. It's an authenticated HTTP client. Any model with an OpenAI-compatible endpoint (Qwen, DeepSeek, Moonshot, etc.) can be swapped in by changing only base_url and api_key.

Layer 2: Tools — The Agent's Hands

Responsibility: define functions the Agent can call, telling the LLM "what I can do."

Every tool needs two things:

1. A Python function — the actual logic
2. A JSON Schema — tells the LLM the name, description, and parameter types

Raw — write both separately and manually connect them:

def check_destination_availability(destination: str) -> str:
    """Check if a destination is available for booking."""
    available = {"Barcelona": True, "Tokyo": True, "Paris": False}
    return f"{destination} {'available' if available.get(destination, False) else 'unavailable'}"
 
TOOLS_SCHEMA = [{
    "type": "function",
    "function": {
        "name": "check_destination_availability",
        "description": "Check if a vacation destination is available for booking.",
        "parameters": {
            "type": "object",
            "properties": {
                "destination": {"type": "string", "description": "The destination to check"},
            },
            "required": ["destination"],
        },
    },
}]
 
TOOL_MAP = {"check_destination_availability": check_destination_availability}

qwen-agent — decoration merges all three:

@register_tool("check_destination_availability")
class CheckDestinationAvailability(BaseTool):
    description = "Check if a vacation destination is available for booking."
    parameters = [
        {"name": "destination", "type": "string", "description": "The destination to check", "required": True},
    ]
 
    def call(self, params: str, **kwargs) -> str:
        args = json.loads(params) if isinstance(params, str) else params
        destination = args.get("destination", "")
        available = {"Barcelona": True, "Tokyo": True, "Paris": False}
        return f"{destination} {'available' if available.get(destination, False) else 'unavailable'}"

What the framework saves you: auto-generating JSON Schema from the parameters list, eliminating both the hand-written schema and the manual name-to-function mapping. But what gets sent to the LLM is still the same JSON Schema.

Key insight: the Tools layer defines the Agent's "capability boundary." What you can do depends on what tools you register. In real projects, tool definitions are often the bulk of the code — not because the framework is complex, but because your business logic is.

Layer 3: Agent — Packaging Everything Together

Responsibility: bundle Client + Instructions + Tools into a single callable object.

Raw — you manually maintain messages + system_prompt + run_agent():

SYSTEM_PROMPT = "You are a travel booking agent. Always check availability before recommending."
 
def run_agent(user_message: str, messages: list[dict]) -> str:
    messages.append({"role": "user", "content": user_message})
    # ... tool calling loop ...
    return result

qwen-agent provides the Assistant class:

agent = Assistant(
    llm=llm_cfg,                          # Client layer
    name="TravelAvailabilityAgent",
    system_message=SYSTEM_PROMPT,          # Instructions
    function_list=["check_destination_availability"],  # Tools layer
)

Assistant is the "package box" for Client + Instructions + Tools. You no longer manually manage messages or the while True loop — both are encapsulated inside agent.run().

Key insight: the Agent layer is the framework's "user experience." Assistant, create_agent(), AgentExecutor — different frameworks, different class names, but the constructor always takes the same three things: model config, system instructions, and a tool list.

Layer 4: Session — Multi-Turn Memory

Responsibility: maintain context across multiple conversation turns.

This is the most misunderstood layer. Many people think a framework's session object is some kind of magic. Here's the truth:

A Session is just a messages list that persists across calls.

Raw — no magic:

session = [{"role": "system", "content": SYSTEM_PROMPT}]
 
# Turn 1
reply = run_agent("I want to go to Paris. Is it available?", session)
print(reply)
 
# Turn 2 — session already contains turn 1's full context
reply = run_agent("I want somewhere warm. What's available?", session)
print(reply)
 
print(f"[Session has {len(session)} messages]")

qwen-agent — no explicit session object, same approach:

session = []  # just a list
 
# Turn 1
reply = run_turn("I want to go to Paris. Is it available?", session)
 
# Turn 2 — session was updated by run_turn
reply = run_turn("I want somewhere warm. What's available?", session)

How different frameworks wrap session:

Underneath, they all do the same thing: append messages to a list. [system, user, assistant, tool, assistant, user, assistant, ...].

Key insight: short-term memory = message list. Long-term memory = external storage (database / vector store) + tools. The framework doesn't "provide" memory — it provides convenience functions for managing the message list.

🔄 Full Mapping

💡 Three Key Takeaways

1. Frameworks don't provide "Agent capability" — they organize your code

Tool calling capability comes from the LLM API itself (every major model supports it). The framework saves you from hand-writing schemas, mappings, loops, and message management. It "does for you," not "provides for you."

2. The four layers are universal

Whether you use qwen-agent today, LangChain tomorrow, or AutoGen next week — these four layers are always there. Only class names and parameter names change, not structure. The messages.append() you learned in raw code is the same thing inside any framework's session object.

3. Understanding layer boundaries tells you where to look when things break

• LLM not calling tools? → Check the Tools layer — is the Schema correct?
• Agent forgetting context? → Check the Session layer — are messages being appended?
• Connection errors? → Check the Client layer — api_key and base_url
• Agent behavior off? → Check the Agent layer — system_message

Each problem maps to one layer. No more randomly searching through code.

🚀 Try It

cd 02-explore-agentic-frameworks/code_samples
 
# Raw first — understand the four layers
python 02-python-agent-framework.py
 
# Then framework — appreciate the packaging
python 02-qwen-agent-framework.py

🔮 Next Up

With these four layers as foundation, the next post dives into Agentic Design Patterns: how to write effective agent instructions, how to get structured JSON output, and how to split tasks across multiple agents.

All code and posts at Building Agent from Scratch.

✍️ Closing

The right order for learning frameworks is not "pick a framework, then write code." It's "write code first, then read the framework docs." Once you've hand-written the four layers, every framework's documentation stops looking like mysterious APIs and starts looking like "oh, this layer is my TOOLS_SCHEMA, that layer is my messages list."

A framework is engineered packaging of code you can already write. Once you understand that, you're free.