ChainBuddy - Two-Phase Agents and Model Tiering

ChainBuddy (Zhang and Arawjo, CHI 2025) is an assistant that generates starter pipelines for ChainForge, an open-source visual toolkit for prompt engineering and LLM evaluation. ChainForge lets you test, compare and analyse model behaviour across many prompts and models without much code. ChainBuddy tackles the blank-page problem: you’ve been handed a tool you need to use and have no idea where to start.

Its backend is built with LangGraph (stateful multi-agent graphs with cycles and persistence), and its architecture has a few patterns I keep reusing.

Two phases

1. Requirement-gathering chat. Targeted questions turn a vague request into a detailed, structured specification. Vague requests otherwise produce irrelevant pipelines.
2. Pipeline generation. A planner, task-specific worker agents, then assembly into a JSON flow exported to ChainForge.

A fast conversational front, a methodical back end. Decouple the user from the heavy processing.

Heterogeneous model tiering

The paper used a faster, cheaper model for the chat (Claude 3.5 Sonnet) and a stronger one for the heavy generation (GPT-4o). Get intent clear early so you don’t burn the expensive model on a misunderstanding.

Chain of Thought vs Chain of Reasoning

• Chain of Thought (CoT): prompt the model to think step by step. Improves a single complex answer.
• Chain of Reasoning (CoR): the agent keeps a whiteboard of internal state (the primary goal, a running list of requirements, preferences discovered so far) to steer a multi-step task. It acts as short-term memory and as a check before acting.

Rule of thumb: CoR for projects, CoT for single tasks.

Smart intake forms

Ask three targeted questions at once: goal clarification, requirements exploration, clarifying details. The user stays in control and can generate at any point. And no defaults: making people engage is a safeguard against over-confidence (see Evaluating AI-Assisted Work - Cognitive Load and Over-Reliance).

The generation pipeline

• Planner (plan-and-solve): builds nothing; writes a complete blueprint the other agents follow.
• The planned workflow has three layers: input (personas, questions) → middle (prompt templates × inputs → LLM) → evaluation (code evaluators or LLM-scorer nodes).
• Variable tracing: every pipe on the blueprint is labelled: each {variable}, the node that creates it and the nodes that consume it. Still human-readable at this stage, so a person can review it before anything is built.
• Worker agents each execute one well-defined slice: input nodes, prompt nodes, and an evaluator agent that picks a code evaluator or an LLM scorer (the same idea as grading marketing copy for sentiment, brand and compliance).
• Connection agents finish the job: an edge agent wires nodes together, a layout agent tidies the canvas, and out comes the final JSON workflow.

Reusable patterns

• Decouple the user from heavy processing.
• Use structured dialogue (CoR) for goal-oriented tasks, and as a safeguard.
• Hand off a work ticket, not a transcript.
• Produce an editable starter, leaving the user as the final judge of quality.

These slot straight into the decompose-and-specialise and latency advice in Agentic System Design Checklist.