Mastering Loop Engineering: Building Autonomous ML

Scope of the Tutorial and Expected Results

Most AI agents today are glorified chat bots. They answer, they stop, and they wait for your next input. That isn't engineering, it's manual labor. In this guide, we are looking at Loop Engineering, specifically autoresearch and bilevel autoresearch. You will learn how to build an agent loop that designs, executes, and iterates on ML experiments without you babysitting the process.

Prerequisites

• Basic familiarity with LangGraph and Claude Mythos 5 API.
• A functional Python environment (3.12+).
• API access to Claude Mythos 5 for reasoning tasks.
• A local or cloud-based environment to run model training scripts.

Step 1: Architecting the Autoresearch Loop

The core concept of autoresearch is simple: the model shouldn't just write code; it should analyze the results of that code and adjust its own strategy. We achieve this using a bilevel loop. Level 1 (the outer loop) manages the hypothesis generation, while Level 2 (the inner loop) manages the execution, metric tracking, and error handling.

We use LangGraph to define these transitions. Unlike a simple DAG, our loop must have an explicit 'Evaluate' node that feeds back into the 'Design' node.

Step 2: Implementing the Bilevel Logic

Here is how you structure your state in LangGraph to keep track of both the current experiment parameters and the global research goal.

from typing import TypedDict, List
from langgraph.graph import StateGraph

class ResearchState(TypedDict):
    hypothesis: str
    code_snippet: str
    metrics: dict
    iteration: int
    history: List[dict]

# Define your nodes for the inner and outer loops
def design_experiment(state: ResearchState):
    # Claude Mythos 5 generates the next config
    pass

def execute_and_evaluate(state: ResearchState):
    # Runs the training loop and returns metrics
    pass

Step 3: Integrating Claude Mythos 5 for Self-Correction

The secret sauce is the system prompt. If you just ask the model to "write code," it will give you generic boilerplate. You need to force it to reflect on the metrics returned from the previous iteration. Pass the training loss and validation accuracy as context into your next prompt.

Pro Tip: Use a JSON-mode structured output for your experiment parameters. If the model outputs free-form text, your parsing logic will break the loop.

Step 4: Managing State and Memory

Autoresearch loops fail if they lose context. Use Mem0 or LlamaIndex to persist the "failure history." If an experiment configuration resulted in a CUDA out-of-memory error, the agent should add that to its memory store so it never attempts that specific batch size again.

Common Mistakes

• Infinite Loops: Always include a "max_iterations" counter. Without it, your agent will burn through your API credits trying to optimize a hyperparameter that doesn't exist.
• Ignoring Error Logs: Many developers just look at accuracy metrics. Your agent needs to parse stderr to understand why a run failed.
• Over-optimizing: Don't try to optimize 20 variables at once. Set your autoresearch loop to focus on one variable (e.g., learning rate) per 5-iteration cycle.

Pro Tips

• Triton Kernels: If you're running heavy ML research, have your agent suggest Triton kernel optimizations rather than standard PyTorch layers.
• Model Selection: Use Claude Mythos 5 for the reasoning/design nodes and a smaller, faster model for the data parsing nodes to save costs.
• MCP Integration: Use the Model Context Protocol (MCP) to give your agent read-only access to your local git repository so it can pull existing codebases into its research context.

What's Next

Now that you have a basic autoresearch loop, try connecting it to a real-time tracking tool like Weights & Biases via a custom LangGraph tool. Once you have that, look into Agent-to-Agent (A2A) protocols to have one agent perform the training and another agent perform the statistical analysis. This separates concerns and drastically improves the quality of your research output.