LangGraph agent over GitHub, web search and files via MCP
Real MCP tool discovery, schema-validated calls, and a deliberately broken server in the test suite.
- Eval questions
- 13/13
- Median response
- 6.6 s
- Cost vs alternative
- 13× lower
Overview
A LangGraph reasoning loop that reaches GitHub, Tavily web search and a sandboxed filesystem through real MCP tool discovery and invocation, using langchain-mcp-adapters behind FastAPI.
Every tool call is validated against its JSON Schema, with automatic fallback between two OpenAI models and explicit handling for MCP connection failures, tool-execution errors and malformed responses.
Challenges
Exposing all 47 discovered MCP tools made selection worse and the endpoint unsafe. Curating down to 15 (read-only GitHub access, filesystem scoped to a single folder) improved tool choice and made a public endpoint defensible.
Tool-calling reliability does not follow general benchmarks, so NVIDIA Nemotron and OpenAI models were compared head to head on reliability, latency and cost.
Decisions
The calls that shaped this project, the alternatives they beat, and what each one cost.
Curate 15 of the 47 available MCP tools
over expose everything discovery returns
WhyTool selection degrades as the list grows, and a public endpoint with write access to GitHub is a liability. Fifteen read-only tools improved accuracy and bounded the blast radius.
Trade-offSome questions a fuller toolset could answer are now out of reach.
Run a head-to-head model bake-off
over default to the best-benchmarked model
WhyTool-calling reliability does not track general benchmarks. Measuring Nemotron against OpenAI on this task's own questions found a model that was both more reliable here and 13× cheaper.
Trade-offEvaluation time spent before any feature work.
Test against a deliberately broken MCP server
over test the happy path
WhyMCP failures are the normal case in production: servers go down, hang, or return malformed responses. Putting a broken server in the suite makes graceful degradation a tested property rather than an intention.
Trade-offMore test scaffolding to maintain than the code it covers.
Validate every tool call against its JSON Schema
over trust the model's arguments
WhyA model that mostly produces well-formed arguments will eventually produce one that is not, and an unvalidated call reaches a real system. Schema validation turns that into a caught error instead of an action.
Outcomes
The chosen model answered 13 of 13 evaluation questions with a 6.6 second median response, at roughly 13× lower cost than the alternative.
Deployed on Google Cloud Run through Docker and Cloud Build (non-root container, scale-to-zero, around 117 MiB peak memory) behind 25 pytest tests including a deliberately broken MCP server.