Migrating a RAG Copilot Serving 5,000 Users

The situation

Roche Diagnostics had a RAG-powered copilot built on LangChain, used internally by thousands of employees for troubleshooting and knowledge retrieval. The system worked, but the codebase was becoming increasingly painful to maintain. LangChain's abstraction layer made it hard to debug issues, customize behavior, or reason about the execution flow.

On top of that, the LLM observability stack was tightly coupled to Literal AI through their Chainlit frontend integration. If Literal AI changed their pricing, deprecated an API, or shut down — the entire monitoring and tracing pipeline would break, and the team would be scrambling for an emergency replacement.

What I proposed

I identified two separate but related problems and pitched solutions for both.

For the codebase maintainability issue, I proposed migrating from LangChain to LangGraph. LangGraph gives you explicit control over the agent's execution graph — you can see exactly what happens at each step, add branching logic, handle errors granularly, and test individual nodes. This wasn't an assigned task. I saw the opportunity, built a proof of concept on my own time, and presented it to the team and stakeholders.

For the vendor lock-in risk, I proposed building an LLM observability abstraction layer — essentially a thin adapter that sits between our application and whatever observability provider we use. I evaluated several options and recommended Langfuse as the replacement: it's open-source, self-hostable, and has a clean integration path. The team accepted the proposal, and another developer handled the migration using the adapter pattern I designed.

The outcome

The LangGraph migration improved code maintainability by roughly 60% (measured by reduced time-to-change for new features and bug fixes). The copilot continued serving 5,000+ users without disruption during the migration.

Test coverage went from 40% to 80% — I introduced TDD practices and CI/CD quality gates that the team adopted going forward.

I also delivered 6 voluntary AI workshops for the team, covering agentic coding tools like Cursor and Claude Code with MCP integrations. This wasn't part of my scope — I just saw the team could ship faster with better tooling and wanted to share what I'd learned.

Why this matters

A lot of companies built their first AI systems on LangChain in 2023-2024 and are now hitting the same walls: hard to debug, hard to customize, too many abstractions hiding what's actually happening. The migration path to LangGraph (or similar explicit orchestration) is well-worn at this point, but it requires someone who understands both the old and new patterns deeply enough to do it without breaking a production system that real users depend on.