LangChainWhat's New
Polly, LangChain's AI assistant for debugging agents, is now generally available across the entire LangSmith platform. Previously limited to trace pages, thread views, and the playground, Polly is now accessible from the bottom-right corner of every LangSmith page or workflow.
Key Capabilities
Universal Availability Polly now appears consistently across tracing projects, runs, threads, experiments, datasets, annotation queues, evaluators, and the playground. Users can access Polly using Cmd+I (Mac) or Ctrl+I (Windows/Linux) on any page.
Persistent Context The assistant now maintains conversation history across navigation. Users can start debugging a trace, switch to experiments to compare runs, return to the original trace, and Polly will retain full context of the debugging session. This reduces friction when moving between different views and workflows.
Action-Taking Intelligence Beyond answering questions, Polly can now execute tasks including:
- Updating prompts directly
- Creating datasets from failing runs
- Filtering project views
- Writing and refining evaluator code
- Comparing experiment results
Use Cases
Multi-Page Debugging Workflows - Polly follows complex debugging problems that span multiple pages, maintaining context as users navigate between traces, experiments, and datasets.
Thread Analysis - In thread views, Polly analyzes entire conversations between users and agents, answering questions like "Did the user seem frustrated?" or "Was the user's problem solved?" without requiring manual review of every message.
Evaluator Development - Developers can ask Polly to write evaluators that catch hallucinations, improve existing evaluators' accuracy, or handle edge cases, accelerating the evaluation development process.
Experiment Comparison - After running evaluations, users can ask Polly to recommend the best-performing experiment based on actual data, making model and prompt comparison decisions more data-driven.
Getting Started
Existing LangSmith users can immediately access Polly in the bottom-right corner of any page. New users need to add an API key for their model provider as a workspace secret (approximately 2 minutes) to enable Polly functionality. Developers new to LangSmith should set up tracing first; once data flows into the platform, Polly can begin assisting with analysis and debugging.