Polly Goes Broadly Available

LangChain has announced that Polly, its AI-powered debugging assistant for LangSmith, is now generally available across all platform pages. Previously limited to trace pages, thread views, and the playground, Polly can now be accessed from any LangSmith workflow—including tracing projects, runs, threads, experiments, datasets, annotation queues, evaluators, and the playground via a button in the bottom-right corner (accessible with Cmd+I on Mac or Ctrl+I on Windows/Linux).

Key New Capabilities

Context-aware navigation and actions: Polly now maintains conversation context as users navigate between different LangSmith pages. This means you can start debugging a trace, switch to experiments to compare runs, return to the original trace, and Polly will retain full context of your debugging session. Beyond answering questions, Polly can now take direct actions like updating prompts, creating datasets from failing runs, filtering project views, writing evaluator code, and comparing experiments.

Thread analysis and user sentiment: In thread views, Polly can analyze entire conversations between users and agents across multiple back-and-forth interactions. Developers can ask Polly questions like "Did the user seem frustrated?" or "Was the user's problem solved?" to quickly understand user sentiment, conversation outcomes, and interaction patterns without manually reading through every message.

Evaluator assistance: Polly helps teams write and refine evaluator logic directly in the Evaluators pane, generating code to check for hallucinations, improving accuracy, and handling edge cases. This reduces scaffolding work and lets teams focus on what evaluators need to catch.

Experiment analysis: After running evaluations, users can ask Polly which experiment performed best or compare two runs directly. Polly grounds its recommendations in actual data, helping teams decide which prompt changes, models, or architectures meaningfully impact performance without manual result parsing.

Getting Started

LangSmith users can access Polly immediately by looking in the bottom-right corner of any page. To use Polly, users must add an API key for their model provider as a workspace secret (a 2-minute setup). New LangSmith users should first set up tracing to get data flowing into the platform before Polly can assist with debugging and optimization.