General Availability

Polly is now generally available to all LangSmith users as a persistent AI assistant accessible from any page in the platform. Previously limited to specific sections like trace pages and the playground, Polly is now available in a unified bottom-right corner interface accessible via keyboard shortcut (Cmd+I on Mac, Ctrl+I on Windows/Linux).

New Capabilities

Expanded Surface Area: Polly now works across all LangSmith pages including projects, runs, threads, experiments, datasets, annotation queues, evaluators, and the playground.

Context Persistence: The AI assistant maintains conversation history across navigation. Users can start debugging a trace, switch to experiments, and return while Polly retains full context of previous work.

Hands-On Actions: Beyond answering questions, Polly can now take direct actions including:

• Updating prompts
• Creating datasets from failing runs
• Filtering project views
• Writing and refining evaluator code
• Comparing experimental results

Use Cases

Multi-Step Debugging: Polly follows developers through complex debugging workflows that span multiple pages—from analyzing traces to comparing experiments to updating prompts, maintaining context throughout.

Thread Analysis: Users can analyze entire conversation threads (multi-turn agent interactions) by asking questions like "Did the user seem frustrated?" or "Was the user's problem solved?" rather than manually reviewing every message.

Evaluator Development: Polly assists in writing and iterating on evaluator logic directly in the Evaluators pane, reducing scaffolding time.

Experiment Decisions: After running evaluations, developers can ask Polly for recommendations on which experiment performed best based on actual data.

Getting Started

Users need to add an API key for their model provider as a workspace secret (a 2-minute setup). The feature is available immediately in the bottom-right corner of any LangSmith page.