Python Client

TypeScript Client

Cookbooks

Literal AI Documentation

Get Started

Evaluate Agent Runs with Tools

Literal AI is the collaborative platform for building **production-grade LLM apps**.

Overview

Install the Literal AI client and get your API key

Quick Start

Monitor your AI application usage, and track performance and reliability.

Dashboard

Logs are essential to monitor and improve your LLM app in production. Literal AI provides flexible and composable SDKs to log your LLM app at different levels of granularity.

Logs

Prompt Management enables collaboration between engineering and product teams to create, version, A/B test, debug, and monitor prompts directly from Literal AI.

Prompts

The playground is a place where you can create, test, and debug prompts.

Playground

Understand your end users, and how they interact with your application.

Users

Scorers are functions (code or LLM-as-a-judge) that take an input, output and expected output and return a score.

Scorers

Automatically evaluate your LLM logs in production, monitor performance and detect issues.

Online Evals

Datasets are collections of input/expected output samples for conducting experiments and non regression tests.

Datasets

Experiments enable continous improvement of your Prompt/Agent -- i.e. guarantee net improvements.

Experiments

Annotation Queues help you organise and streamline the human evaluation process. You can easily set-up queues, add data for review and provide an accessible and efficient ui for domain experts to score, tag and add to datasets crucial insights.

Annotation Queues

General

Scoring

Team

Learn how to evaluate your LLM applications and agents.

Evaluation

Improve your LLM applications over time with continuous improvement.

Continuous Improvement

Fine-tune your LLM models to improve performance and reduce costs.

Get Started

Application

Evaluation

Settings

Guides

Integrations

Self Hosting

More

Evaluate Agent Runs with Tools