​

From the Literal AI Platform

Automating the evaluation of your Run outputs or LLM generations can significantly help detect patterns and areas of improvement for your LLM app in production, especially with large volumes of data.

An Online Eval is composed of:

• Name: A name to identify the rule.
• Log Type: Either Agent Run or LLM Generation, it’s the target to evaluate.
• Sample Rate: The percentage of logs to evaluate.
• Filters: Additional conditions to selectively evaluate certain logs.
• Scorer: The scorer to use for the evaluation.

To create an Online Eval, go to the Online Evals page and click on the + button in the upper right corner of the table.

Once the Online Eval in place, your Runs or LLM Generations get automatically evaluated.

You can check the distribution of scores on an Online Eval’s page:

​

From the SDKs

The SDKs provide Score creation APIs with all fields exposed.

If your metrics are code-based or combine LLM calls with arithmetic operations, like Ragas, you can directly use the SDKs to create scores from your application code.

import os
from literalai import LiteralClient

literalai_client = LiteralClient(api_key=os.getenv("LITERAL_API_KEY"))

score = literalai_client.api.create_score(
    step_id="<STEP_UUID>",
    name="Sentiment",
    label="Positive",
    type="AI",
    comment="The user tipped $100, that means she is happy with the service.",
    value=1.0
)