Skip to main content

​
Semantics

Literal AI Platform
Literal AI approaches LLM logging at three levels:
  1. Generation: Log of a single LLM call. (Generations are Steps.)
  2. Step: Log of a regular function execution, which is usually an intermediate step in an LLM system. Possible types are: tool, embedding, retrieval, rerank, undefined, etc. Steps can be considered as Spans.
  3. Run: Trace of an Agent/Chain run, including its intermediate steps. Can contain one or multiple generations.
  4. Thread: A collection of Runs that are part of a single conversation.
Literal AI Platform
You can log a generation only (typically for extraction use cases), or log a run only (typically for task automation), or combine them in threads (typically for chatbots).
See installation to get your API key and instantiate the SDK

​
Log an LLM Generation

Generations are logged by integrations with LLM providers. They capture the prompt, completion, settings, and token latency. Here is an example with OpenAI: 
literalai_client.instrument_openai()

# Run a regular OpenAI chat completion
from openai import OpenAI

oai = OpenAI()

def call_openai(user_input: str):
    oai.chat.completions.create(
        model="gpt-4o",
        messages=[{ "role": "user", "content": user_input }]
    )

call_openai("Hello world")

literalAiClient.instrumentation.openai();

const openai = new OpenAI();

async function callOpenAI(userInput: string) {
  const stream = await openai.chat.completions.create({
    model: 'gpt-4',
    stream: true,
    messages: [{ role: 'user', content: userInput }]
  });
};
callOpenAI("Hello world");
Check out the TypeScript client to learn more about the wrap function.

​
Multimodal LLM

You can leverage multimodal capabilities on Literal AI in two ways:
  • Simple logging on API calls to Multimodal LLM APIs, like gpt-4o
  • Save multimodal files as Attachments. Image, videos, audio and other files are shown as Attachment in the Literal AI platform, which can be accessed and downloaded via a Step.
A logged multimodal LLM call

​
Log a Run

A Run represents a trace of an Agent or Chain execution, capturing all intermediate steps and actions. Runs can be logged manually using decorators or through framework integrations such as Llama Index or LangChain.

​
Log a Run with Intermediate Steps

Here’s how you can log a Run with intermediate steps using Python and TypeScript: 
@literalai_client.step(type="tool")
def get_temperature(city: str):
    return "10C"

@literalai_client.run
def my_agent(user_input: str):
    # Reusing the OpenAI call from the previous example
    call_openai(user_input)
    # Naive tool example
    get_temperature("paris")
    return "Success"
    
my_agent("Hello world")

# Wait for all steps to be sent. This is NOT needed in production code.
literalai_client.flush()

async function getTemperature(city: string) {
  return literalAiClient
    .step({
      type: 'tool',
      name: 'Get Temperature',
      input: { city },
    })
    .wrap(() => {
      return { content: '10C' };
    });
}

async function myAgent(userInput: string) {
  return literalAiClient
    .run({
      name: 'My Agent',
      input: { userInput }
    })
    .wrap(async () => {
      // Reusing the OpenAI call from the previous example
      await callOpenAI(userInput);
      // Naive tool example
      await getTemperature("Paris");
      return { content: 'Success' };
    });
}

myAgent("Hello world");

​
Add Metadata and Tags to Steps

Tags and Metadata can be added to both Runs and Steps to provide additional context and facilitate filtering and categorization. 
@literalai_client.run
def my_step(input):
    current_step = literalai_client.get_current_step()
    # some code, llm call, tool call, etc.
    current_step.metadata = {"region": "europe"}
    current_step.tags = ["to_review"]
    return "answer"

import { LiteralClient } from "@literalai/client";

const literalAiClient = new LiteralClient({apiKey: process.env["LITERAL_API_KEY"]});

// The Assistant could have intermediary steps
async function myRun() {
  return literalAiClient
    .step({
      type: 'run',
      name: 'My Run',
      input: { content: 'My run Input' },
      metadata: { region: "europe" },
      tags: ["to_review"],
    })
    .wrap(() => {
      return { content: 'Success' };
    });
}

​
Add Attachments to Steps

You can attach files to a Run or any of its intermediate steps, which is particularly useful for multimodal use cases.
Attachments on a step
@literalai_client.step(type="tool")
def load_document():
    with open ("./some.pdf", "rb") as file:
        literalai_client.api.create_attachment(
          name="pdf_document",
          content=file.read()
          )
    return "doc loaded"

async function loadDocument(city: string) {
  return literalAiClient
    .step({
      type: 'tool',
      input: { document },
    })
    .wrap(() => {
      const fileStream = fs.createReadStream('./some.pdf');
      const mime = 'application/pdf';
      await literalAiClient.api.createAttachment({
        content: fileStream,
        mime,
        name: 'pdf_document'
      });
      return { content: "doc loaded" };
    });
}

​
Learn More

The intermediate steps and the agent itself are logged using the Step class. You can learn more about the Step API in the following references:

Python Step API reference

Learn how to use the Python Step API.

TypeScript Step API reference

Learn how to use the TypeScript Step API.

​
Log a Thread

You can interact with an example Thread in the platform here. It is up to the application to keep track of the thread ID and pass it to the Literal AI client. Every run logged with the same thread ID will be part of the same conversation. Here is an example: 
import uuid

def process_message(thread_id: str, user_input: str):
    with literalai_client.thread(thread_id=thread_id) as thread:
        # Reusing the Agent from the previous example
        my_agent(user_input)

thread_id = str(uuid.uuid4())
# Calling the agent a first time
process_message(thread_id=thread_id, user_input="foo")
# Calling the agent a second time with the same thread ID
process_message(thread_id=thread_id, user_input="bar")

# Wait for all steps to be sent. This is NOT needed in production code.
literalai_client.flush()

import { v4 as uuidv4 } from 'uuid';

async function processMessage(threadId: string, userInput: string) {
    await literalAiClient
        .thread({ id: threadId})
        .wrap(async () => {
            await myAgent(userInput);
        });
}

const threadId = uuidv4();
// Calling the agent a first time
processMessage(threadId, "foo");
// Calling the agent a second time with the same thread ID
processMessage(threadId, "bar");
You can learn more about the Thread API in the following references:

Python Thread API reference

Learn how to use the Python Thread API.

TypeScript Thread API reference

Learn how to use the TypeScript Thread API.

​
Bind a Thread to a User

You can bind a Thread to a User to track their activity: quite handy for chatbots and conversational AIs! Simply provide a unique User identifier, such as an email. 
# If the user `john.doe@example.com` does not exist, it's automatically created.
def process_message(thread_id: str, user_input: str):
    with literalai_client.thread(thread_id=thread_id, 
                                 participant_id="john.doe@example.com") as thread:
        # Reusing the Agent from the previous example
        my_agent(user_input)

// If the user `john.doe@example.com` does not exist, it's automatically created.
async function processMessage(threadId: string, userInput: string) {
    await literalAiClient
        .thread({ participantId: "john.doe@example.com", id: threadId})
        .wrap(async () => {
            await myAgent(userInput);
        });
}
You can create a User at any time with the create_user API.
If your User already exists, you may update its metadata with the update_user API.The Literal AI client method thread() takes a participant_id (participantId in TypeScript) argument which accepts any of:
  • User.id: the unique ID of your User — it’s a UUID
  • User.identifier: the unique identifier of your User — it can be an email, a username, etc.
Careful with collisions when letting users pick their own identifier!

​
Log to a Specific Environment

Literal AI supports logging to different environments, which allows you to separate your development, staging, and production data: dev, staging, prod. This is particularly useful for managing your LLM application lifecycle. To specify an environment when initializing the LiteralClient, you can use the environment parameter: 
literalai_client = LiteralClient(environment="dev")

const literalAiClient = new LiteralClient({ environment: "dev" });

​
Log with a Release

Literal AI supports pairing your logs to a release, a release is a version of your deployed code to help you identify new issues and regressions. This is particularly useful for managing your LLM application once in production. The value can be arbitrary, but we recommend Semantic Versioning, Calendar Versioning, or the Git commit SHA. To specify a release when initializing the LiteralClient, you can use the release parameter: 
literalai_client = LiteralClient(release="81bec25")

const literalAiClient = new LiteralClient({ release: "81bec25" });
Your logs will have a new key release in the metadata.

​
Log a Distributed Trace

Distributed Tracing Cookbook

Learn how to log distributed traces with Literal AI.

​
Add a Score

Scores allow you to evaluate the LLM system performance at three levels: LLM generations, Agent Runs and Conversation Threads.
Scores can be human generated (human feedback, like a thump up or down), or AI generated (hallucination evaluation for instance). They can be visualized on the dashboard charts and used as filters.

​
Add a User Feedback

def add_user_feedback(run_id: str, value: int, comment: str):
    literalai_client.api.create_score(
        step_id=run_id,
        name="user-feedback",
        type="HUMAN",
        value=value,
        comment=comment,
    )

async function addUserFeedback(runId: string, value: number, comment: string) {
    const score = await literalAiClient.api.createScore({
        stepId: runId,
        name: 'user-feedback',
        type: 'HUMAN',
        comment,
        value,
      });
};
Correlate your LLM system to a product metric, such as conversion, churn, upsell, etc. This can be done by:
  • Adding a specific product-related score on Literal AI.
  • Sending the logged run id to your analytics system, such as PostHog or Amplitude.

​
Add an AI Evaluation Result

Refer to Online Evals

​
Fetch Existing Logs

You can fetch existing logs using the SDKs. Here is an example to fetch the last 5 threads where a user participated: 
from literalai import LiteralClient

literalai_client = LiteralClient()

user_id = 'uuid'

threads = literalai_client.api.list_threads(
    first=5,
    filters=[{"operator": "eq", "field": "participantId", "value": user_id}]
)

for d in threads.data:
    print(d.to_dict())

literalai_client.flush_and_stop()

import { LiteralClient } from '@literalai/client';
const literalAiClient = new LiteralClient("YOUR_API_KEY");

async function main() {

  const userId = 'uuid'
  const threads = await literalAiClient.api.getThreads({first: 5, filters: [{"operator": "eq", "field": "participantId", "value": userId}]});
  console.log(threads);
}
More generally, you can fetch any Literal AI object. Check out the SDKs and API reference to learn how.

​
On Literal AI

​
Filter logs

Leverage the powerful filters on Literal AI. Use these same filters to export your data using the SDKs.
Filter on existing logs

​
Debug logged LLM generations

Debug a logged LLM generation

​
Add Tags and Scores from the UI

You can add tags and scores directly from the user interface.
Add a Tag

​
Conclusion

Logging with Literal AI is composable and unopinionated. It can be done at different levels depending on your use case.