Logs are essential to monitor and improve your LLM app in production. Literal AI provides flexible and composable SDKs to log your LLM app at different levels of granularity.
Log Hierarchy on Literal AI
Literal AI approaches LLM logging at three levels:
type
s are: tool
, embedding
, retrieval
, rerank
, undefined
, etc. Steps can be considered as Spans.A `Thread` with runs & intermediate steps
You can log a generation only (typically for extraction use cases), or log a run only (typically for task automation), or combine them in threads (typically for chatbots).
Generations are logged by integrations with LLM providers. They capture the prompt, completion, settings, and token latency.
Here is an example with OpenAI:
Check out the TypeScript client to learn more about the wrap
function.
You can leverage multimodal capabilities on Literal AI in two ways:
gpt-4o
Attachment
in the Literal AI platform, which can be accessed and downloaded via a Step
.Example of a logged multimodal LLM call
A Run represents a trace of an Agent or Chain execution, capturing all intermediate steps and actions.
Runs can be logged manually using decorators or through framework integrations such as Llama Index or LangChain.
Here’s how you can log a Run with intermediate steps using Python and TypeScript:
Tags and Metadata can be added to both Runs and Steps to provide additional context and facilitate filtering and categorization.
You can attach files to a Run or any of its intermediate steps, which is particularly useful for multimodal use cases.
Example of attachments
The intermediate steps and the agent itself are logged using the Step
class. You can learn more about the Step API in the following references:
Learn how to use the Python Step API.
Learn how to use the TypeScript Step API.
You can interact with an example Thread in the platform here.
It is up to the application to keep track of the thread ID and pass it to the Literal AI client. Every run logged with the same thread ID will be part of the same conversation.
Here is an example:
You can learn more about the Thread API in the following references:
Learn how to use the Python Thread API.
Learn how to use the TypeScript Thread API.
Thread
to a User
You can bind a Thread
to a User
to track their activity: quite handy for chatbots and conversational AIs!
Simply provide a unique User
identifier, such as an email.
You can create a User
at any time with the create_user API.
If your User
already exists, you may update its metadata
with the update_user API.
The Literal AI client method thread()
takes a participant_id
(participantId
in TypeScript) argument which accepts any of:
User.id
: the unique ID of your User
— it’s a UUIDUser.identifier
: the unique identifier of your User
— it can be an email, a username, etc.Careful with collisions when letting users pick their own identifier
!
Literal AI supports logging to different environments, which allows you to separate your development, staging, and production data: dev
, staging
, prod
.
This is particularly useful for managing your LLM application lifecycle.
To specify an environment when initializing the LiteralClient, you can use the environment
parameter:
Literal AI supports pairing your logs to a release, a release is a version of your deployed code to help you identify new issues and regressions.
This is particularly useful for managing your LLM application once in production. The value can be arbitrary, but we recommend Semantic Versioning, Calendar Versioning, or the Git commit SHA.
To specify a release when initializing the LiteralClient, you can use the release
parameter:
Your logs will have a new key release
in the metadata.
Learn how to log distributed traces with Literal AI.
Scores can be human generated (human feedback, like a thump up or down), or AI generated (hallucination evaluation for instance).
They can be visualized on the dashboard charts and used as filters.
Correlate your LLM system to a product metric, such as conversion, churn, upsell, etc. This can be done by:
Refer to Online Evals
You can fetch existing logs using the SDKs. Here is an example to fetch the last 5 threads where a user participated:
More generally, you can fetch any Literal AI object. Check out the SDKs and API reference to learn how.
Leverage the powerful filters on Literal AI. Use these same filters to export your data using the SDKs.
Filter on logs
Replay a logged LLM generation in the Playground
You can add tags and scores directly from the user interface.
Add a Tag to a Thread
Logging with Literal AI is composable and unopinionated. It can be done at different levels depending on your use case.
Logs are essential to monitor and improve your LLM app in production. Literal AI provides flexible and composable SDKs to log your LLM app at different levels of granularity.
Log Hierarchy on Literal AI
Literal AI approaches LLM logging at three levels:
type
s are: tool
, embedding
, retrieval
, rerank
, undefined
, etc. Steps can be considered as Spans.A `Thread` with runs & intermediate steps
You can log a generation only (typically for extraction use cases), or log a run only (typically for task automation), or combine them in threads (typically for chatbots).
Generations are logged by integrations with LLM providers. They capture the prompt, completion, settings, and token latency.
Here is an example with OpenAI:
Check out the TypeScript client to learn more about the wrap
function.
You can leverage multimodal capabilities on Literal AI in two ways:
gpt-4o
Attachment
in the Literal AI platform, which can be accessed and downloaded via a Step
.Example of a logged multimodal LLM call
A Run represents a trace of an Agent or Chain execution, capturing all intermediate steps and actions.
Runs can be logged manually using decorators or through framework integrations such as Llama Index or LangChain.
Here’s how you can log a Run with intermediate steps using Python and TypeScript:
Tags and Metadata can be added to both Runs and Steps to provide additional context and facilitate filtering and categorization.
You can attach files to a Run or any of its intermediate steps, which is particularly useful for multimodal use cases.
Example of attachments
The intermediate steps and the agent itself are logged using the Step
class. You can learn more about the Step API in the following references:
Learn how to use the Python Step API.
Learn how to use the TypeScript Step API.
You can interact with an example Thread in the platform here.
It is up to the application to keep track of the thread ID and pass it to the Literal AI client. Every run logged with the same thread ID will be part of the same conversation.
Here is an example:
You can learn more about the Thread API in the following references:
Learn how to use the Python Thread API.
Learn how to use the TypeScript Thread API.
Thread
to a User
You can bind a Thread
to a User
to track their activity: quite handy for chatbots and conversational AIs!
Simply provide a unique User
identifier, such as an email.
You can create a User
at any time with the create_user API.
If your User
already exists, you may update its metadata
with the update_user API.
The Literal AI client method thread()
takes a participant_id
(participantId
in TypeScript) argument which accepts any of:
User.id
: the unique ID of your User
— it’s a UUIDUser.identifier
: the unique identifier of your User
— it can be an email, a username, etc.Careful with collisions when letting users pick their own identifier
!
Literal AI supports logging to different environments, which allows you to separate your development, staging, and production data: dev
, staging
, prod
.
This is particularly useful for managing your LLM application lifecycle.
To specify an environment when initializing the LiteralClient, you can use the environment
parameter:
Literal AI supports pairing your logs to a release, a release is a version of your deployed code to help you identify new issues and regressions.
This is particularly useful for managing your LLM application once in production. The value can be arbitrary, but we recommend Semantic Versioning, Calendar Versioning, or the Git commit SHA.
To specify a release when initializing the LiteralClient, you can use the release
parameter:
Your logs will have a new key release
in the metadata.
Learn how to log distributed traces with Literal AI.
Scores can be human generated (human feedback, like a thump up or down), or AI generated (hallucination evaluation for instance).
They can be visualized on the dashboard charts and used as filters.
Correlate your LLM system to a product metric, such as conversion, churn, upsell, etc. This can be done by:
Refer to Online Evals
You can fetch existing logs using the SDKs. Here is an example to fetch the last 5 threads where a user participated:
More generally, you can fetch any Literal AI object. Check out the SDKs and API reference to learn how.
Leverage the powerful filters on Literal AI. Use these same filters to export your data using the SDKs.
Filter on logs
Replay a logged LLM generation in the Playground
You can add tags and scores directly from the user interface.
Add a Tag to a Thread
Logging with Literal AI is composable and unopinionated. It can be done at different levels depending on your use case.