LLM Inference servers (vLLM, etc.)

By integrating the OpenAI SDK with Literal AI’s instrumentation, you can also effectively monitor message-based inference servers such as LMStudio, vLLM or HuggingFace, ensuring that you have full visibility into the performance and usage of your AI models.

The same works for HuggingFace messages API with

base_url="https://api-inference.huggingface.co/models/mistralai/Mistral-7B-Instruct-v0.2/v1"