Quickstart - LLM tracing

LLM tracing "Hello world."

This quickstart shows how to instrument a simple LLM app to send inputs and outputs to Evidently Cloud. You will use the open-source Tracely library.

You will need an OpenAI key to create a toy LLM app.

Need help? Ask on Discord.

1. Set up Evidently Cloud

Set up your Evidently Cloud workspace:

• Sign up. If you do not have one yet, sign up for a free Evidently Cloud account.

• Create an Organization. When you log in the first time, create and name your Organization.

• Create a Project. Click + button under Project List. Create a Project, copy and save the Project ID. (Projects page)

• Get your API token. Click the Key icon in the left menu. Generate and save the token. (Token page).

You can now go to your Python environment.

2. Installation

Install the Tracely library to instrument your app:

!pip install tracely

Install the Evidently library to interact with Evidently Cloud:

!pip install evidently

Install the OpenAI library to create a toy app:

!pip install openai

Imports:

import os
import openai
import time
from tracely import init_tracing
from tracely import trace_event

2. Initialize Tracing

Initialize the OpenAI client. Pass the token as an environment variable:

# os.environ["OPENAI_API_KEY"] = "YOUR_KEY"
client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

Set up tracing parameters. Copy the Team ID from the Teams page, and give a name to identify your tracing dataset.

init_tracing(
    address="https://app.evidently.cloud/",
    api_key="EVIDENTLY_API_KEY",
    project_id="YOUR_PROJECT_ID"
    export_name="LLM tracing example"
    )

3. Trace a simple function

Create a simple function to send questions to Open AI API and receive a completion. Set the questions list:

question_list = [
    "What is Evidently Python library?",
    "What is LLM observability?",
    "How is MLOps different from LLMOps?"
]

Create a function and use the trace_event() decorator to trace it:

@trace_event()
def pseudo_assistant(question):
    system_prompt = "You are a helpful assistant. Please answer the following question concisely."
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": question},
    ]
    return client.chat.completions.create(model="gpt-4o-mini", messages=messages).choices[0].message.content

# Iterate over the list of questions and pass each to the assistant
for question in question_list:
    response = pseudo_assistant(question=question)
    time.sleep(1)

4. View Traces

Go to the Evidently Cloud, open Datasets in the left menu (Datasets Page), and view your Traces.

What's next?

Want to run evaluations over this data? See a Quickstart.

Quickstart - LLM evaluations

Check out a more in-depth tutorial to learn more about tracing:

Tutorial - Tracing