- Tracely: An open-source tracing library based on OpenTelemetry.
- Evidently: An open-source library to run LLM evaluations and interact with Evidently Cloud.
- Evidently Cloud: A web platform to view traces and run evaluations.
- OpenAI: Used to simulate an LLM application.
Need help? Ask on Discord.
1. Installation
Install the necessary libraries:2. Set up workspace
2.1. Set up Evidently Cloud
- Sign up for a free Evidently Cloud account.
- Create an Organization if you log in for the first time. Get an ID of your organization. (Link).
- Get an API token. Click the Key icon in the left menu. Generate and save the token. (Link).
2.2. Create a Project
Connect to Evidently Cloud using your API token:2.3. Get Open AI key
Set up the OpenAI key (Token page) as an environment variable. See Open AI docs.3. Configure tracing
Set up and initialize tracing:-
The
address
is the destination backend to store collected traces. -
Project_id
is the ID of the Evidently Project you just created. Go to the Home page, enter the Project and copy its ID from above the dashboard. -
Dataset_name
helps identify the resulting Tracing dataset. All data with the same ID is grouped into a single dataset.
4. Trace a simple LLM app
Let’s create and trace a simple function that sends a list of questions to the LLM. Initialize the OpenAI client with the API key:create_trace_event
from Tracely
to trace the execution of the function and treat each as a separate session. This loops through the list of questions, captures input arguments and outputs and sends the data to Evidently Cloud:
5. View traces
Go to the Evidently Cloud, open your Project, and navigate to the “Traces” in the left menu. Open the traces you just sent. It might take a few moments until OpenAI processes all the inputs. You can now view, sort, export, and work with the traced dataset. You can switch between Traces, Dataset and Dialog view (select session there).
6. Run an evaluation (Optional)
You can run evaluations on this dataset both in the Cloud and locally. For local evaluations, first load the dataset to your Python environment: