How to collect data from a live LLM app.
tracely
package from PyPi:
init_tracing
:
init_tracing()
Function ArgumentsParameter | Description | Environment Variable |
---|---|---|
address | Trace collector address. Defaults to https://app.evidently.cloud/ . | EVIDENTLY_TRACE_COLLECTOR |
api_key | Evidently Cloud API key. | EVIDENTLY_TRACE_COLLECTOR_API_KEY or EVIDENTLY_API_KEY |
export_name | Tracing dataset name. Traces with the same name are grouped into a single dataset. | EVIDENTLY_TRACE_COLLECTOR_EXPORT_NAME |
project_id | Destination Project ID in Evidently Cloud. | EVIDENTLY_TRACE_COLLECTOR_PROJECT_ID |
exporter_type | Trace export protocol: grpc or http . | - |
as_global | Registers the tracing provider globally (True ) or locally (False ). Default: True . Set to false if you want to initiate tracing to multiple datasets from the same environment. | - |
export_id
of the tracing dataset, run:
export_id
as a dataset id for download. See datasets API.
Tracely
is initialized, you can decorate your functions with trace_event
to start collecting traces for a specific function:
trace_event
Decorator ArgumentsParameter | Description | Default |
---|---|---|
span_name: Optional[str] | The name of the span to send in the event. | Function name |
track_args: Optional[List[str]] | A list of function arguments to include in the event. | None (all arguments included) |
ignore_args: Optional[List[str]] | A list of function arguments to exclude, e.g., arguments that contain sensitive data. | None (no arguments ignored) |
track_output: Optional[bool] | Indicates whether to track the function’s return value. | True |
parse_output: Optional[bool] | Indicates whether the result should be parsed, e.g., dict , list , and tuple types will be split into separate fields. | True |
@trace_event
decorator and nesting the functions. If a traced function is called inside another traced function, it will automatically appear as a nested child span, as long as it’s executed in the same call context (same thread).
For example:
create_trace_event()
calls inline inside the same function, using with
blocks.
create_trace_event
Function ArgumentsParameter | Description | Default |
---|---|---|
name | Span name. | - |
parse_output | Whether to parse the result into separate fields for dict , list , or tuple types. | True |
params | Key-value parameters to set as attributes. | - |
event
Object MethodsMethod | Description |
---|---|
set_attribute | Sets a custom attribute for the event. |
set_result | Sets a result for the event. Only one result can be set per event. |
session_id
. In this case traces will be separate but you can view the session in the UI to join them together - e.g. to read the chat conversation.
See the example above the “Context Manager” session.
get_current_span()
to get access to the current span:
get_current_span()
Object MethodsMethod | Description |
---|---|
set_attribute | Adds a new attribute to the active span. |
set_result | Sets a result field for the active span. (Has no effect in decorated functions that define return values). |
tracely.bind_to_trace
:
trace_id
.
trace_id
yourself, so you need to make sure it’s truly unique. If you reuse the same trace_id
, all events will be joined, even if they don’t belong together.