Data for Recommendations
How to define the data schema for ranking and recommendations.
Last updated
How to define the data schema for ranking and recommendations.
Last updated
To evaluate data from recommender systems, you must correctly map the input data schema. You can also pass an optional additional dataset with training data.
Note: this mapping will also apply to search and retrieval systems. Treat "user_id" as "query_id".
Notebook example on using column mapping and additional data for recommender systems:
You must define column mapping to run evaluations for recommender or ranking systems on your current and (optional) reference data. Column mapping helps point to the columns with user ID, item ID, prediction, and target.
To evaluate the quality of a ranking or a recommendation system, you must pass:
The score or rank generated by the system as the prediction.
The relevance labels as the target (e.g., this could be an interaction result like user click, assigned relevance label, etc.)
Here are the examples of the expected data inputs.
If the model prediction is a score (expected by default):
| user_id | item_id | prediction (score) | target (relevance) |
|---|---|---|---|
user_1 | item_1 | 1.95 | 0 |
user_1 | item_2 | 0.8 | 1 |
user_1 | item_3 | 0.05 | 0 |
If the model prediction is a rank:
| user_id | item_id | prediction (rank) | target (relevance) |
|---|---|---|---|
user_1 | item_1 | 1 | 0 |
user_1 | item_2 | 2 | 1 |
user_1 | item_3 | 3 | 0 |
The target column with the interaction result or relevance label can contain either:
a binary label (where 1 is a positive outcome)
any true labels or scores (any positive values, where a higher value corresponds to a better match or a more valuable user action).
You might need to add additional details about your dataset via column mapping:
recommendations_type: score (default) or rank. Helps specify whether the prediction column contains ranking or predicted score.
user_id: helps specify the column that contains user IDs.
item_id: helps specify the column that contains ranked items.
Some metrics like novelty or popularity bias require training data, which has a different structure from production data. To pass it, use the additional_data object. You can pass your training data as current_train_data and (optional) reference_train_data.
Example:
The additional training dataset should have the following structure:
| user | item | target |
|---|---|---|
id1 | id1 | 1 |
id2 | id9 | 1 |
id3 | id2 | 1 |
id3 | id1 | 1 |
id4 | id6 | 1 |
The names of the columns with user_id and item_id should match the corresponding columns in the current (and optional reference) data.
The name of the column with the interaction result should match the name of the target column in the current (and optional reference) data.
If you use metrics that refer to specific columns (such as UserBiasMetric metric), these columns must also be present in the training dataset.
You can pass a single training dataset or two datasets (in case your reference and current dataset have different training data).
The reference dataset can belong to a previous production period or a different model you compare against. The training dataset is used to train the model. Their structure usually differs:
Production data typically includes a list of all recommended items, where some of them earn a positive interaction result. It also contains negative examples (ignored recommendations) and data about model prediction (predicted rank or score).
Training data typically contains a history of positive actions, such as user viewing history, page reads, or upvotes. Since it only includes the interaction results, it lacks negative examples (e.g., ignored recommendations) and column with the model output (predicted rank or score).