Adversarial testing

Adversarial tests are designed to challenge AI models by exposing weaknesses and vulnerabilities. These inputs may attempt to:

• Bypass safety protections and generate harmful responses.
• Trick the model into revealing sensitive or unintended information.
• Exploit edge cases to evaluate system robustness.

Evidently Cloud lets you automate adversarial test generation based on defined categories of risk.

​

Create an adversarial test dataset

You can configure your own adversarial dataset.

​

1. Create a Project

In the Evidently UI, start a new Project or open an existing one.

• Navigate to “Datasets” in the left menu.
• Click “Generate” and select the “Adversarial testing” option.

​

2. Select a test scenario

Choose a predefined adversarial scenario: You can choose the following categories:

• Harmful content (e.g., profanity, toxicity, illegal advice).
• Forbidden topics (e.g., financial, legal, medical queries).
• Brand image (eliciting negative feedback on a company or product).
• Competition (comparisons with competitor products).
• Offers and promises (attempting to get AI to make commitments).
• Hijacking (out-of-scope questions unrelated to the intended purpose).
• Prompt leakage (extracting system instructions or hidden prompts).

​

3. Configure the dataset

After selecting a scenario

• Provide an optional dataset name and description. (This applies if you export each dataset separately).
• Set the number of inputs to generate.

Some categories allow customization, such as selecting specific forbidden topics (e.g., legal, financial, or medical advice). You can configure multiple scenarios at once.

​

4. Generate the data

You can choose to:

• Combine multiple scenarios into a single dataset. If you select multiple categories (e.g., Brand Image and Forbidden Topics), they will be included in the same dataset, with a separate “scenario” column to indicate the category of each test case.
• Export each scenario separately. Generate individual datasets for each selected test type.

Once generated, you can:

• Open and edit each dataset as needed.
• Download it as a CSV file.
• Access it via the Python API using the dataset ID.