# AI Experiments Lab for response testing

Ask Powerfront to replay test datasets against AI features and compare responses, scores and usage.

*Release 3.3 · admin, developer*

## What it is

**AI Experiments Lab** runs saved test examples against an AI feature or a guardrail service. It records outputs, errors, scores, token usage and duration so you can compare runs. Powerfront operates these tests for you in this release.

## Who uses it and why

- Administrators supply representative examples and expected outcomes.
- Developers work with Powerfront to compare feature settings, accounts or guardrail behaviour.

## Where to find it

Ask Powerfront to run an experiment in **AI Experiments Lab**. Its workspace includes **New Experiment**, **Datasets** and comparison views.

## How it works

1. Prepare a dataset of text examples or supported conversation examples, with expected outputs where useful.
2. Powerfront chooses **Mode**, then an **AI Feature** or guardrail **Service**.
3. Powerfront selects a compatible **Dataset**, optionally names the run and chooses **Scoring**.
4. Powerfront selects **Start Experiment**. Enabled internal dataset items run one after another.
5. Review each output and error, then compare completed runs using the same examples.

Feature tests support Autopilot, Copilot, translation, summaries, spelling checks, spam checks and conversation metrics. Guardrail tests specify inbound or outbound checks per example and compare the expected allow, block or routing result.

## Configure it

Give Powerfront the feature to test, examples, expected answers and the change you want to compare. Keep the dataset consistent across comparative runs.

The editor defaults **Account Override** to **Default (from feature)** and **Scoring** to **None**. **Exact Match** compares text after lowercasing, trimming and normalising line endings. Powerfront can also arrange AI evaluator scoring.

**Contains** uses the same text normalisation. For an expected answer shorter than ten characters, it checks whether that text appears in the response. For longer answers, it normally scores the proportion of expected words longer than three characters that appear in the response. If there are no such words, it uses the whole-text check.

Ask Powerfront to review test tools and account overrides before testing a feature used for live conversations. This release does not establish complete separation from live configuration or external actions.

### Fields

| Field | Where | Required | What it is for |
|---|---|---|---|
| Mode | AI Experiments Lab | Yes | Chooses a feature test or guardrail test. |
| AI Feature | AI Experiments Lab | Yes | Selects the feature for a feature test. |
| Service | AI Experiments Lab | Yes | Selects the guardrail service for a guardrail test, instead of an AI feature. |
| Dataset | AI Experiments Lab | Yes | Selects compatible test examples for the run. |
| Run Name | AI Experiments Lab | No | Gives the run a recognisable name; generated automatically if blank. |
| Description | AI Experiments Lab | No | Records the purpose of the run. |
| Account Override | AI Experiments Lab | No | Uses another configured AI account for the test; otherwise uses the feature account. |
| Scoring | AI Experiments Lab | No | Chooses None, Contains or Exact Match in the experiment editor. |
| Target Language | AI Experiments Lab | No | Sets the target language for supported language tests. |
| Source Language | AI Experiments Lab | No | Sets the source language for supported language tests; blank requests detection. |

### Permissions

| Permission | Lets you |
|---|---|
| Powerfront | Manage test datasets and run and review AI experiments for your website. |

## Limits and good to know

- These are dataset replays. They do not allocate live visitors between competing versions.
- The dataset must match the selected test type. An unsupported feature cannot run as an experiment.
- Built-in scoring needs an expected output and a successful test response.
- A run can finish with item errors. Check individual results as well as totals.
- Cancellation stops further processing; it does not promise to undo work already done.
- Saved results do not guarantee that an interrupted run resumes after a service restart.

## Depends on and works with

### This feature uses

- **Configured in** [AI feature settings and change history](AI-001.md): Feature-mode experiments use the selected feature configuration.
- **Configured in** [AI guardrail routing and fallback actions](AI-025.md): Guardrail experiments evaluate the routing rules configured on the selected connection.
- **Configured in** [AI accounts and shared defaults](AIP-001.md): Feature-mode experiments use the selected feature account or an account override.
- **Configured in** [Shared AI service connections](AIP-013.md): The selected external connection supplies remote datasets and receives experiment results.
- **Works better with** [AI response quality evaluators](AI-031.md): Selected AI evaluators can judge successful test outputs.
- **Works better with** [AI execution tracing with Langfuse](AIP-015.md): A configured tracing service can receive experiment traces.
- **Works better with** [AI guardrail policy connections](AIP-022.md): A configured guardrail service enables guardrail-policy experiments.

### Used by

- **Enhances** [Shared AI service connections](AIP-013.md): LangFuse connections can provide experiment datasets and results.
- **Enhances** [AI execution tracing with Langfuse](AIP-015.md): Experiment runs can link dataset results and scores with external traces.
- **Related** [AI response quality evaluators](AI-031.md): Experiments can use selected evaluators to score successful test outputs.

## FAQ and troubleshooting

### Why is a dataset missing from the selection?

The editor filters datasets by the selected feature or guardrail test type. Ask Powerfront to check the dataset type.

### Why is a successful response unscored?

Scoring may be **None**, the example may lack an expected output, or no enabled evaluator may have produced a score.

### Can we compare two models?

Yes. Powerfront can use configured accounts for separate runs against the same dataset. Account overrides need care when the feature is also used live.

### Does a high Contains score mean the whole answer is correct?

No. It checks text or word coverage, depending on the expected answer's length. Review the full answer or use a suitable evaluator for wider quality checks.