AI Experiments Lab for response testing
Ask Powerfront to replay test datasets against AI features and compare responses, scores and usage.
What it is
AI Experiments Lab runs saved test examples against an AI feature or a guardrail service. It records outputs, errors, scores, token usage and duration so you can compare runs. Powerfront operates these tests for you in this release.
Who uses it and why
- Administrators supply representative examples and expected outcomes.
- Developers work with Powerfront to compare feature settings, accounts or guardrail behaviour.
Where to find it
Ask Powerfront to run an experiment in AI Experiments Lab. Its workspace includes New Experiment, Datasets and comparison views.
How it works
- Prepare a dataset of text examples or supported conversation examples, with expected outputs where useful.
- Powerfront chooses Mode, then an AI Feature or guardrail Service.
- Powerfront selects a compatible Dataset, optionally names the run and chooses Scoring.
- Powerfront selects Start Experiment. Enabled internal dataset items run one after another.
- Review each output and error, then compare completed runs using the same examples.
Feature tests support Autopilot, Copilot, translation, summaries, spelling checks, spam checks and conversation metrics. Guardrail tests specify inbound or outbound checks per example and compare the expected allow, block or routing result.
Configure it
Give Powerfront the feature to test, examples, expected answers and the change you want to compare. Keep the dataset consistent across comparative runs.
The editor defaults Account Override to Default (from feature) and Scoring to None. Exact Match compares text after lowercasing, trimming and normalising line endings. Powerfront can also arrange AI evaluator scoring.
Contains uses the same text normalisation. For an expected answer shorter than ten characters, it checks whether that text appears in the response. For longer answers, it normally scores the proportion of expected words longer than three characters that appear in the response. If there are no such words, it uses the whole-text check.
Ask Powerfront to review test tools and account overrides before testing a feature used for live conversations. This release does not establish complete separation from live configuration or external actions.
Fields
| Field | Where | Required | What it is for |
|---|---|---|---|
| Mode | AI Experiments Lab | Yes | Chooses a feature test or guardrail test. |
| AI Feature | AI Experiments Lab | Yes | Selects the feature for a feature test. |
| Service | AI Experiments Lab | Yes | Selects the guardrail service for a guardrail test, instead of an AI feature. |
| Dataset | AI Experiments Lab | Yes | Selects compatible test examples for the run. |
| Run Name | AI Experiments Lab | No | Gives the run a recognisable name; generated automatically if blank. |
| Description | AI Experiments Lab | No | Records the purpose of the run. |
| Account Override | AI Experiments Lab | No | Uses another configured AI account for the test; otherwise uses the feature account. |
| Scoring | AI Experiments Lab | No | Chooses None, Contains or Exact Match in the experiment editor. |
| Target Language | AI Experiments Lab | No | Sets the target language for supported language tests. |
| Source Language | AI Experiments Lab | No | Sets the source language for supported language tests; blank requests detection. |
Permissions
| Permission | Lets you |
|---|---|
| Powerfront | Manage test datasets and run and review AI experiments for your website. |
Limits and good to know
- These are dataset replays. They do not allocate live visitors between competing versions.
- The dataset must match the selected test type. An unsupported feature cannot run as an experiment.
- Built-in scoring needs an expected output and a successful test response.
- A run can finish with item errors. Check individual results as well as totals.
- Cancellation stops further processing; it does not promise to undo work already done.
- Saved results do not guarantee that an interrupted run resumes after a service restart.
Depends on and works with
View all connections as a list
This feature uses
- AI feature settings and change historyConfigured by
Feature-mode experiments use the selected feature configuration.
- AI guardrail routing and fallback actionsConfigured by
Guardrail experiments evaluate the routing rules configured on the selected connection.
- AI response quality evaluatorsEnhances
Selected AI evaluators can judge successful test outputs.
- AI accounts and shared defaultsConfigured by
Feature-mode experiments use the selected feature account or an account override.
- Shared AI service connectionsConfigured by
The selected external connection supplies remote datasets and receives experiment results.
- AI execution tracing with LangfuseEnhances
A configured tracing service can receive experiment traces.
- AI guardrail policy connectionsEnhances
A configured guardrail service enables guardrail-policy experiments.
Used by
- AI response quality evaluatorsRelated
Experiments can use selected evaluators to score successful test outputs.
- Shared AI service connectionsEnhances
LangFuse connections can provide experiment datasets and results.
- AI execution tracing with LangfuseEnhances
Experiment runs can link dataset results and scores with external traces.
FAQ and troubleshooting
Why is a dataset missing from the selection?
The editor filters datasets by the selected feature or guardrail test type. Ask Powerfront to check the dataset type.
Why is a successful response unscored?
Scoring may be None, the example may lack an expected output, or no enabled evaluator may have produced a score.
Can we compare two models?
Yes. Powerfront can use configured accounts for separate runs against the same dataset. Account overrides need care when the feature is also used live.
Does a high Contains score mean the whole answer is correct?
No. It checks text or word coverage, depending on the expected answer's length. Review the full answer or use a suitable evaluator for wider quality checks.