Testing AI Features
Best practices for testing AI-integrated features and AI-assisted test generation.
Overview
Testing AI features presents unique challenges:
- Non-deterministic outputs
- Context-dependent behavior
- Prompt sensitivity
- Performance variability
- Cost considerations
This guide covers both:
- Using AI to generate tests for your code
- Testing AI-integrated features in your application
Part 1: AI-Assisted Test Generation
Generating Unit Tests
typescript
async function generateUnitTests(
code: string,
framework: 'jest' | 'mocha' | 'vitest'
): Promise<string> {
const prompt = `
Generate comprehensive unit tests for this code using ${framework}.
**Requirements:**
- Test happy paths
- Test edge cases
- Test error conditions
- Mock external dependencies
- Use descriptive test names
- Achieve high code coverage
**Code to test:**
${code}
Provide complete, runnable test file.
`;
const response = await llm.complete({
model: 'claude-3-5-sonnet-20241022',
messages: [{ role: 'user', content: prompt }],
});
return response.content;
}Test Case Generation
typescript
interface TestCase {
description: string;
input: any;
expected: any;
category: 'happy' | 'edge' | 'error';
}
async function generateTestCases(
functionSignature: string,
description: string
): Promise<TestCase[]> {
const prompt = `
Generate comprehensive test cases for this function.
**Function:**
${functionSignature}
**Description:**
${description}
**Categories needed:**
1. Happy path (normal, expected inputs)
2. Edge cases (boundaries, empty, large values)
3. Error cases (invalid inputs, unexpected states)
Return as JSON array of test cases:
[{
"description": "test description",
"input": {...},
"expected": {...},
"category": "happy|edge|error"
}]
`;
const response = await llm.complete({
model: 'claude-3-5-sonnet-20241022',
messages: [{ role: 'user', content: prompt }],
});
return JSON.parse(extractJSON(response.content));
}Integration Test Generation
typescript
async function generateIntegrationTests(
apiEndpoint: string,
spec: string
): Promise<string> {
const prompt = `
Generate integration tests for this API endpoint.
**Endpoint:** ${apiEndpoint}
**Specification:**
${spec}
**Include tests for:**
- Successful requests
- Authentication/authorization
- Input validation
- Error responses
- Rate limiting
- Concurrent requests
Use supertest for HTTP testing.
Format as complete test file.
`;
const response = await llm.complete({
model: 'claude-3-5-sonnet-20241022',
messages: [{ role: 'user', content: prompt }],
});
return response.content;
}Test Data Generation
typescript
async function generateTestData(
schema: any,
count: number,
constraints?: any
): Promise<any[]> {
const prompt = `
Generate ${count} realistic test data records.
**Schema:**
${JSON.stringify(schema, null, 2)}
${constraints ? `**Constraints:**\n${JSON.stringify(constraints, null, 2)}` : ''}
**Requirements:**
- Realistic, diverse data
- Valid according to schema
- Edge cases included (if count > 10)
- No duplicate IDs
Return as JSON array.
`;
const response = await llm.complete({
model: 'claude-3-5-sonnet-20241022',
messages: [{ role: 'user', content: prompt }],
});
return JSON.parse(extractJSON(response.content));
}Part 2: Testing AI-Integrated Features
Testing Strategy
typescript
interface AIFeatureTest {
// Input to the AI
input: string;
// Expected characteristics of output
expectations: {
contains?: string[]; // Must contain these strings
notContains?: string[]; // Must not contain these
minLength?: number; // Minimum response length
maxLength?: number; // Maximum response length
format?: 'json' | 'markdown' | 'code';
sentiment?: 'positive' | 'negative' | 'neutral';
};
// Validation function
validate?: (output: string) => boolean;
}Golden Dataset Testing
Create a dataset of known inputs and acceptable outputs:
typescript
const goldenDataset: AIFeatureTest[] = [
{
input: 'Explain what a closure is in JavaScript',
expectations: {
contains: ['function', 'scope', 'variable'],
minLength: 100,
format: 'markdown',
},
validate: (output) => {
// Custom validation
return output.includes('function') &&
output.includes('scope');
},
},
// ... more test cases
];
async function runGoldenDatasetTests() {
const results = [];
for (const test of goldenDataset) {
const output = await aiFeature(test.input);
const passed = validateOutput(output, test.expectations);
results.push({
input: test.input,
output,
passed,
expectations: test.expectations,
});
}
return results;
}Regression Testing
Track AI behavior over time:
typescript
interface RegressionBaseline {
version: string;
testCases: Array<{
input: string;
output: string;
quality: number;
}>;
}
class AIRegressionTester {
private baseline: RegressionBaseline;
async createBaseline(version: string): Promise<void> {
const testCases = await this.runAllTests();
this.baseline = { version, testCases };
await this.saveBaseline();
}
async testAgainstBaseline(): Promise<RegressionReport> {
const current = await this.runAllTests();
const diffs = [];
for (let i = 0; i < current.length; i++) {
const similarity = this.calculateSimilarity(
current[i].output,
this.baseline.testCases[i].output
);
if (similarity < 0.8) { // Significant change
diffs.push({
testCase: i,
similarity,
baseline: this.baseline.testCases[i].output,
current: current[i].output,
});
}
}
return { diffs, overallSimilarity: this.avg(diffs.map(d => d.similarity)) };
}
private calculateSimilarity(a: string, b: string): number {
// Implement similarity calculation (e.g., Levenshtein distance)
return similarity(a, b);
}
}Property-Based Testing
Test properties that should always hold:
typescript
import fc from 'fast-check';
describe('AI Code Generator', () => {
it('should always generate valid JavaScript', async () => {
await fc.assert(
fc.asyncProperty(
fc.string(), // Random description
async (description) => {
const code = await generateCode(description);
// Property: Output should be parseable JavaScript
try {
new Function(code); // Try to parse
return true;
} catch {
return false;
}
}
)
);
});
it('should never include sensitive patterns', async () => {
await fc.assert(
fc.asyncProperty(
fc.string(),
async (input) => {
const output = await aiFeature(input);
// Property: Never output API keys or secrets
const sensitivePatterns = [
/sk-[a-zA-Z0-9]{32,}/,
/-----BEGIN .* KEY-----/,
];
return !sensitivePatterns.some(p => p.test(output));
}
)
);
});
});Performance Testing
Test AI feature performance:
typescript
interface PerformanceMetrics {
latency: number; // Response time
tokens: number; // Tokens used
cost: number; // API cost
cacheHits: number; // Cache performance
}
async function testPerformance(
testCases: string[],
iterations: number = 100
): Promise<PerformanceMetrics> {
const metrics: PerformanceMetrics = {
latency: 0,
tokens: 0,
cost: 0,
cacheHits: 0,
};
for (let i = 0; i < iterations; i++) {
const testCase = testCases[i % testCases.length];
const start = Date.now();
const result = await aiFeature(testCase);
metrics.latency += Date.now() - start;
metrics.tokens += result.usage.total_tokens;
metrics.cost += calculateCost(result.usage);
metrics.cacheHits += result.cached ? 1 : 0;
}
// Calculate averages
metrics.latency /= iterations;
metrics.tokens /= iterations;
metrics.cost /= iterations;
return metrics;
}
// Set performance budgets
describe('AI Feature Performance', () => {
it('should respond within budget', async () => {
const metrics = await testPerformance(testCases);
expect(metrics.latency).toBeLessThan(2000); // 2s max
expect(metrics.tokens).toBeLessThan(1000); // Token budget
expect(metrics.cost).toBeLessThan(0.01); // Cost budget
});
});Monitoring and Observability
typescript
class AIFeatureMonitor {
private metrics: {
successRate: number;
averageLatency: number;
averageQuality: number;
errorTypes: Map<string, number>;
};
async logInvocation(
input: string,
output: string,
metadata: any
): Promise<void> {
// Log to monitoring system
await this.logger.log({
timestamp: new Date(),
input: this.sanitize(input),
output: this.sanitize(output),
latency: metadata.latency,
tokens: metadata.tokens,
model: metadata.model,
success: metadata.success,
});
// Update metrics
this.updateMetrics(metadata);
}
async checkHealth(): Promise<HealthStatus> {
const recentMetrics = await this.getRecentMetrics();
return {
healthy: recentMetrics.successRate > 0.95,
metrics: recentMetrics,
alerts: this.generateAlerts(recentMetrics),
};
}
}Testing Best Practices
1. Separate Deterministic and Non-Deterministic Tests
typescript
describe('Code Generation Feature', () => {
describe('Deterministic tests', () => {
// These should always pass
it('should validate input', () => {
expect(() => generateCode('')).toThrow();
});
it('should return a string', async () => {
const result = await generateCode('hello');
expect(typeof result).toBe('string');
});
});
describe('Behavioral tests', () => {
// These test AI behavior (may need flexibility)
it('should generate valid code', async () => {
const result = await generateCode('function to add two numbers');
expect(result).toMatch(/function.*add/i);
// Validate it's parseable JavaScript
expect(() => new Function(result)).not.toThrow();
});
});
});2. Use Snapshots Carefully
typescript
// ⚠️ Snapshots can be brittle for AI outputs
describe('AI Feature', () => {
it('should generate expected output', async () => {
const output = await aiFeature('test input');
// Instead of exact snapshot:
// expect(output).toMatchSnapshot(); // ❌ Too brittle
// Test specific properties:
expect(output).toContain('expected phrase'); // ✅
expect(output.length).toBeGreaterThan(50); // ✅
});
});3. Mock AI Calls in Unit Tests
typescript
// Mock AI service for fast, deterministic unit tests
class MockAIService implements AIService {
private responses: Map<string, string>;
constructor(responses: Record<string, string>) {
this.responses = new Map(Object.entries(responses));
}
async complete(prompt: string): Promise<string> {
return this.responses.get(prompt) || 'default response';
}
}
// Use in tests
describe('Feature using AI', () => {
it('should process AI response correctly', async () => {
const mockAI = new MockAIService({
'test prompt': 'test response',
});
const feature = new FeatureUsingAI(mockAI);
const result = await feature.process('test prompt');
expect(result).toBe('processed: test response');
});
});4. Test Error Handling
typescript
describe('AI Feature Error Handling', () => {
it('should handle rate limits', async () => {
// Simulate rate limit
mockAI.setError({ status: 429, message: 'Rate limited' });
const result = await aiFeatureWithRetry(input);
expect(result).toBeDefined(); // Should retry and succeed
expect(mockAI.callCount).toBeGreaterThan(1); // Retried
});
it('should handle invalid responses', async () => {
mockAI.setResponse('invalid json {{{');
await expect(parseAIResponse()).rejects.toThrow('Parse error');
});
});5. Test Cost Control
typescript
describe('Cost Controls', () => {
it('should not exceed token budget', async () => {
const monitor = new CostMonitor({ maxTokensPerRequest: 1000 });
await expect(
aiFeature('very long input...', { monitor })
).rejects.toThrow('Token budget exceeded');
});
it('should use caching effectively', async () => {
const result1 = await aiFeature(input);
const result2 = await aiFeature(input); // Same input
expect(result2.cached).toBe(true);
expect(result2.cost).toBeLessThan(result1.cost);
});
});Test Generation Checklist
When using AI to generate tests:
- [ ] Review generated tests for correctness
- [ ] Verify test cases cover edge cases
- [ ] Ensure mocks are appropriate
- [ ] Check that tests actually test the right thing
- [ ] Run tests to verify they pass
- [ ] Validate test assertions are meaningful
- [ ] Ensure tests are maintainable
AI Feature Testing Checklist
When testing AI features:
- [ ] Defined expected behavior (even if not exact)
- [ ] Created golden dataset for regression
- [ ] Implemented performance budgets
- [ ] Added error handling tests
- [ ] Set up monitoring and logging
- [ ] Tested cost controls
- [ ] Documented non-deterministic behavior
- [ ] Implemented graceful degradation
Resources
Remember: Testing AI features requires a different mindset than testing deterministic code. Focus on properties, behaviors, and acceptable ranges rather than exact matches.