Skip to content

Testing AI Features ​

Best practices for testing AI-integrated features and AI-assisted test generation.

Overview ​

Testing AI features presents unique challenges:

  • Non-deterministic outputs
  • Context-dependent behavior
  • Prompt sensitivity
  • Performance variability
  • Cost considerations

This guide covers both:

  1. Using AI to generate tests for your code
  2. Testing AI-integrated features in your application

Part 1: AI-Assisted Test Generation ​

Generating Unit Tests ​

typescript
async function generateUnitTests(
  code: string,
  framework: 'jest' | 'mocha' | 'vitest'
): Promise<string> {
  const prompt = `
Generate comprehensive unit tests for this code using ${framework}.

**Requirements:**
- Test happy paths
- Test edge cases
- Test error conditions
- Mock external dependencies
- Use descriptive test names
- Achieve high code coverage

**Code to test:**
${code}

Provide complete, runnable test file.
`;

  const response = await llm.complete({
    model: 'claude-3-5-sonnet-20241022',
    messages: [{ role: 'user', content: prompt }],
  });

  return response.content;
}

Test Case Generation ​

typescript
interface TestCase {
  description: string;
  input: any;
  expected: any;
  category: 'happy' | 'edge' | 'error';
}

async function generateTestCases(
  functionSignature: string,
  description: string
): Promise<TestCase[]> {
  const prompt = `
Generate comprehensive test cases for this function.

**Function:**
${functionSignature}

**Description:**
${description}

**Categories needed:**
1. Happy path (normal, expected inputs)
2. Edge cases (boundaries, empty, large values)
3. Error cases (invalid inputs, unexpected states)

Return as JSON array of test cases:
[{
  "description": "test description",
  "input": {...},
  "expected": {...},
  "category": "happy|edge|error"
}]
`;

  const response = await llm.complete({
    model: 'claude-3-5-sonnet-20241022',
    messages: [{ role: 'user', content: prompt }],
  });

  return JSON.parse(extractJSON(response.content));
}

Integration Test Generation ​

typescript
async function generateIntegrationTests(
  apiEndpoint: string,
  spec: string
): Promise<string> {
  const prompt = `
Generate integration tests for this API endpoint.

**Endpoint:** ${apiEndpoint}

**Specification:**
${spec}

**Include tests for:**
- Successful requests
- Authentication/authorization
- Input validation
- Error responses
- Rate limiting
- Concurrent requests

Use supertest for HTTP testing.
Format as complete test file.
`;

  const response = await llm.complete({
    model: 'claude-3-5-sonnet-20241022',
    messages: [{ role: 'user', content: prompt }],
  });

  return response.content;
}

Test Data Generation ​

typescript
async function generateTestData(
  schema: any,
  count: number,
  constraints?: any
): Promise<any[]> {
  const prompt = `
Generate ${count} realistic test data records.

**Schema:**
${JSON.stringify(schema, null, 2)}

${constraints ? `**Constraints:**\n${JSON.stringify(constraints, null, 2)}` : ''}

**Requirements:**
- Realistic, diverse data
- Valid according to schema
- Edge cases included (if count > 10)
- No duplicate IDs

Return as JSON array.
`;

  const response = await llm.complete({
    model: 'claude-3-5-sonnet-20241022',
    messages: [{ role: 'user', content: prompt }],
  });

  return JSON.parse(extractJSON(response.content));
}

Part 2: Testing AI-Integrated Features ​

Testing Strategy ​

typescript
interface AIFeatureTest {
  // Input to the AI
  input: string;

  // Expected characteristics of output
  expectations: {
    contains?: string[];      // Must contain these strings
    notContains?: string[];   // Must not contain these
    minLength?: number;       // Minimum response length
    maxLength?: number;       // Maximum response length
    format?: 'json' | 'markdown' | 'code';
    sentiment?: 'positive' | 'negative' | 'neutral';
  };

  // Validation function
  validate?: (output: string) => boolean;
}

Golden Dataset Testing ​

Create a dataset of known inputs and acceptable outputs:

typescript
const goldenDataset: AIFeatureTest[] = [
  {
    input: 'Explain what a closure is in JavaScript',
    expectations: {
      contains: ['function', 'scope', 'variable'],
      minLength: 100,
      format: 'markdown',
    },
    validate: (output) => {
      // Custom validation
      return output.includes('function') &&
             output.includes('scope');
    },
  },
  // ... more test cases
];

async function runGoldenDatasetTests() {
  const results = [];

  for (const test of goldenDataset) {
    const output = await aiFeature(test.input);
    const passed = validateOutput(output, test.expectations);

    results.push({
      input: test.input,
      output,
      passed,
      expectations: test.expectations,
    });
  }

  return results;
}

Regression Testing ​

Track AI behavior over time:

typescript
interface RegressionBaseline {
  version: string;
  testCases: Array<{
    input: string;
    output: string;
    quality: number;
  }>;
}

class AIRegressionTester {
  private baseline: RegressionBaseline;

  async createBaseline(version: string): Promise<void> {
    const testCases = await this.runAllTests();
    this.baseline = { version, testCases };
    await this.saveBaseline();
  }

  async testAgainstBaseline(): Promise<RegressionReport> {
    const current = await this.runAllTests();
    const diffs = [];

    for (let i = 0; i < current.length; i++) {
      const similarity = this.calculateSimilarity(
        current[i].output,
        this.baseline.testCases[i].output
      );

      if (similarity < 0.8) {  // Significant change
        diffs.push({
          testCase: i,
          similarity,
          baseline: this.baseline.testCases[i].output,
          current: current[i].output,
        });
      }
    }

    return { diffs, overallSimilarity: this.avg(diffs.map(d => d.similarity)) };
  }

  private calculateSimilarity(a: string, b: string): number {
    // Implement similarity calculation (e.g., Levenshtein distance)
    return similarity(a, b);
  }
}

Property-Based Testing ​

Test properties that should always hold:

typescript
import fc from 'fast-check';

describe('AI Code Generator', () => {
  it('should always generate valid JavaScript', async () => {
    await fc.assert(
      fc.asyncProperty(
        fc.string(), // Random description
        async (description) => {
          const code = await generateCode(description);

          // Property: Output should be parseable JavaScript
          try {
            new Function(code);  // Try to parse
            return true;
          } catch {
            return false;
          }
        }
      )
    );
  });

  it('should never include sensitive patterns', async () => {
    await fc.assert(
      fc.asyncProperty(
        fc.string(),
        async (input) => {
          const output = await aiFeature(input);

          // Property: Never output API keys or secrets
          const sensitivePatterns = [
            /sk-[a-zA-Z0-9]{32,}/,
            /-----BEGIN .* KEY-----/,
          ];

          return !sensitivePatterns.some(p => p.test(output));
        }
      )
    );
  });
});

Performance Testing ​

Test AI feature performance:

typescript
interface PerformanceMetrics {
  latency: number;      // Response time
  tokens: number;       // Tokens used
  cost: number;         // API cost
  cacheHits: number;    // Cache performance
}

async function testPerformance(
  testCases: string[],
  iterations: number = 100
): Promise<PerformanceMetrics> {
  const metrics: PerformanceMetrics = {
    latency: 0,
    tokens: 0,
    cost: 0,
    cacheHits: 0,
  };

  for (let i = 0; i < iterations; i++) {
    const testCase = testCases[i % testCases.length];
    const start = Date.now();

    const result = await aiFeature(testCase);

    metrics.latency += Date.now() - start;
    metrics.tokens += result.usage.total_tokens;
    metrics.cost += calculateCost(result.usage);
    metrics.cacheHits += result.cached ? 1 : 0;
  }

  // Calculate averages
  metrics.latency /= iterations;
  metrics.tokens /= iterations;
  metrics.cost /= iterations;

  return metrics;
}

// Set performance budgets
describe('AI Feature Performance', () => {
  it('should respond within budget', async () => {
    const metrics = await testPerformance(testCases);

    expect(metrics.latency).toBeLessThan(2000);  // 2s max
    expect(metrics.tokens).toBeLessThan(1000);   // Token budget
    expect(metrics.cost).toBeLessThan(0.01);     // Cost budget
  });
});

Monitoring and Observability ​

typescript
class AIFeatureMonitor {
  private metrics: {
    successRate: number;
    averageLatency: number;
    averageQuality: number;
    errorTypes: Map<string, number>;
  };

  async logInvocation(
    input: string,
    output: string,
    metadata: any
  ): Promise<void> {
    // Log to monitoring system
    await this.logger.log({
      timestamp: new Date(),
      input: this.sanitize(input),
      output: this.sanitize(output),
      latency: metadata.latency,
      tokens: metadata.tokens,
      model: metadata.model,
      success: metadata.success,
    });

    // Update metrics
    this.updateMetrics(metadata);
  }

  async checkHealth(): Promise<HealthStatus> {
    const recentMetrics = await this.getRecentMetrics();

    return {
      healthy: recentMetrics.successRate > 0.95,
      metrics: recentMetrics,
      alerts: this.generateAlerts(recentMetrics),
    };
  }
}

Testing Best Practices ​

1. Separate Deterministic and Non-Deterministic Tests ​

typescript
describe('Code Generation Feature', () => {
  describe('Deterministic tests', () => {
    // These should always pass
    it('should validate input', () => {
      expect(() => generateCode('')).toThrow();
    });

    it('should return a string', async () => {
      const result = await generateCode('hello');
      expect(typeof result).toBe('string');
    });
  });

  describe('Behavioral tests', () => {
    // These test AI behavior (may need flexibility)
    it('should generate valid code', async () => {
      const result = await generateCode('function to add two numbers');
      expect(result).toMatch(/function.*add/i);
      // Validate it's parseable JavaScript
      expect(() => new Function(result)).not.toThrow();
    });
  });
});

2. Use Snapshots Carefully ​

typescript
// ⚠️ Snapshots can be brittle for AI outputs
describe('AI Feature', () => {
  it('should generate expected output', async () => {
    const output = await aiFeature('test input');

    // Instead of exact snapshot:
    // expect(output).toMatchSnapshot(); // ❌ Too brittle

    // Test specific properties:
    expect(output).toContain('expected phrase'); // ✅
    expect(output.length).toBeGreaterThan(50);   // ✅
  });
});

3. Mock AI Calls in Unit Tests ​

typescript
// Mock AI service for fast, deterministic unit tests
class MockAIService implements AIService {
  private responses: Map<string, string>;

  constructor(responses: Record<string, string>) {
    this.responses = new Map(Object.entries(responses));
  }

  async complete(prompt: string): Promise<string> {
    return this.responses.get(prompt) || 'default response';
  }
}

// Use in tests
describe('Feature using AI', () => {
  it('should process AI response correctly', async () => {
    const mockAI = new MockAIService({
      'test prompt': 'test response',
    });

    const feature = new FeatureUsingAI(mockAI);
    const result = await feature.process('test prompt');

    expect(result).toBe('processed: test response');
  });
});

4. Test Error Handling ​

typescript
describe('AI Feature Error Handling', () => {
  it('should handle rate limits', async () => {
    // Simulate rate limit
    mockAI.setError({ status: 429, message: 'Rate limited' });

    const result = await aiFeatureWithRetry(input);

    expect(result).toBeDefined(); // Should retry and succeed
    expect(mockAI.callCount).toBeGreaterThan(1); // Retried
  });

  it('should handle invalid responses', async () => {
    mockAI.setResponse('invalid json {{{');

    await expect(parseAIResponse()).rejects.toThrow('Parse error');
  });
});

5. Test Cost Control ​

typescript
describe('Cost Controls', () => {
  it('should not exceed token budget', async () => {
    const monitor = new CostMonitor({ maxTokensPerRequest: 1000 });

    await expect(
      aiFeature('very long input...', { monitor })
    ).rejects.toThrow('Token budget exceeded');
  });

  it('should use caching effectively', async () => {
    const result1 = await aiFeature(input);
    const result2 = await aiFeature(input); // Same input

    expect(result2.cached).toBe(true);
    expect(result2.cost).toBeLessThan(result1.cost);
  });
});

Test Generation Checklist ​

When using AI to generate tests:

  • [ ] Review generated tests for correctness
  • [ ] Verify test cases cover edge cases
  • [ ] Ensure mocks are appropriate
  • [ ] Check that tests actually test the right thing
  • [ ] Run tests to verify they pass
  • [ ] Validate test assertions are meaningful
  • [ ] Ensure tests are maintainable

AI Feature Testing Checklist ​

When testing AI features:

  • [ ] Defined expected behavior (even if not exact)
  • [ ] Created golden dataset for regression
  • [ ] Implemented performance budgets
  • [ ] Added error handling tests
  • [ ] Set up monitoring and logging
  • [ ] Tested cost controls
  • [ ] Documented non-deterministic behavior
  • [ ] Implemented graceful degradation

Resources ​


Remember: Testing AI features requires a different mindset than testing deterministic code. Focus on properties, behaviors, and acceptable ranges rather than exact matches.

Enterprise AI Documentation Hub