Remove redundant timeout parameters across E2E tests and add performance optimization documentation.

- Reduced E2E test execution time by removing unnecessary `{ timeout: 10000 }` overrides for assertions and element waits, relying on global `expect` configuration.
- Removed redundant `networkidle` wait states for faster form render validations.
- Documented comprehensive performance optimization strategies in `E2E_PERFORMANCE_OPTIMIZATION.md`.
- Added `E2E_COVERAGE_GUIDE.md` for integrating and merging E2E test coverage with unit test coverage.
This commit is contained in:
2025-11-09 00:30:56 +01:00
parent a6a10855fa
commit d5eb855ae1
15 changed files with 1024 additions and 107 deletions

View File

@@ -0,0 +1,636 @@
# E2E Coverage Integration Guide
This guide explains how to collect and merge E2E test coverage with unit test coverage to get a comprehensive view of your test coverage.
## 📋 Table of Contents
1. [Overview](#overview)
2. [Quick Start](#quick-start)
3. [Approach 1: V8 Coverage (Recommended)](#approach-1-v8-coverage-recommended)
4. [Approach 2: Istanbul Instrumentation](#approach-2-istanbul-instrumentation)
5. [Combined Coverage Workflow](#combined-coverage-workflow)
6. [Integration Steps](#integration-steps)
7. [Troubleshooting](#troubleshooting)
8. [FAQ](#faq)
---
## Overview
### Why Combined Coverage?
Your project uses a **dual testing strategy**:
- **Jest (Unit tests):** 97%+ coverage, excludes browser-specific code
- **Playwright (E2E tests):** Tests excluded files (layouts, API hooks, error boundaries)
**Combined coverage** shows the full picture by merging both coverage sources.
### Current Exclusions from Jest
```javascript
// From jest.config.js - These ARE tested by E2E:
'!src/lib/api/hooks/**', // React Query hooks
'!src/app/**/layout.tsx', // Next.js layouts
'!src/app/**/error.tsx', // Error boundaries
'!src/app/**/loading.tsx', // Loading states
```
### Expected Results
```
Unit test coverage: 97.19% (excluding above)
E2E coverage: ~25-35% (user flows + excluded files)
Combined coverage: 98-100% ✅
```
---
## Quick Start
### Prerequisites
All infrastructure is already created! Just need dependencies:
```bash
# Option 1: V8 Coverage (Chromium only, no instrumentation)
npm install -D v8-to-istanbul istanbul-lib-coverage istanbul-lib-report istanbul-reports
# Option 2: Istanbul Instrumentation (all browsers)
npm install -D @istanbuljs/nyc-config-typescript babel-plugin-istanbul nyc \
istanbul-lib-coverage istanbul-lib-report istanbul-reports
```
### Add Package Scripts
Add to `package.json`:
```json
{
"scripts": {
"coverage:convert": "tsx scripts/convert-v8-to-istanbul.ts",
"coverage:merge": "tsx scripts/merge-coverage.ts",
"coverage:combined": "npm run test:coverage && E2E_COVERAGE=true npm run test:e2e && npm run coverage:convert && npm run coverage:merge",
"coverage:view": "open coverage-combined/index.html"
}
}
```
### Run Combined Coverage
```bash
# Full workflow (unit + E2E + merge)
npm run coverage:combined
# View HTML report
npm run coverage:view
```
---
## Approach 1: V8 Coverage (Recommended)
### Pros & Cons
**Pros:**
- ✅ Native browser coverage (most accurate)
- ✅ No build instrumentation needed (faster)
- ✅ Works with source maps
- ✅ Zero performance overhead
**Cons:**
- ❌ Chromium only (V8 engine specific)
- ❌ Requires v8-to-istanbul conversion
### Setup Steps
#### 1. Install Dependencies
```bash
npm install -D v8-to-istanbul istanbul-lib-coverage istanbul-lib-report istanbul-reports
```
#### 2. Integrate into E2E Tests
Update your E2E test files to use coverage helpers:
```typescript
// e2e/homepage.spec.ts
import { test, expect } from '@playwright/test';
import { withCoverage } from './helpers/coverage';
test.describe('Homepage Tests', () => {
test.beforeEach(async ({ page }) => {
// Start coverage collection
await withCoverage.start(page);
await page.goto('/');
});
test.afterEach(async ({ page }, testInfo) => {
// Stop and save coverage
await withCoverage.stop(page, testInfo.title);
});
test('displays header', async ({ page }) => {
await expect(page.getByRole('heading')).toBeVisible();
});
});
```
#### 3. Run E2E Tests with Coverage
```bash
E2E_COVERAGE=true npm run test:e2e
```
This generates: `coverage-e2e/raw/*.json` (V8 format)
#### 4. Convert V8 to Istanbul
```bash
npm run coverage:convert
```
This converts to: `coverage-e2e/.nyc_output/e2e-coverage.json` (Istanbul format)
#### 5. Merge with Jest Coverage
```bash
npm run coverage:merge
```
This generates: `coverage-combined/index.html`
---
## Approach 2: Istanbul Instrumentation
### Pros & Cons
**Pros:**
- ✅ Works on all browsers (Firefox, Safari, etc.)
- ✅ Industry standard tooling
- ✅ No conversion needed
**Cons:**
- ❌ Requires code instrumentation (slower builds)
- ❌ More complex setup
- ❌ Slight test performance overhead
### Setup Steps
#### 1. Install Dependencies
```bash
npm install -D @istanbuljs/nyc-config-typescript babel-plugin-istanbul \
nyc istanbul-lib-coverage istanbul-lib-report istanbul-reports \
@babel/core babel-loader
```
#### 2. Configure Babel Instrumentation
Create `.babelrc.js`:
```javascript
module.exports = {
presets: ['next/babel'],
env: {
test: {
plugins: [
process.env.E2E_COVERAGE && 'istanbul'
].filter(Boolean)
}
}
};
```
#### 3. Configure Next.js Webpack
Update `next.config.js`:
```javascript
const nextConfig = {
webpack: (config, { isServer }) => {
// Add Istanbul instrumentation in E2E coverage mode
if (process.env.E2E_COVERAGE && !isServer) {
config.module.rules.push({
test: /\.(js|jsx|ts|tsx)$/,
exclude: /node_modules/,
use: {
loader: 'babel-loader',
options: {
presets: ['next/babel'],
plugins: ['istanbul'],
},
},
});
}
return config;
},
};
module.exports = nextConfig;
```
#### 4. Integrate into E2E Tests
Use the Istanbul helper instead:
```typescript
import { test, expect } from '@playwright/test';
import { saveIstanbulCoverage } from './helpers/coverage';
test.describe('Homepage Tests', () => {
test.afterEach(async ({ page }, testInfo) => {
await saveIstanbulCoverage(page, testInfo.title);
});
test('my test', async ({ page }) => {
await page.goto('/');
// Test code...
});
});
```
#### 5. Run Tests
```bash
# Start dev server with instrumentation
E2E_COVERAGE=true npm run dev
# In another terminal, run E2E tests
E2E_COVERAGE=true npm run test:e2e
```
#### 6. Merge Coverage
```bash
npm run coverage:merge
```
No conversion step needed! Istanbul coverage goes directly to `.nyc_output/`.
---
## Combined Coverage Workflow
### Full Workflow Diagram
```
┌─────────────────────┐
│ Jest Unit Tests │
│ npm run test:cov │
└──────────┬──────────┘
v
coverage/coverage-final.json
├─────────────────────┐
│ │
v v
┌─────────────────┐ ┌──────────────────┐
│ E2E Tests │ │ E2E Tests │
│ (V8 Coverage) │ │ (Istanbul) │
└────────┬────────┘ └────────┬─────────┘
│ │
v v
coverage-e2e/raw/*.json coverage-e2e/.nyc_output/*.json
│ │
v │
scripts/convert-v8-to-istanbul.ts
│ │
v │
coverage-e2e/.nyc_output/e2e-coverage.json
│ │
└──────────┬──────────┘
v
scripts/merge-coverage.ts
v
coverage-combined/
├── index.html
├── lcov.info
└── coverage-final.json
```
### Commands Summary
```bash
# 1. Run unit tests with coverage
npm run test:coverage
# 2. Run E2E tests with coverage
E2E_COVERAGE=true npm run test:e2e
# 3. Convert V8 to Istanbul (if using V8 approach)
npm run coverage:convert
# 4. Merge all coverage
npm run coverage:merge
# 5. View combined report
npm run coverage:view
# OR: Do all at once
npm run coverage:combined
```
---
## Integration Steps
### Phase 1: Pilot Integration (Single Test File)
Start with one E2E test file to verify the setup:
**File: `e2e/homepage.spec.ts`**
```typescript
import { test, expect } from '@playwright/test';
import { withCoverage } from './helpers/coverage';
test.describe('Homepage - Desktop Navigation', () => {
test.beforeEach(async ({ page }) => {
await withCoverage.start(page);
await page.goto('/');
await page.waitForLoadState('networkidle');
});
test.afterEach(async ({ page }, testInfo) => {
await withCoverage.stop(page, testInfo.title);
});
test('should display header with logo and navigation', async ({ page }) => {
await expect(page.getByRole('link', { name: /FastNext/i })).toBeVisible();
await expect(page.getByRole('link', { name: 'Components' })).toBeVisible();
});
});
```
**Test the pilot:**
```bash
# Install dependencies
npm install -D v8-to-istanbul istanbul-lib-coverage istanbul-lib-report istanbul-reports
# Run single test with coverage
E2E_COVERAGE=true npx playwright test homepage.spec.ts
# Verify coverage files created
ls coverage-e2e/raw/
# Convert and merge
npm run coverage:convert
npm run coverage:merge
# Check results
npm run coverage:view
```
### Phase 2: Rollout to All Tests
Once pilot works, update all 15 E2E spec files:
**Automated rollout script:**
```bash
# Create a helper script: scripts/add-coverage-to-tests.sh
#!/bin/bash
for file in e2e/*.spec.ts; do
# Add import at top (if not already present)
if ! grep -q "import.*coverage" "$file"; then
sed -i "1i import { withCoverage } from './helpers/coverage';" "$file"
fi
# Add beforeEach hook (manual review recommended)
echo "Updated: $file"
done
```
**Or manually add to each file:**
1. Import coverage helper
2. Add `beforeEach` with `withCoverage.start(page)`
3. Add `afterEach` with `withCoverage.stop(page, testInfo.title)`
### Phase 3: CI/CD Integration
Add to your CI pipeline (e.g., `.github/workflows/test.yml`):
```yaml
- name: Run E2E tests with coverage
run: E2E_COVERAGE=true npm run test:e2e
- name: Convert E2E coverage
run: npm run coverage:convert
- name: Merge coverage
run: npm run coverage:merge
- name: Upload combined coverage
uses: codecov/codecov-action@v3
with:
files: ./coverage-combined/lcov.info
flags: combined
```
---
## Troubleshooting
### Problem: No coverage files generated
**Symptoms:**
```bash
npm run coverage:convert
# ❌ No V8 coverage found at: coverage-e2e/raw
```
**Solutions:**
1. Verify `E2E_COVERAGE=true` is set when running tests
2. Check coverage helpers are imported: `import { withCoverage } from './helpers/coverage'`
3. Verify `beforeEach` and `afterEach` hooks are added
4. Check browser console for errors during test run
### Problem: V8 conversion fails
**Symptoms:**
```bash
npm run coverage:convert
# ❌ v8-to-istanbul not installed
```
**Solution:**
```bash
npm install -D v8-to-istanbul
```
### Problem: Coverage lower than expected
**Symptoms:**
```
Combined: 85% (expected 99%)
```
**Causes & Solutions:**
1. **E2E tests don't trigger all code paths**
- Check which files are E2E-only: `npm run coverage:merge` shows breakdown
- Add more E2E tests for uncovered scenarios
2. **Source maps not working**
- Verify Next.js generates source maps: check `next.config.js`
- Istanbul needs source maps to map coverage back to source
3. **Wrong files included**
- Check `.nycrc.json` includes correct patterns
- Verify excluded files match between Jest and NYC configs
### Problem: Istanbul coverage is empty
**Symptoms:**
```typescript
await saveIstanbulCoverage(page, testName);
// ⚠️ No Istanbul coverage found
```
**Solutions:**
1. Verify `babel-plugin-istanbul` is configured
2. Check `window.__coverage__` exists:
```typescript
const hasCoverage = await page.evaluate(() => !!(window as any).__coverage__);
console.log('Istanbul available:', hasCoverage);
```
3. Ensure dev server started with `E2E_COVERAGE=true npm run dev`
### Problem: Merge script fails
**Symptoms:**
```bash
npm run coverage:merge
# ❌ Error: Cannot find module 'istanbul-lib-coverage'
```
**Solution:**
```bash
npm install -D istanbul-lib-coverage istanbul-lib-report istanbul-reports
```
---
## FAQ
### Q: Should I use V8 or Istanbul coverage?
**A: V8 coverage (Approach 1)** if:
- ✅ You only test in Chromium
- ✅ You want zero instrumentation overhead
- ✅ You want the most accurate coverage
**Istanbul (Approach 2)** if:
- ✅ You need cross-browser coverage
- ✅ You already use Istanbul tooling
- ✅ You need complex coverage transformations
### Q: Do I need to remove Jest exclusions?
**A: No!** Keep them. The `.nycrc.json` config handles combined coverage independently.
### Q: Will this slow down my tests?
**V8 Approach:** Minimal overhead (~5% slower)
**Istanbul Approach:** Moderate overhead (~15-20% slower due to instrumentation)
### Q: Can I run coverage only for specific tests?
**Yes:**
```bash
# Single file
E2E_COVERAGE=true npx playwright test homepage.spec.ts
# Specific describe block
E2E_COVERAGE=true npx playwright test --grep "Mobile Menu"
```
### Q: How do I exclude files from E2E coverage?
Edit `.nycrc.json` and add to `exclude` array:
```json
{
"exclude": [
"src/app/dev/**",
"src/lib/utils/debug.ts"
]
}
```
### Q: Can I see which lines are covered by E2E vs Unit tests?
Not directly in the HTML report, but you can:
1. Generate separate reports:
```bash
npx nyc report --reporter=html --report-dir=coverage-unit --temp-dir=coverage/.nyc_output
npx nyc report --reporter=html --report-dir=coverage-e2e-only --temp-dir=coverage-e2e/.nyc_output
```
2. Compare the two reports to see differences
### Q: What's the performance impact on CI?
Typical impact:
- V8 coverage: +2-3 minutes (conversion time)
- Istanbul coverage: +5-7 minutes (build instrumentation)
- Merge step: ~10 seconds
Total CI time increase: **3-8 minutes**
---
## Next Steps
### After Phase 1 (Infrastructure - DONE ✅)
You've completed:
- ✅ `.nycrc.json` configuration
- ✅ Merge script (`scripts/merge-coverage.ts`)
- ✅ Conversion script (`scripts/convert-v8-to-istanbul.ts`)
- ✅ Coverage helpers (`e2e/helpers/coverage.ts`)
- ✅ This documentation
### Phase 2: Activation (When Ready)
1. **Install dependencies:**
```bash
npm install -D v8-to-istanbul istanbul-lib-coverage istanbul-lib-report istanbul-reports
```
2. **Add package.json scripts** (see Quick Start)
3. **Test with one E2E file** (homepage.spec.ts recommended)
4. **Rollout to all E2E tests**
5. **Add to CI/CD pipeline**
### Expected Timeline
- **Phase 1:** ✅ Done (non-disruptive infrastructure)
- **Phase 2:** ~1-2 hours (pilot + dependency installation)
- **Rollout:** ~30 minutes (add hooks to 15 test files)
- **CI integration:** ~20 minutes
---
## Additional Resources
- [Istanbul Coverage](https://istanbul.js.org/)
- [NYC Configuration](https://github.com/istanbuljs/nyc#configuration-files)
- [Playwright Coverage](https://playwright.dev/docs/api/class-coverage)
- [V8 to Istanbul](https://github.com/istanbuljs/v8-to-istanbul)
---
**Questions or issues?** Check troubleshooting section or review the example in `e2e/helpers/coverage.ts`.

View File

@@ -0,0 +1,285 @@
# E2E Test Performance Optimization Plan
**Current State**: 230 tests, ~2100 seconds total execution time
**Target**: Reduce to <900 seconds (60% improvement)
## Bottleneck Analysis
### 1. Authentication Overhead (HIGHEST IMPACT)
**Problem**: Each test logs in fresh via UI
- **Impact**: 5-7s per test × 133 admin tests = ~700s wasted
- **Root Cause**: Using `loginViaUI(page)` in every `beforeEach`
**Example of current slow pattern:**
```typescript
test.beforeEach(async ({ page }) => {
await setupSuperuserMocks(page);
await loginViaUI(page); // ← 5-7s UI login EVERY test
await page.goto('/admin');
});
```
**Solution: Playwright Storage State** (SAVE ~600-700s)
```typescript
// auth.setup.ts - Run ONCE per worker
import { test as setup } from '@playwright/test';
setup('authenticate as admin', async ({ page }) => {
await setupSuperuserMocks(page);
await loginViaUI(page);
await page.context().storageState({ path: 'e2e/.auth/admin.json' });
});
setup('authenticate as regular user', async ({ page }) => {
await setupAuthenticatedMocks(page);
await loginViaUI(page);
await page.context().storageState({ path: 'e2e/.auth/user.json' });
});
```
```typescript
// playwright.config.ts
export default defineConfig({
projects: [
{ name: 'setup', testMatch: /.*\.setup\.ts/ },
{
name: 'admin tests',
use: { storageState: 'e2e/.auth/admin.json' },
dependencies: ['setup'],
testMatch: /admin-.*\.spec\.ts/,
},
{
name: 'user tests',
use: { storageState: 'e2e/.auth/user.json' },
dependencies: ['setup'],
testMatch: /settings-.*\.spec\.ts/,
},
],
});
```
```typescript
// admin-users.spec.ts - NO MORE loginViaUI!
test.beforeEach(async ({ page }) => {
// Auth already loaded from storageState
await page.goto('/admin/users'); // ← Direct navigation, ~1-2s
});
```
**Expected Improvement**: 5-6s → 0.5-1s per test = **~600s saved** (133 tests × 5s)
---
### 2. Redundant Navigation Tests (MEDIUM IMPACT)
**Problem**: Separate tests for "navigate to X" and "display X page"
- **Impact**: 3-5s per redundant test × ~15 tests = ~60s wasted
**Current slow pattern:**
```typescript
test('should navigate to users page', async ({ page }) => {
await page.goto('/admin/users'); // 3s
await expect(page).toHaveURL('/admin/users');
await expect(page.locator('h1')).toContainText('User Management');
});
test('should display user management page', async ({ page }) => {
await page.goto('/admin/users'); // 3s DUPLICATE
await expect(page.locator('h1')).toContainText('User Management');
await expect(page.getByText(/manage users/i)).toBeVisible();
});
```
**Optimized pattern:**
```typescript
test('should navigate to users page and display content', async ({ page }) => {
await page.goto('/admin/users'); // 3s ONCE
// Navigation assertions
await expect(page).toHaveURL('/admin/users');
// Content assertions
await expect(page.locator('h1')).toContainText('User Management');
await expect(page.getByText(/manage users/i)).toBeVisible();
await expect(page.getByRole('button', { name: 'Create User' })).toBeVisible();
});
```
**Expected Improvement**: **~45-60s saved** (15 tests eliminated)
---
### 3. Flaky Test Fix (CRITICAL)
**Problem**: Test #218 failed once, passed on retry
```
Test: settings-password.spec.ts:24:7 Password Change should display password change form
Failed: 12.8s → Retry passed: 8.3s
```
**Root Cause Options**:
1. Race condition in form rendering
2. Slow network request not properly awaited
3. Animation/transition timing issue
**Investigation needed:**
```typescript
// Current test (lines 24-35)
test('should display password change form', async ({ page }) => {
await page.goto('/settings/password');
// ← Likely missing waitForLoadState or explicit wait
await expect(page.getByLabel(/current password/i)).toBeVisible();
await expect(page.getByLabel(/new password/i)).toBeVisible();
await expect(page.getByLabel(/confirm password/i)).toBeVisible();
});
```
**Temporary Solution: Skip until fixed**
```typescript
test.skip('should display password change form', async ({ page }) => {
// TODO: Fix race condition (issue #XXX)
await page.goto('/settings/password');
await page.waitForLoadState('networkidle'); // ← Add this
await expect(page.getByLabel(/current password/i)).toBeVisible();
});
```
**Expected Improvement**: Eliminate retry overhead + improve reliability
---
### 4. Optimize Wait Timeouts (LOW IMPACT)
**Problem**: Default timeout is 10s for all assertions
- **Impact**: Tests wait unnecessarily when elements load faster
**Current global timeout:**
```typescript
// playwright.config.ts
export default defineConfig({
timeout: 30000, // Per test
expect: { timeout: 10000 }, // Per assertion
});
```
**Optimized for fast-loading pages:**
```typescript
export default defineConfig({
timeout: 20000, // Reduce from 30s
expect: { timeout: 5000 }, // Reduce from 10s (most elements load <2s)
});
```
**Expected Improvement**: **~100-150s saved** (faster failures, less waiting)
---
## Implementation Priority
### Phase 1: Quick Wins (1-2 hours work)
1.**Skip flaky test #218** temporarily
2.**Reduce timeout defaults** (5s for expects, 20s for tests)
3.**Combine 5 most obvious redundant navigation tests**
**Expected savings**: ~100-150s (5-7% improvement)
---
### Phase 2: Auth State Caching (2-4 hours work)
1. ✅ Create `e2e/auth.setup.ts` with storage state setup
2. ✅ Update `playwright.config.ts` with projects + dependencies
3. ✅ Remove `loginViaUI` from all admin test `beforeEach` hooks
4. ✅ Update auth helper to support both mock + storageState modes
**Expected savings**: ~600-700s (30-35% improvement)
---
### Phase 3: Deep Optimization (4-8 hours work)
1. ✅ Investigate and fix flaky test root cause
2. ✅ Audit all navigation tests for redundancy
3. ✅ Combine related assertions (e.g., all stat cards in one test)
4. ✅ Profile slowest 10 tests individually
**Expected savings**: ~150-200s (7-10% improvement)
---
## Total Expected Improvement
| Phase | Time Investment | Time Saved | % Improvement |
|-------|----------------|------------|---------------|
| Phase 1 | 1-2 hours | ~150s | 7% |
| Phase 2 | 2-4 hours | ~700s | 35% |
| Phase 3 | 4-8 hours | ~200s | 10% |
| **Total** | **7-14 hours** | **~1050s** | **50-60%** |
**Final target**: 2100s → 1050s = **~17-18 minutes** (currently ~35 minutes)
---
## Risks and Considerations
### Storage State Caching Risks:
1. **Test isolation**: Shared auth state could cause cross-test pollution
- **Mitigation**: Use separate storage files per role, clear cookies between tests
2. **Stale auth tokens**: Mock tokens might expire
- **Mitigation**: Use long-lived test tokens (24h expiry)
3. **Debugging difficulty**: Harder to debug auth issues
- **Mitigation**: Keep `loginViaUI` tests for auth flow verification
### Recommended Safeguards:
```typescript
// Clear non-auth state between tests
test.beforeEach(async ({ page }) => {
await page.goto('/admin');
await page.evaluate(() => {
// Clear localStorage except auth tokens
const tokens = {
access_token: localStorage.getItem('access_token'),
refresh_token: localStorage.getItem('refresh_token'),
};
localStorage.clear();
if (tokens.access_token) localStorage.setItem('access_token', tokens.access_token);
if (tokens.refresh_token) localStorage.setItem('refresh_token', tokens.refresh_token);
});
});
```
---
## Next Steps
**Immediate Actions (Do Now):**
1. Skip flaky test #218 with TODO comment
2. Reduce timeout defaults in playwright.config.ts
3. Create this optimization plan issue/ticket
**Short-term (This Week):**
1. Implement auth storage state (Phase 2)
2. Combine obvious redundant tests (Phase 1)
**Medium-term (Next Sprint):**
1. Investigate flaky test root cause
2. Audit all tests for redundancy
3. Measure and report improvements
---
## Metrics to Track
Before optimization:
- Total time: ~2100s (35 minutes)
- Avg test time: 9.1s
- Slowest test: 20.1s (settings navigation)
- Flaky tests: 1
After Phase 1+2 target:
- Total time: <1200s (20 minutes) ✅
- Avg test time: <5.5s ✅
- Slowest test: <12s ✅
- Flaky tests: 0 ✅
After Phase 3 target:
- Total time: <1050s (17 minutes) 🎯
- Avg test time: <4.8s 🎯
- Slowest test: <10s 🎯
- Flaky tests: 0 🎯