cardosofelipe/pragma-stack
Template
1
0
Fork 1
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

Update coverage report with resolved tracking issue and 88% overall coverage

Browse Source 
Resolved `pytest-cov` tracking for async routes by adjusting `.coveragerc` to include `greenlet` concurrency. Coverage improved from 79% to 88%, with significant gains across key modules like `admin.py` (46% → 98%). Updated details on coverage gaps and priorities for reaching 95% target.
This commit is contained in:
Felipe Cardoso
2025-11-02 05:27:24 +01:00
parent 1aab73cb72
commit 406c3bcc82
1 changed files with 163 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files

238
backend/docs/COVERAGE_REPORT.md
View File

@@ -1,56 +1,96 @@
# Test Coverage Analysis Report # Test Coverage Analysis Report
**Date**: 2025-11-01 **Date**: 2025-11-02 (Updated)
**Current Coverage**: 79% (1,932/2,439 lines) **Current Coverage**: 88% (2,157/2,455 lines)
**Previous Coverage**: 79% (1,932/2,439 lines)
**Target Coverage**: 95% **Target Coverage**: 95%
**Gap**: 270 lines needed to reach 90%, ~390 lines for 95% **Gap**: ~175 lines needed to reach 95%
## Executive Summary ## Executive Summary
This report documents the current state of test coverage, identified issues with coverage tracking, and actionable paths to reach the 95% coverage target. This report documents the **successful resolution** of the coverage tracking issue and the path to reach the 95% coverage target.
### Current Status ### Current Status
- **Total Tests**: 596 passing - **Total Tests**: 598 passing ✅
- **Overall Coverage**: 79% - **Overall Coverage**: 88% (up from 79%)
- **Lines Covered**: 1,932 / 2,439 - **Lines Covered**: 2,157 / 2,455
- **Lines Missing**: 507 - **Lines Missing**: 298 (down from 507)
- **Improvement**: +9 percentage points (+225 lines covered)
### Key Finding: Coverage Tracking Issue ### ✅ RESOLVED: Coverage Tracking Issue
**Critical Issue Identified**: Pytest-cov is not properly recording coverage for FastAPI route files when tests are executed, despite: **Problem**: Pytest-cov was not properly recording coverage for FastAPI route files executed through httpx's `ASGITransport`, despite:
1. Tests passing successfully (596/596 ✓) 1. Tests passing successfully (598/598 ✓)
2. Manual verification showing code paths ARE being executed 2. Manual verification showing code paths ARE being executed
3. Correct responses being returned from endpoints 3. Correct responses being returned from endpoints
**Root Cause**: Suspected interaction between pytest-cov, pytest-xdist (parallel execution), and the FastAPI async test client causing coverage data to not be collected for certain modules. **Root Cause Identified**: Coverage.py was not configured to track async code execution through ASGI transport's greenlet-based concurrency model.
**Evidence**: **Solution**: Added `concurrency = thread,greenlet` to `.coveragerc`
```bash
# Running with xdist shows "Module was never imported" warning ```ini
pytest --cov=app/api/routes/admin --cov-report=term-missing [run]
# Warning: Module app/api/routes/admin was never imported source = app
# Warning: No data was collected concurrency = thread,greenlet # ← THIS WAS THE FIX!
omit = ...
``` ```
## Detailed Coverage Breakdown **Results After Fix**:
- **admin.py**: 46% → **98%** (+52 points!)
- **auth.py**: 79% → **95%** (+16 points)
- **sessions.py**: 49% → **84%** (+35 points)
- **users.py**: 60% → **93%** (+33 points)
- **Overall**: 79% → **88%** (+9 points)
### Files with Complete Coverage (100%) ✓ ## Detailed Coverage Breakdown (Post-Fix)
- `app/crud/session.py`
- `app/utils/security.py`
- `app/schemas/sessions.py`
- `app/utils/device.py` (97%)
- 12 other files with 100% coverage
### Files Requiring Coverage Improvement ### Files with Excellent Coverage (95%+) ✅
- **app/crud/session.py**: 100%
- **app/utils/security.py**: 100%
- **app/schemas/sessions.py**: 100%
- **app/schemas/errors.py**: 100%
- **app/services/email_service.py**: 100%
- **app/services/session_cleanup.py**: 100%
- **app/api/main.py**: 100%
- **app/api/routes/admin.py**: **98%** (was 46%!)
- **app/core/config.py**: 98%
- **app/schemas/common.py**: 97%
- **app/utils/device.py**: 97%
- **app/auth.py**: 95%
- **app/core/exceptions.py**: 95%
#### 1. **app/api/routes/admin.py** - Priority: HIGH ### Files Requiring Coverage Improvement (to reach 95%)
- **Coverage**: 46% (118/259 lines)
- **Missing Lines**: 141 #### 1. **app/api/routes/organizations.py** - Priority: CRITICAL ⚠️
- **Impact**: Largest single coverage gap - **Coverage**: 35% (23/66 lines)
- **Missing Lines**: 43
- **Impact**: Largest remaining gap, NO TESTS EXIST
**Missing Coverage Areas**: **Missing Coverage Areas**:
``` ```
Lines 109-116 : Pagination metadata creation (list users) Lines 54-83 : List organizations endpoint (entire function)
Lines 103-128 : Get organization by ID (entire function)
Lines 150-172 : Add member to organization (entire function)
Lines 193-221 : Remove member from organization (entire function)
```
**Required Tests**: Create `tests/api/test_organizations.py` with ~12-15 tests
---
#### 2. **app/crud/base.py** - Priority: HIGH
- **Coverage**: 73% (164/224 lines)
- **Missing Lines**: 60
- **Impact**: Foundation class for all CRUD operations
**Missing Coverage Areas**:
```
Lines 77-78 : Exception handling in get()
Lines 119-120 : Exception handling in get_multi()
Lines 130-152 : Advanced filtering logic in get_multi()
Lines 254-296 : Pagination, sorting, filtering in get_multi_with_total()
Lines 342-343 : Exception handling in update()
Lines 383-384 : Exception handling in remove()
Lines 143-144 : User creation success logging Lines 143-144 : User creation success logging
Lines 146-147 : User creation error handling (ValueError) Lines 146-147 : User creation error handling (ValueError)
Lines 170-175 : Get user NotFoundError Lines 170-175 : Get user NotFoundError
@@ -297,7 +337,36 @@ Lines 170-183 : Password strength validation (length, uppercase, lowercase, di
--- ---
## Path to 95% Coverage ---
## **UPDATED** Path to 95% Coverage (Post-Fix)
### Current State: 88% → Target: 95% (Need to cover ~175 more lines)
**Breakdown by Priority:**
| File | Current | Missing Lines | Priority | Estimated Tests Needed |
|------|---------|---------------|----------|------------------------|
| `organizations.py` (routes) | 35% | 43 | CRITICAL | 12-15 tests |
| `base.py` (crud) | 73% | 60 | HIGH | 15-20 tests |
| `organization.py` (crud) | 80% | 41 | MEDIUM | 12 tests |
| `permissions.py` (deps) | 53% | 20 | MEDIUM | 12-15 tests |
| `main.py` | 80% | 16 | LOW | 5-8 tests |
| `database.py` (core) | 78% | 14 | LOW | 5-8 tests |
| `validators.py` (schemas) | 62% | 10 | LOW | 8-10 tests |
**Quick Win Strategy** (Estimated 15-20 hours):
1. **Phase 1** (5h): Create `tests/api/test_organizations.py` → +43 lines (+1.8%)
2. **Phase 2** (6h): Test base CRUD advanced features → +60 lines (+2.4%)
3. **Phase 3** (4h): Test organization CRUD exceptions → +41 lines (+1.7%)
4. **Phase 4** (3h): Test permission dependencies → +20 lines (+0.8%)
5. **Phase 5** (2h): Misc coverage (validators, database utils) → +20 lines (+0.8%)
**Expected Result**: 88% + 7.5% = **95.5%** ✅
---
## Path to 95% Coverage (Historical - Pre-Fix)
### Recommended Prioritization ### Recommended Prioritization
@@ -396,54 +465,54 @@ Mark initialization and setup code with `# pragma: no cover`.
--- ---
## Critical Action Items ## Critical Action Items (UPDATED)
### Immediate (Do First) ### ✅ Completed
1. ✅ **Investigate coverage tracking issue** - This is blocking accurate measurement 1. ✅ **RESOLVED: Coverage tracking issue** - Added `concurrency = thread,greenlet` to `.coveragerc`
2. ✅ **Generate HTML coverage report** - Visual confirmation of what's actually covered 2. ✅ **Generated HTML coverage report** - Visualized actual vs missing coverage
3. ✅ **Run coverage in single-process mode** - Eliminate xdist as variable 3. ✅ **Ran coverage in single-process mode** - Confirmed xdist was not the issue
4. ✅ **Achieved 88% coverage** - Up from 79% (+9 percentage points)
### High Priority (Do Next) ### High Priority (Path to 95%)
4. ⬜ **Create organization routes tests** - Highest uncovered file (35%) 1. ⬜ **Create organization routes tests** - Highest uncovered file (35%, 43 lines missing)
5. ⬜ **Complete organization CRUD exception tests** - Low-hanging fruit (80% → 95%+) - Estimated: 12-15 tests, 5 hours
6. ⬜ **Test base CRUD advanced features** - Foundation for all CRUD operations - Impact: +1.8% coverage
2. ⬜ **Test base CRUD advanced features** - Foundation for all CRUD operations (73%, 60 lines)
- Estimated: 15-20 tests, 6 hours
- Impact: +2.4% coverage
3. ⬜ **Complete organization CRUD exception tests** - Exception handling (80%, 41 lines)
- Estimated: 12 tests, 4 hours
- Impact: +1.7% coverage
### Medium Priority ### Medium Priority
7. ⬜ **Test permission dependencies thoroughly** - Important for security 4. ⬜ **Test permission dependencies thoroughly** - Security-critical (53%, 20 lines)
8. ⬜ **Complete validator tests** - Data integrity - Estimated: 12-15 tests, 3 hours
- Impact: +0.8% coverage
### Low Priority ### Low Priority
9. ⬜ **Review init_db.py** - Consider excluding setup code 5. ⬜ **Miscellaneous coverage** - Validators, database utils, main.py (~40 lines total)
10. ⬜ **Test auth.py edge cases** - Already 93% - Estimated: 15-20 tests, 2 hours
- Impact: +1.6% coverage
--- ---
## Known Issues and Blockers ## Known Issues and Blockers (UPDATED)
### 1. Coverage Not Being Recorded for Routes ### ✅ RESOLVED: Coverage Not Being Recorded for Routes
**Symptoms**:
- Tests pass: 596/596 ✓
- Endpoints return correct data (manually verified)
- Coverage shows 46% for admin.py despite 20+ tests
**Attempted Solutions**: **Problem**: Coverage.py was not tracking async code execution through httpx's ASGITransport
- ✅ Added tests for all missing line ranges
- ✅ Verified tests execute and pass
- ✅ Manually confirmed endpoints work
- ⬜ Need to investigate pytest-cov configuration
**Hypothesis**: **Solution**: Added `concurrency = thread,greenlet` to `.coveragerc`
- FastAPI async test client may not be compatible with pytest-cov's default tracing
- xdist parallel execution interferes with coverage collection
- Dependency overrides may hide actual route execution from coverage
**Next Steps**: **Result**: Coverage jumped from 79% → 88%, with route files now properly tracked:
1. Run with `-n 0` (single process) - admin.py: 46% → 98%
2. Try `--cov-branch` for branch coverage - auth.py: 79% → 95%
3. Use coverage HTML report to visualize - sessions.py: 49% → 84%
4. Consider using `coverage run -m pytest` directly - users.py: 60% → 93%
### 2. Dead Code in users.py ### Remaining Issue: Dead Code in users.py
**Issue**: Lines 150-154 and 270-275 check for `is_superuser` field in `UserUpdate`, but the schema doesn't include this field. **Issue**: Lines 150-154 and 270-275 check for `is_superuser` field in `UserUpdate`, but the schema doesn't include this field.
**Solution**: ✅ Marked with `# pragma: no cover` **Solution**: ✅ Marked with `# pragma: no cover`
@@ -492,25 +561,44 @@ Mark initialization and setup code with `# pragma: no cover`.
--- ---
## Conclusion ## Conclusion (UPDATED)
Current coverage is **79%** with a path to **93%+** through systematic testing. The primary blocker is the coverage tracking issue with route tests - once resolved, coverage should jump significantly. With focused effort on organization routes, CRUD operations, and permission testing, the 95% goal is achievable within 20-30 hours of dedicated work. ✅ **Coverage tracking issue RESOLVED!** Coverage improved from **79% → 88%** by adding `concurrency = thread,greenlet` to `.coveragerc`.
Current coverage is **88%** with a clear path to **95%+** through systematic testing of:
1. Organization routes (43 lines)
2. Base CRUD advanced features (60 lines)
3. Organization CRUD exceptions (41 lines)
4. Permission dependencies (20 lines)
5. Misc utilities (40 lines)
**Key Success Factors**: **Key Success Factors**:
1. Resolve pytest-cov tracking issue (blocks 5-10% coverage) 1. ✅ **RESOLVED**: pytest-cov tracking issue (+9% coverage)
2. Test organization module (highest gap) 2. Test organization module (highest remaining gap)
3. Exception path testing (low-hanging fruit) 3. Exception path testing (low-hanging fruit)
4. Advanced CRUD feature testing (pagination, filtering, search) 4. Advanced CRUD feature testing (pagination, filtering, search)
**Estimated Timeline to 95%**: **Estimated Timeline to 95%**:
- With coverage fix: 2-3 days of focused work - **15-20 hours of focused work** across 5 phases
- Without coverage fix: 4-5 days (includes investigation) - Can be completed in **2-3 days** with dedicated effort
- Most impactful: Phase 1 (organization routes) and Phase 2 (base CRUD)
--- ---
## References ## References
- Coverage run output: `TOTAL 2439 507 79%` **Original Report** (2025-11-01):
- Coverage: 79% (2,439 statements, 507 missing)
- Test count: 596 passing - Test count: 596 passing
- Tests added this session: 30+ - Issue: Coverage not tracking async routes
- Coverage improvement: 58% → 63% (users.py)
**Updated Report** (2025-11-02):
- Coverage: **88%** (2,455 statements, 298 missing) ✅
- Test count: **598 passing**
- **Fix Applied**: `concurrency = thread,greenlet` in `.coveragerc`
- Coverage improvement: **+9 percentage points (+225 lines)**
- Major improvements:
- admin.py: 46% → 98% (+52 points)
- auth.py: 79% → 95% (+16 points)
- sessions.py: 49% → 84% (+35 points)
- users.py: 60% → 93% (+33 points)