Playwright Test Failures - Executive Summary
Date: 2025-11-30 Status: BLOCKED - Tests Cannot Validate Pre-Merge UI Changes Severity: HIGH Time to Fix: 30 minutes Owner: DevOps/QA Team
The Problem in One Sentence
Playwright E2E tests are configured to run against a local development server (localhost:3000) but CI needs to test the deployed production app (https://momentum.cloudnnj.com), causing all tests to fail.
Impact
Current State
- ❌ E2E tests fail in CI on every PR
- ❌ Cannot validate UI changes before merging to main
- ❌ Risk of shipping broken UI to production
- ❌ No automated testing of deployed environment
- ❌ Manual testing required for all UI changes
Desired State
- ✅ E2E tests pass in CI
- ✅ Automated pre-merge UI validation
- ✅ Confidence in deployed code quality
- ✅ Faster development cycle
- ✅ Reduced manual QA burden
Root Cause Analysis
Configuration Issues
| Issue | Current | Required | Priority |
|---|---|---|---|
| Base URL | http://localhost:3000 |
https://momentum.cloudnnj.com |
CRITICAL |
| Web Server | Tries to start local dev server | Should test deployed app | CRITICAL |
| Test Users | Expects users that don’t exist | Auto-provision before tests | HIGH |
| Environment Vars | Not configured in CI | Need AWS/Cognito secrets | HIGH |
| Test Data | Expects mock data | Need database seeding | MEDIUM |
| URL Patterns | Hardcoded localhost URLs | Use relative paths | MEDIUM |
Architecture Mismatch
┌─────────────────────────────────────────────────────────────┐
│ CURRENT (BROKEN) │
├─────────────────────────────────────────────────────────────┤
│ │
│ GitHub Actions CI │
│ │ │
│ ├─> Starts npm run dev (local server) │
│ │ │
│ ├─> Playwright tests │
│ │ └─> Try to connect to localhost:3000 │
│ │ └─> ❌ FAILS (no deployed app to test) │
│ │ │
│ └─> Deployed app at momentum.cloudnnj.com │
│ └─> ⚠️ Never tested! │
│ │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ DESIRED (FIXED) │
├─────────────────────────────────────────────────────────────┤
│ │
│ GitHub Actions CI │
│ │ │
│ ├─> Provision test users in Cognito │
│ │ └─> ✅ admin@momentum.test created │
│ │ └─> ✅ user@momentum.test created │
│ │ │
│ ├─> Playwright tests │
│ │ └─> Connect to https://momentum.cloudnnj.com │
│ │ └─> ✅ Tests real deployed app │
│ │ └─> ✅ Tests with real Cognito auth │
│ │ └─> ✅ Tests with real database │
│ │ │
│ └─> Test results published to PR │
│ └─> ✅ Team sees pass/fail status │
│ │
└─────────────────────────────────────────────────────────────┘
Solution Overview
3-Step Fix
1. Configure CI Environment (5 min)
- Add GitHub secrets for AWS/Cognito access
- Update workflow to set
BASE_URL=https://momentum.cloudnnj.com
2. Provision Test Users (10 min)
- Add user provisioning step to CI workflow
- Creates
admin@momentum.testanduser@momentum.test
3. Update Test Files (15 min)
- Change hardcoded URLs from
http://localhost:3000/to/ - Tests use
baseURLfrom config (environment-agnostic)
Files to Modify
.github/workflows/playwright.yml ← Add user provisioning + BASE_URL
tests/e2e/*.spec.ts ← Change localhost to relative URLs
Files Already Created (Ready to Use)
✅ scripts/setup/provision-test-users.sh ← User provisioning script
✅ scripts/database/seed-test-data.mjs ← Database seeding (optional)
✅ playwright.config.ci.ts ← CI-specific config
✅ docs/playwright-test-analysis.md ← Full technical analysis
✅ docs/PLAYWRIGHT_FIXES_QUICKSTART.md ← Step-by-step guide
Implementation Plan
Phase 1: Quick Fix (Week 1) - PRIORITY
Goal: Get tests passing in CI
Tasks:
- Add GitHub secrets (AWS credentials, Cognito pool ID)
- Update
.github/workflows/playwright.yml - Update test files to use relative URLs
- Run test in CI and verify success
Deliverable: Tests pass in CI, PR shows test results
Time: 30 minutes implementation + 1 hour validation
Phase 2: Stabilization (Week 2) - RECOMMENDED
Goal: Make tests reliable and maintainable
Tasks:
- Add database test data seeding
- Implement Page Object Models
- Add test result notifications
- Document testing guidelines
Deliverable: <5% flaky test rate
Time: 2-3 days
Phase 3: Enhancement (Week 3-4) - OPTIONAL
Goal: Comprehensive test coverage
Tasks:
- Add authentication state caching
- Parallelize independent tests
- Add visual regression testing
- Optimize test execution time
Deliverable: <10 minute test runs, 90%+ coverage
Time: 1 week
Risk Assessment
Risks of NOT Fixing
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Ship broken UI to production | HIGH | CRITICAL | Manual QA (slow, error-prone) |
| Regression bugs in production | MEDIUM | HIGH | Hope users don’t notice |
| Slower development velocity | HIGH | MEDIUM | Accept slower releases |
| Team frustration | HIGH | MEDIUM | N/A |
Risks of Fixing
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Tests still flaky after fix | MEDIUM | LOW | Increase retries, add timeouts |
| AWS costs increase | LOW | LOW | Minimal - only test user creation |
| Implementation takes longer | LOW | LOW | Follow quickstart guide |
Success Metrics
Immediate (Week 1)
- ✅ 50%+ of tests pass in CI
- ✅ Test results visible in PRs
- ✅ No manual intervention needed
Short-term (Week 2-4)
- ✅ 90%+ of tests pass consistently
- ✅ <5% flaky test rate
- ✅ Test execution time <15 minutes
Long-term (Month 2+)
- ✅ 95%+ test pass rate
- ✅ Zero manual QA for standard features
- ✅ Automated deployment confidence
Resource Requirements
People
- 1 Developer: 4 hours (Phase 1)
- 1 DevOps: 2 hours (setup AWS credentials)
- 1 QA: 4 hours (validation, Phase 2)
Infrastructure
- AWS Cognito User Pool: $0 (existing)
- AWS RDS Aurora: $0 (existing, just add test data)
- GitHub Actions minutes: ~5 min/run (existing quota)
Cost Impact
- One-time: 0 hours (scripts already created)
- Recurring: +5 min per PR (test execution)
- Savings: -2 hours manual QA per PR
ROI: Positive after 3 PRs (6 hrs saved vs 4 hrs investment)
Decision Required
Option A: Implement Quick Fix Now (RECOMMENDED)
- Time: 30 minutes
- Risk: Low
- Benefit: Immediate test validation
- Next Step: Follow PLAYWRIGHT_FIXES_QUICKSTART.md
Option B: Full Implementation (Phases 1-3)
- Time: 2 weeks
- Risk: Low
- Benefit: Comprehensive testing infrastructure
- Next Step: Assign to QA team, follow project plan
Option C: Do Nothing
- Time: 0
- Risk: HIGH (shipping broken code)
- Benefit: None
- Next Step: Continue manual QA (not recommended)
Recommended Action
Implement Option A immediately (Quick Fix), then schedule Option B (Full Implementation) for next sprint.
Next Steps (Prioritized)
- NOW (5 min): Read
/docs/PLAYWRIGHT_FIXES_QUICKSTART.md - TODAY (30 min): Implement quick fix
- THIS WEEK (1 hr): Validate tests pass in CI
- NEXT SPRINT: Plan Phase 2 (stabilization)
Key Contacts
Implementation Questions: See /docs/PLAYWRIGHT_FIXES_QUICKSTART.md
Technical Details: See /docs/playwright-test-analysis.md
Test Credentials: See /docs/TEST_CREDENTIALS.md
Support Channel: #momentum-dev (Slack)
Appendix: Test Coverage Breakdown
Test Files (11 total)
| Test File | Status | Depends On | Priority |
|---|---|---|---|
home.spec.ts |
✅ Should pass immediately | None | HIGH |
navigation.spec.ts |
✅ Should pass immediately | None | HIGH |
courses.spec.ts |
⚠️ Needs test data | Database | HIGH |
auth-user.spec.ts |
⚠️ Needs test users | Cognito | HIGH |
auth-admin.spec.ts |
⚠️ Needs test users | Cognito | HIGH |
dashboard.spec.ts |
⚠️ Needs enrollment | Database + Auth | MEDIUM |
course-detail.spec.ts |
⚠️ Needs courses | Database | MEDIUM |
lesson-detail.spec.ts |
⚠️ Needs lessons | Database | MEDIUM |
enrollment.spec.ts |
❌ Needs payment | Stripe + Database | LOW |
admin-panel.spec.ts |
⚠️ Needs admin user | Cognito | LOW |
example.spec.ts |
✅ Should pass | None | LOW |
Legend:
- ✅ Ready to pass after quick fix
- ⚠️ Needs additional setup (user provisioning or data seeding)
- ❌ Needs external services (Stripe integration)
Summary: Fix is straightforward, low-risk, and high-value. Recommend immediate implementation of Phase 1 (30 minutes) to unblock pre-merge UI validation.