177 lines
8.5 KiB
Markdown
177 lines
8.5 KiB
Markdown
# DocFast QA Test Results - 2026-02-14
|
||
|
||
## Test Session Overview
|
||
Backend changes tested:
|
||
1. Removed "Custom templates" from Pro plan on landing page
|
||
2. Re-added email requirement for free tier signup
|
||
3. Migrated from JSON storage to SQLite
|
||
|
||
## Issues Found
|
||
|
||
### 🔴 CRITICAL: Mobile Horizontal Scrolling
|
||
- **What I tested**: Mobile responsiveness at 375x812 (iPhone X)
|
||
- **Expected**: No horizontal scrolling
|
||
- **Found**: Page has horizontal scrolling (scroll width: 488px vs viewport: 375px)
|
||
- **Impact**: Poor mobile user experience, makes content inaccessible
|
||
|
||
### 🟡 MEDIUM: Rate Limiting Not User-Friendly
|
||
- **What I tested**: Multiple signup attempts
|
||
- **Expected**: Clear indication of rate limiting before user hits it
|
||
- **Found**: No upfront warning about rate limits, users only discover after filling form
|
||
- **Response**: "Too many signup attempts. Please try again in 1 hour."
|
||
- **Improvement**: Show rate limit status or warning before form submission
|
||
|
||
## ✅ TESTS PASSED
|
||
|
||
### Console Errors ✅
|
||
- **Tested**: page.on('pageerror') and page.on('console', type=error)
|
||
- **Result**: ZERO errors found on homepage
|
||
- **Status**: PASS
|
||
|
||
### "Custom Templates" Removal ✅
|
||
- **Tested**: Full page content and visible text search
|
||
- **Result**: "Custom templates" text NOT found anywhere on page
|
||
- **Status**: PASS - Feature successfully removed from landing page
|
||
|
||
### Email Requirement Implementation ✅
|
||
- **Tested**: Signup form validation
|
||
- **Result**: Email input field has "required" attribute, prevents empty submission
|
||
- **Status**: PASS - Email is now required for free tier signup
|
||
|
||
### Pro Checkout Integration ✅
|
||
- **Tested**: Pro "Get Started" button functionality
|
||
- **Result**: Successfully redirects to Stripe checkout (checkout.stripe.com)
|
||
- **Status**: PASS - Stripe integration working correctly
|
||
|
||
### API Error Handling ✅
|
||
- **Tested**: Bad API keys, missing params, malformed JSON, missing auth
|
||
- **Results**:
|
||
- Invalid API key: 403 + {"error":"Invalid API key"}
|
||
- Missing auth: 401 + {"error":"Missing API key. Use: Authorization: Bearer <key>"}
|
||
- Malformed JSON: 400 + HTML error page
|
||
- **Status**: PASS - Proper HTTP status codes and error messages
|
||
|
||
### CORS Configuration ✅
|
||
- **Tested**: CORS headers on both signup and API endpoints
|
||
- **Results**:
|
||
- Signup endpoint: `Access-Control-Allow-Origin: https://docfast.dev` (CORRECT - restrictive)
|
||
- API endpoints: `Access-Control-Allow-Origin: *` (CORRECT - allows all for API usage)
|
||
- **Status**: PASS - CORS policy correctly implemented per requirements
|
||
|
||
### Documentation Page ✅
|
||
- **Tested**: /docs endpoint accessibility and content
|
||
- **Result**: 200 status, contains API documentation content
|
||
- **Status**: PASS
|
||
|
||
### Mobile Touch Targets ✅
|
||
- **Tested**: Button sizes for touch accessibility (44px minimum)
|
||
- **Result**: All tested buttons meet minimum touch target requirements
|
||
- **Status**: PASS
|
||
|
||
## ⚠️ UNABLE TO FULLY TEST
|
||
|
||
### API Key Generation (Rate Limited)
|
||
- **Attempted**: Multiple signup flows to generate API key
|
||
- **Blocker**: Hit rate limit "Too many signup attempts. Please try again in 1 hour."
|
||
- **Unable to test**: Actual API key functionality, duplicate email handling, SSRF protection
|
||
- **Note**: Rate limiting is good security, but prevents comprehensive testing
|
||
|
||
### SQLite Migration Verification
|
||
- **Status**: No obvious errors, but couldn't verify data persistence due to rate limiting
|
||
- **Recommendation**: Backend team should verify data integrity after migration
|
||
|
||
## FINAL VERDICT
|
||
|
||
**2 issues found**, **8 tests passed**, **2 tests blocked by rate limiting**.
|
||
|
||
The critical mobile responsiveness issue needs immediate fixing. The rate limiting UX could be improved but isn't breaking functionality.
|
||
|
||
---
|
||
|
||
## QA Run — 2026-02-14 18:00 UTC
|
||
|
||
**Tester:** Automated Playwright + curl
|
||
**Context:** Post-deploy verification of BUG-015, BUG-019, BUG-020, and mobile scroll fix
|
||
|
||
### Results: ALL 8 TESTS PASSED ✅
|
||
|
||
| # | Test | Result |
|
||
|---|------|--------|
|
||
| 1 | Console errors (desktop) | ✅ PASS — 0 errors |
|
||
| 2 | Mobile horizontal scroll (375×812) | ✅ PASS — scrollWidth=375, no overflow |
|
||
| 3 | Free signup flow (email → API key) | ✅ PASS — key `df_free_73671ae7...` returned |
|
||
| 4 | Pro checkout → Stripe | ✅ PASS — redirected to checkout.stripe.com |
|
||
| 5 | "Custom templates" removed | ✅ PASS — text not found on page |
|
||
| 6 | API: HTML→PDF conversion | ✅ PASS — 200, application/pdf |
|
||
| 7 | PDF validation | ✅ PASS — 7149 bytes, valid PDF 1.4 |
|
||
| 8a | Error: bad API key | ✅ PASS — 403 |
|
||
| 8b | Error: missing params | ✅ PASS — 400 |
|
||
| 8c | Error: wrong content-type | ✅ PASS — 415 |
|
||
|
||
### Verified Fixes
|
||
- **BUG-019:** "Custom templates" text completely absent from page ✅
|
||
- **BUG-020:** Email input present and required for free signup ✅
|
||
- **CRITICAL mobile scroll:** scrollWidth matches viewport at 375px, no overflow ✅
|
||
- **BUG-015:** Signup succeeded (implying SQLite backend works) ✅
|
||
|
||
### Issues Found
|
||
**None.** All tests passed cleanly. Zero console errors on both desktop and mobile viewports.
|
||
|
||
---
|
||
|
||
## QA Run — 2026-02-14 18:29 UTC
|
||
|
||
**Tester:** Automated Playwright + curl
|
||
**Context:** Email verification flow added (2-step signup: email → code → API key)
|
||
|
||
### Test Results
|
||
|
||
| # | Test | Result | Details |
|
||
|---|------|--------|---------|
|
||
| 1 | Console errors (desktop) | ✅ PASS | 0 errors on clean load |
|
||
| 2 | Browser signup flow | ⚠️ PARTIAL | Modal opens, email input works, verify step exists in DOM. Rate-limited (429) before full flow could complete |
|
||
| 3 | API signup: POST /v1/signup/free | ✅ PASS | Returns `{"status":"verification_required","code":"843266"}` — code in response (see BUG-021) |
|
||
| 3b | API verify: POST /v1/signup/verify | ✅ PASS | Returns `{"status":"verified","apiKey":"df_free_...","tier":"free"}` |
|
||
| 3c | API: use key for PDF | ✅ PASS | PDF generated (valid PDF 1.4), requires `Authorization: Bearer` header |
|
||
| 4 | Wrong code → rejected | ✅ PASS | 400 `{"error":"Invalid verification code."}` |
|
||
| 5 | Already verified email → 409 | 🔴 FAIL | Returns 429 (rate limit) instead of 409. See BUG-022 |
|
||
| 6 | Pro checkout → Stripe | ✅ PASS | POST /v1/billing/checkout returns Stripe checkout URL |
|
||
| 7 | Mobile 375×812 | ✅ PASS | scrollWidth=375, no horizontal overflow |
|
||
| 8a | Bad API key | ✅ PASS | 403 `{"error":"Invalid API key"}` |
|
||
| 8b | Missing params | ✅ PASS | 400 `{"error":"A valid email address is required."}` |
|
||
| 8c | Wrong content-type | 🟡 PARTIAL | Returns 429 (rate limit) instead of 415. When not rate-limited, likely works (passed in prior run) |
|
||
|
||
### New Bugs
|
||
|
||
#### ✅ BUG-021: FIXED (Session 25) — Verification code no longer in API response
|
||
- Postfix + OpenDKIM installed on server
|
||
- Nodemailer sends code via email (noreply@docfast.dev)
|
||
- API response: `{"status":"verification_required","message":"Check your email for the verification code."}`
|
||
- Commit: 210fb26
|
||
- DNS records needed at INWX for deliverability (SPF/DKIM/DMARC)
|
||
|
||
#### 🟡 BUG-022: Already-verified email returns 429 instead of 409
|
||
- **Endpoint:** POST /v1/signup/free with previously verified email
|
||
- **Expected:** 409 Conflict `{"error":"Email already verified"}`
|
||
- **Actual:** 429 `{"error":"Too many signup attempts. Please try again in 1 hour."}`
|
||
- **Problem:** Rate limiting fires before the duplicate-email check. Users can't tell if their email is already registered or if they're rate-limited.
|
||
|
||
#### 🟡 BUG-023: Rate limit too aggressive for testing/development
|
||
- **Observation:** Hit 429 after ~3 signup attempts within minutes
|
||
- **Impact:** Developers integrating the API will hit this quickly during testing
|
||
- **Suggestion:** Consider higher limits or a sandbox/test mode
|
||
|
||
#### 🟡 BUG-024: X-API-Key header not supported (docs inconsistency)
|
||
- **Endpoint:** POST /v1/convert/html
|
||
- **Tested:** `X-API-Key: <key>` header → 401 "Missing API key"
|
||
- **Works with:** `Authorization: Bearer <key>` header → 200
|
||
- **Problem:** If any docs reference X-API-Key, they're wrong. Only Bearer auth works.
|
||
|
||
### Observations
|
||
- **Signup modal UI:** Clean 3-step flow (email → verify code → show key). Code input has proper `inputmode="numeric"`, `maxlength="6"`, `pattern` validation.
|
||
- **PDF endpoint:** `/v1/convert/html` (not `/v1/pdf` or `/v1/generate`)
|
||
- **Checkout:** Uses POST `/v1/billing/checkout` → returns `{url: "https://checkout.stripe.com/..."}` → client redirects
|
||
- **JS bundle:** Single `app.js`, clean event listeners on DOMContentLoaded
|
||
|
||
### Verdict
|
||
**BUG-021 is a showstopper.** The verification code in the API response makes the entire email verification feature security theater. Fix this before shipping.
|