Session 27: rate limits, BUG-025/022 fixes, QA review, state update
This commit is contained in:
Hoid
parent
eb46e5635f
commit
f5d11bedc1
5 changed files with 112 additions and 186 deletions
|
|
@ -1,184 +1,67 @@
|
|||
# DocFast QA Test Results - 2026-02-14
|
||||
# DocFast QA Session 27 - 2026-02-15
|
||||
|
||||
## Test Session Overview
|
||||
Backend changes tested:
|
||||
1. Removed "Custom templates" from Pro plan on landing page
|
||||
2. Re-added email requirement for free tier signup
|
||||
3. Migrated from JSON storage to SQLite
|
||||
## BROWSER TESTS RESULTS
|
||||
|
||||
## Issues Found
|
||||
### ✅ PASSED
|
||||
1. **JS Errors Check**: ZERO JavaScript errors on initial load, BUT errors appeared during testing:
|
||||
- ❌ 400 Bad Request error from /v1/signup/verify (during fake verification attempt)
|
||||
- ⚠️ EvalError from px-cloud.net (third-party security script)
|
||||
- ⚠️ CSP warnings from Stripe checkout (expected)
|
||||
2. **Signup Flow**: Modal opens correctly, email validation works, verification code screen appears properly
|
||||
3. **Error Handling**: Invalid verification code shows proper error message "Invalid verification code."
|
||||
4. **Pro Checkout**: Stripe redirect works perfectly - proper checkout form with $9/month pricing
|
||||
5. **Mobile Responsive**: 375x812 mobile test shows NO horizontal scroll, proper responsive layout
|
||||
6. **Docs Page**: /docs loads correctly with complete API documentation
|
||||
|
||||
### 🔴 CRITICAL: Mobile Horizontal Scrolling
|
||||
- **What I tested**: Mobile responsiveness at 375x812 (iPhone X)
|
||||
- **Expected**: No horizontal scrolling
|
||||
- **Found**: Page has horizontal scrolling (scroll width: 488px vs viewport: 375px)
|
||||
- **Impact**: Poor mobile user experience, makes content inaccessible
|
||||
## API TESTS RESULTS
|
||||
|
||||
### 🟡 MEDIUM: Rate Limiting Not User-Friendly
|
||||
- **What I tested**: Multiple signup attempts
|
||||
- **Expected**: Clear indication of rate limiting before user hits it
|
||||
- **Found**: No upfront warning about rate limits, users only discover after filling form
|
||||
- **Response**: "Too many signup attempts. Please try again in 1 hour."
|
||||
- **Improvement**: Show rate limit status or warning before form submission
|
||||
### ✅ PASSED
|
||||
1. **Bad API Key (403)**: Returns proper 403 Forbidden with "Invalid API key" message
|
||||
2. **Rate Limit Headers**: All responses include proper X-RateLimit headers (RateLimit-Policy, RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset)
|
||||
3. **Concurrency Endpoint**: /v1/concurrency exists and properly requires authentication (401 when no key provided)
|
||||
|
||||
## ✅ TESTS PASSED
|
||||
### ⚠️ NEEDS INVESTIGATION
|
||||
1. **BUG-022 (Duplicate Email Check)**: Duplicate email `test@example.com` returned 200 OK instead of expected 409 Conflict. Response: `{"status":"verification_required","message":"Check your email for the verification code."}` - This could be intentional UX (don't reveal email existence) or the bug fix might not be working correctly.
|
||||
|
||||
### Console Errors ✅
|
||||
- **Tested**: page.on('pageerror') and page.on('console', type=error)
|
||||
- **Result**: ZERO errors found on homepage
|
||||
- **Status**: PASS
|
||||
### ❌ NEEDS TESTING WITH VALID KEY
|
||||
- Parameter validation (400 errors) - blocked by 403 auth check
|
||||
- Content-type validation (415 errors) - blocked by 403 auth check
|
||||
- PDF generation testing - need valid API key
|
||||
- Pro plan rate limits (30/min) vs Free (10/min) - need both key types
|
||||
|
||||
### "Custom Templates" Removal ✅
|
||||
- **Tested**: Full page content and visible text search
|
||||
- **Result**: "Custom templates" text NOT found anywhere on page
|
||||
- **Status**: PASS - Feature successfully removed from landing page
|
||||
## INCOMPLETE TESTS
|
||||
- **BUG-025 (Copy Button)**: COULD NOT TEST - Unable to reach API key display screen without valid verification code. Browser timeout prevented further testing of recovery flow.
|
||||
- **Concurrency stats**: COULD NOT TEST - Requires valid API key for GET /v1/concurrency response format
|
||||
|
||||
### Email Requirement Implementation ✅
|
||||
- **Tested**: Signup form validation
|
||||
- **Result**: Email input field has "required" attribute, prevents empty submission
|
||||
- **Status**: PASS - Email is now required for free tier signup
|
||||
## NOTES
|
||||
- All rate limiting headers are present and properly formatted
|
||||
- Stripe integration works flawlessly
|
||||
- Mobile layout is perfectly responsive
|
||||
- Error messaging is clear and user-friendly
|
||||
- Authentication security is properly enforced
|
||||
|
||||
### Pro Checkout Integration ✅
|
||||
- **Tested**: Pro "Get Started" button functionality
|
||||
- **Result**: Successfully redirects to Stripe checkout (checkout.stripe.com)
|
||||
- **Status**: PASS - Stripe integration working correctly
|
||||
## FINAL ASSESSMENT - Session 27
|
||||
|
||||
### API Error Handling ✅
|
||||
- **Tested**: Bad API keys, missing params, malformed JSON, missing auth
|
||||
- **Results**:
|
||||
- Invalid API key: 403 + {"error":"Invalid API key"}
|
||||
- Missing auth: 401 + {"error":"Missing API key. Use: Authorization: Bearer <key>"}
|
||||
- Malformed JSON: 400 + HTML error page
|
||||
- **Status**: PASS - Proper HTTP status codes and error messages
|
||||
**OVERALL VERDICT**: 🔴 ISSUES FOUND requiring immediate attention
|
||||
|
||||
### CORS Configuration ✅
|
||||
- **Tested**: CORS headers on both signup and API endpoints
|
||||
- **Results**:
|
||||
- Signup endpoint: `Access-Control-Allow-Origin: https://docfast.dev` (CORRECT - restrictive)
|
||||
- API endpoints: `Access-Control-Allow-Origin: *` (CORRECT - allows all for API usage)
|
||||
- **Status**: PASS - CORS policy correctly implemented per requirements
|
||||
**CRITICAL ISSUES**:
|
||||
- Console errors during normal usage flow (400 error + third-party EvalError)
|
||||
- BUG-022 verification status unclear (duplicate email handling)
|
||||
|
||||
### Documentation Page ✅
|
||||
- **Tested**: /docs endpoint accessibility and content
|
||||
- **Result**: 200 status, contains API documentation content
|
||||
- **Status**: PASS
|
||||
**UNABLE TO VERIFY**:
|
||||
- BUG-025 (copy button fix) - needs complete signup flow
|
||||
- Complete PDF generation pipeline - needs valid API keys
|
||||
|
||||
### Mobile Touch Targets ✅
|
||||
- **Tested**: Button sizes for touch accessibility (44px minimum)
|
||||
- **Result**: All tested buttons meet minimum touch target requirements
|
||||
- **Status**: PASS
|
||||
**WHAT WORKS WELL**:
|
||||
- Clean UI/UX with zero JS errors
|
||||
- Proper error handling and messaging
|
||||
- Stripe checkout integration is flawless
|
||||
- Mobile responsiveness is perfect
|
||||
- API authentication and rate limiting works correctly
|
||||
- Documentation is comprehensive
|
||||
|
||||
## ⚠️ UNABLE TO FULLY TEST
|
||||
|
||||
### API Key Generation (Rate Limited)
|
||||
- **Attempted**: Multiple signup flows to generate API key
|
||||
- **Blocker**: Hit rate limit "Too many signup attempts. Please try again in 1 hour."
|
||||
- **Unable to test**: Actual API key functionality, duplicate email handling, SSRF protection
|
||||
- **Note**: Rate limiting is good security, but prevents comprehensive testing
|
||||
|
||||
### SQLite Migration Verification
|
||||
- **Status**: No obvious errors, but couldn't verify data persistence due to rate limiting
|
||||
- **Recommendation**: Backend team should verify data integrity after migration
|
||||
|
||||
## FINAL VERDICT
|
||||
|
||||
**2 issues found**, **8 tests passed**, **2 tests blocked by rate limiting**.
|
||||
|
||||
The critical mobile responsiveness issue needs immediate fixing. The rate limiting UX could be improved but isn't breaking functionality.
|
||||
|
||||
---
|
||||
|
||||
## QA Run — 2026-02-14 18:00 UTC
|
||||
|
||||
**Tester:** Automated Playwright + curl
|
||||
**Context:** Post-deploy verification of BUG-015, BUG-019, BUG-020, and mobile scroll fix
|
||||
|
||||
### Results: ALL 8 TESTS PASSED ✅
|
||||
|
||||
| # | Test | Result |
|
||||
|---|------|--------|
|
||||
| 1 | Console errors (desktop) | ✅ PASS — 0 errors |
|
||||
| 2 | Mobile horizontal scroll (375×812) | ✅ PASS — scrollWidth=375, no overflow |
|
||||
| 3 | Free signup flow (email → API key) | ✅ PASS — key `df_free_73671ae7...` returned |
|
||||
| 4 | Pro checkout → Stripe | ✅ PASS — redirected to checkout.stripe.com |
|
||||
| 5 | "Custom templates" removed | ✅ PASS — text not found on page |
|
||||
| 6 | API: HTML→PDF conversion | ✅ PASS — 200, application/pdf |
|
||||
| 7 | PDF validation | ✅ PASS — 7149 bytes, valid PDF 1.4 |
|
||||
| 8a | Error: bad API key | ✅ PASS — 403 |
|
||||
| 8b | Error: missing params | ✅ PASS — 400 |
|
||||
| 8c | Error: wrong content-type | ✅ PASS — 415 |
|
||||
|
||||
### Verified Fixes
|
||||
- **BUG-019:** "Custom templates" text completely absent from page ✅
|
||||
- **BUG-020:** Email input present and required for free signup ✅
|
||||
- **CRITICAL mobile scroll:** scrollWidth matches viewport at 375px, no overflow ✅
|
||||
- **BUG-015:** Signup succeeded (implying SQLite backend works) ✅
|
||||
|
||||
### Issues Found
|
||||
**None.** All tests passed cleanly. Zero console errors on both desktop and mobile viewports.
|
||||
|
||||
---
|
||||
|
||||
## QA Run — 2026-02-14 18:29 UTC
|
||||
|
||||
**Tester:** Automated Playwright + curl
|
||||
**Context:** Email verification flow added (2-step signup: email → code → API key)
|
||||
|
||||
### Test Results
|
||||
|
||||
| # | Test | Result | Details |
|
||||
|---|------|--------|---------|
|
||||
| 1 | Console errors (desktop) | ✅ PASS | 0 errors on clean load |
|
||||
| 2 | Browser signup flow | ⚠️ PARTIAL | Modal opens, email input works, verify step exists in DOM. Rate-limited (429) before full flow could complete |
|
||||
| 3 | API signup: POST /v1/signup/free | ✅ PASS | Returns `{"status":"verification_required","code":"843266"}` — code in response (see BUG-021) |
|
||||
| 3b | API verify: POST /v1/signup/verify | ✅ PASS | Returns `{"status":"verified","apiKey":"df_free_...","tier":"free"}` |
|
||||
| 3c | API: use key for PDF | ✅ PASS | PDF generated (valid PDF 1.4), requires `Authorization: Bearer` header |
|
||||
| 4 | Wrong code → rejected | ✅ PASS | 400 `{"error":"Invalid verification code."}` |
|
||||
| 5 | Already verified email → 409 | 🔴 FAIL | Returns 429 (rate limit) instead of 409. See BUG-022 |
|
||||
| 6 | Pro checkout → Stripe | ✅ PASS | POST /v1/billing/checkout returns Stripe checkout URL |
|
||||
| 7 | Mobile 375×812 | ✅ PASS | scrollWidth=375, no horizontal overflow |
|
||||
| 8a | Bad API key | ✅ PASS | 403 `{"error":"Invalid API key"}` |
|
||||
| 8b | Missing params | ✅ PASS | 400 `{"error":"A valid email address is required."}` |
|
||||
| 8c | Wrong content-type | 🟡 PARTIAL | Returns 429 (rate limit) instead of 415. When not rate-limited, likely works (passed in prior run) |
|
||||
|
||||
### New Bugs
|
||||
|
||||
#### ✅ BUG-021: FIXED (Session 25) — Verification code no longer in API response
|
||||
- Postfix + OpenDKIM installed on server
|
||||
- Nodemailer sends code via email (noreply@docfast.dev)
|
||||
- API response: `{"status":"verification_required","message":"Check your email for the verification code."}`
|
||||
- Commit: 210fb26
|
||||
- DNS records needed at INWX for deliverability (SPF/DKIM/DMARC)
|
||||
|
||||
#### 🟡 BUG-022: Already-verified email returns 429 instead of 409
|
||||
- **Endpoint:** POST /v1/signup/free with previously verified email
|
||||
- **Expected:** 409 Conflict `{"error":"Email already verified"}`
|
||||
- **Actual:** 429 `{"error":"Too many signup attempts. Please try again in 1 hour."}`
|
||||
- **Problem:** Rate limiting fires before the duplicate-email check. Users can't tell if their email is already registered or if they're rate-limited.
|
||||
|
||||
#### 🟡 BUG-023: Rate limit too aggressive for testing/development
|
||||
- **Observation:** Hit 429 after ~3 signup attempts within minutes
|
||||
- **Impact:** Developers integrating the API will hit this quickly during testing
|
||||
- **Suggestion:** Consider higher limits or a sandbox/test mode
|
||||
|
||||
#### 🟡 BUG-024: X-API-Key header not supported (docs inconsistency)
|
||||
- **Endpoint:** POST /v1/convert/html
|
||||
- **Tested:** `X-API-Key: <key>` header → 401 "Missing API key"
|
||||
- **Works with:** `Authorization: Bearer <key>` header → 200
|
||||
- **Problem:** If any docs reference X-API-Key, they're wrong. Only Bearer auth works.
|
||||
|
||||
#### 🔴 BUG-025: Copy button on API key display not working
|
||||
- **Reported by:** Investor (manual testing)
|
||||
- **Location:** Signup modal, step 3 (API key display)
|
||||
- **Expected:** Click copy button → key copied to clipboard
|
||||
- **Actual:** Nothing happens
|
||||
- **Priority:** HIGH — users can't easily copy their API key
|
||||
|
||||
### Observations
|
||||
- **Signup modal UI:** Clean 3-step flow (email → verify code → show key). Code input has proper `inputmode="numeric"`, `maxlength="6"`, `pattern` validation.
|
||||
- **PDF endpoint:** `/v1/convert/html` (not `/v1/pdf` or `/v1/generate`)
|
||||
- **Checkout:** Uses POST `/v1/billing/checkout` → returns `{url: "https://checkout.stripe.com/..."}` → client redirects
|
||||
- **JS bundle:** Single `app.js`, clean event listeners on DOMContentLoaded
|
||||
|
||||
### Verdict
|
||||
**BUG-021 is a showstopper.** The verification code in the API response makes the entire email verification feature security theater. Fix this before shipping.
|
||||
**RECOMMENDATIONS FOR NEXT SESSION**:
|
||||
1. Get valid API keys through backend team to test copy button functionality
|
||||
2. Verify BUG-022 duplicate email behavior is intentional
|
||||
3. Test actual PDF generation with valid keys
|
||||
4. Test pro vs free rate limit differences in practice
|
||||
Loading…
Add table
Add a link
Reference in a new issue