1c545c93b4f15659de8fdd0093f9bfe5b54e9078
Security: - Add CRON_SECRET auth to /api/cron/* endpoints - Add admin role verification to /api/admin/* routes - Add org membership check to /api/billing/usage - Add security headers (HSTS, X-Frame-Options, CSP, etc.) - Add env variable validation at startup - Add rate limiting to backend API (30 req/min per IP) Infrastructure: - Multi-stage Dockerfiles with non-root user + healthchecks - Updated cron workflow to pass CRON_SECRET header - Updated .env.example with all optional vars Smart subpage scanning: - Crawler now computes template_hash (DOM structure without content) - Scanner scans ALL unique-layout pages, not just main page - Pages with same layout (e.g. product pages) scanned only once - Deduplication by template_hash, fallback to content_hash - Main page always scanned with high priority - Re-checks subscription limits before each page scan Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
🔍 Website Monitoring Platform
Full-stack website monitoring platform that uses Google Lighthouse to audit performance, SEO, accessibility, and best practices — with real-time progress tracking, team collaboration, and alerting.
┌─────────────────────────────────────────────────────────┐
│ Website Monitoring │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │
│ │ Frontend │───▶│ Backend │───▶│ PostgreSQL (DB) │ │
│ │ Next.js │ │ Express │ │ via Supabase │ │
│ │ Port 3000 │◀───│ Port 5000│ └──────────────────┘ │
│ └──────────┘ └──────────┘ │
│ │ │ │
│ │ ▼ │
│ │ ┌──────────┐ │
│ │ │Lighthouse│ │
│ └─── SSE ──│ + Chrome │ │
│ (progress) │ Headless │ │
│ └──────────┘ │
└─────────────────────────────────────────────────────────┘
✨ Features
| Feature | Status | Description |
|---|---|---|
| Lighthouse Audits | ✅ Real | Performance, SEO, accessibility, best practices scores |
| Real-time Progress | ✅ Real | Server-Sent Events stream during scans |
| Dashboard | ✅ Real | Overview with charts, metrics, and website list |
| Performance Monitoring | ✅ Real | Track scores over time with Recharts |
| SEO Analysis | ✅ Real | SEO score tracking and recommendations |
| Uptime Monitoring | ✅ Real | HTTP HEAD checks every 5 min, response time + SSL tracking |
| Alert Engine | ✅ Real | Evaluates scans against thresholds, auto-resolves on recovery |
| Notifications | ✅ Real | Email (Resend) + webhook delivery with debouncing |
| Admin Dashboard | ✅ Real | System stats, user CRUD, org management (role-protected) |
| Billing & Usage | ✅ Real | 4 tiers (free/starter/pro/enterprise), usage bars, limit enforcement |
| Competitor Analysis | ✅ Real | Lighthouse comparison + response time benchmarking |
| Team/Organization | ✅ Real | Multi-user orgs with 4-level RBAC |
| Authentication | ✅ Real | Supabase Auth (email, OAuth) |
| Scheduled Scans | ✅ Real | Cron-based Lighthouse scans (6h) + uptime checks (5min) |
| Website Crawler | ✅ Real | Crawl and discover pages on your websites |
🛠 Tech Stack
| Layer | Technology | Cost |
|---|---|---|
| Frontend | Next.js 15, React 19, TypeScript, Tailwind CSS 4, Shadcn/UI | Free |
| Backend | Express.js, TypeScript, Node.js 18+ | Free |
| Database | PostgreSQL via Supabase | Free tier |
| Auth | Supabase Auth | Free tier |
| Auditing | Google Lighthouse + Headless Chrome | Free (OSS) |
| Charts | Recharts + Chart.js | Free (OSS) |
| CI/CD | GitHub Actions | Free (public repos) |
| Containers | Docker + Docker Compose | Free |
| Linting | ESLint + Prettier | Free (OSS) |
| Testing | Jest + Supertest + Testing Library | Free (OSS) |
| Resend | Free tier (3000/mo) | |
| Pre-commit | Husky | Free (OSS) |
🚀 Quick Start
Prerequisites
- Node.js 18+
- npm
- Docker & Docker Compose (for full stack)
- Supabase account (free tier)
Option 1: Local Development
# Clone the repo
git clone <repo-url>
cd website-monitoring
# Install root dependencies (Husky, concurrently)
npm install
# Setup backend
cd website-monitoring-backend
cp .env.example .env
npm install
npm run build
cd ..
# Setup frontend
cd website-monitoring-frontend
cp .env.example .env # Fill in your Supabase keys
npm install
cd ..
# Run everything
npm run dev
Option 2: Docker Compose
# Copy and fill in environment variables
cp .env.example .env
# Start all services
npm run docker:up
# Access at http://localhost:3000
📁 Project Structure
website-monitoring/
├── website-monitoring-backend/ # Express.js API + Lighthouse engine
│ ├── src/
│ │ ├── index.ts # Server entry, health check, routing
│ │ └── routes/
│ │ └── lighthouse.ts # Lighthouse audit + SSE progress
│ ├── Dockerfile
│ └── package.json
│
├── website-monitoring-frontend/ # Next.js 15 dashboard
│ ├── src/
│ │ ├── app/ # Pages & API routes (20+ endpoints)
│ │ ├── components/ # React components (dashboard, UI, auth)
│ │ ├── services/ # Business logic (scanning, monitoring)
│ │ └── types/ # TypeScript type definitions
│ ├── Dockerfile
│ └── package.json
│
├── website-monitoring-devops/ # Infrastructure
│ ├── docker-compose.yml # Full stack orchestration
│ └── .devcontainer/ # VS Code Dev Container config
│
├── .github/
│ ├── workflows/
│ │ ├── backend.yml # Backend CI: lint, test, build
│ │ ├── frontend.yml # Frontend CI: lint, test, build
│ │ └── docker.yml # Docker Compose integration test
│ └── pull_request_template.md
│
├── CONTRIBUTING.md # Branch strategy, code review, guidelines
├── .env.example # Unified environment template
└── package.json # Root scripts (dev, build, test, lint)
🧪 Testing
# Run all tests (22 total)
npm test
# Backend only (11 tests: health, routes, validation, CORS, 404)
npm run test:backend
# Frontend only (11 tests: tier limits, chart components)
npm run test:frontend
📊 CI/CD Pipelines
| Workflow | Trigger | What it does |
|---|---|---|
backend.yml |
Push/PR to backend | Lint → Test → Build (Node 18 & 20) |
frontend.yml |
Push/PR to frontend | Lint → Test → Build (Node 18 & 20) |
docker.yml |
Push/PR to main | Docker Compose build → Backend health check |
🔑 Key API Routes
| Route | Method | Description |
|---|---|---|
/api/cron/uptime |
GET | Runs uptime checks on all active websites |
/api/cron/scan |
GET | Triggers Lighthouse scans for due websites |
/api/notifications/process |
POST | Processes pending alerts and sends notifications |
/api/admin/stats |
GET | System-wide stats (admin only) |
/api/admin/users |
GET/PATCH/DELETE | User management (admin only) |
/api/admin/organizations |
GET/PATCH | Organization management (admin only) |
/api/billing/usage |
GET | Current org usage vs tier limits |
/api/competitor-analysis |
GET/POST | Competitor benchmarking |
🏗 Tier System
| Tier | Websites | Scans/mo | Members | Price |
|---|---|---|---|---|
| Free | 3 | 50 | 2 | $0 |
| Starter | 10 | 500 | 5 | $9/mo |
| Professional | 50 | 5,000 | 25 | $29/mo |
| Enterprise | ∞ | ∞ | ∞ | $99/mo |
🤝 Contributing
See CONTRIBUTING.md for branch strategy, code style, and PR process.
📄 License
ISC
Description
Languages
TypeScript
90%
PLpgSQL
7.7%
Shell
0.8%
CSS
0.7%
JavaScript
0.3%
Other
0.5%