perf: pre-compress gamedata JSON, tune MariaDB, fix avatar imager, docs

- scripts/compress-gamedata.mjs: pre-compress large gamedata JSON to .gz (gzip level 9, idempotent mtime check) served via nginx gzip_static (48 MB FurnitureData.json -> ~2.4 MB, ~0.3s -> ~0.005s per request) - package.json: add gamedata:compress script - fix(imaging): accept real Habbo figure strings in avatar route (allow optional second number per part, e.g. hd-180-1.ch-210-66) - README: document avatar imager container (avatar-imaging-pixinode, port 8082, /docker/Polaris-imager), nginx /imaging proxying, MariaDB tuning, gamedata pre-compression cron and caching layers
2026-09-06 12:06:38 +02:00
parent 251738fdda
commit 9dc9d1fa4b
4 changed files with 260 additions and 9 deletions
@@ -112,7 +112,7 @@ Open `http://localhost:3002` in your browser.

 ## Docker Deployment

-The CMS ships with a multi-stage Dockerfile that automatically detects your package manager (pnpm, npm, or yarn) based on which lockfile is present. Only the **CMS** runs in Docker; the Nitro/Octane client and its renderer stay outside and are served by nginx on the host.
+The CMS ships with a multi-stage Dockerfile that automatically detects your package manager (pnpm, npm, or yarn) based on which lockfile is present. The **CMS** runs in Docker; the Nitro/Octane client stays outside and is served by nginx on the host. The server-side **avatar imager** also runs in its own Docker container (`avatar-imaging-pixinode`, port 8082) — see [Avatar Imaging](#avatar-imaging).

 ### Prerequisites

@@ -212,7 +212,7 @@ This means you can use any package manager on your host machine — the Docker b

 ### Volumes

-Only the CMS runs in Docker. The heavy client assets and gamedata stay on the host and are shared into the container so imports and uploads persist:
+Only the CMS (and the optional avatar imager container, see [Avatar Imaging](#avatar-imaging)) run in Docker. The heavy client assets and gamedata stay on the host and are shared into the container so imports and uploads persist:

 | Container Path             | Host Path                  | Mode | Purpose                              |
 | -------------------------- | -------------------------- | ---- | ------------------------------------ |
@@ -228,7 +228,7 @@ Only the CMS runs in Docker. The heavy client assets and gamedata stay on the ho
 | Component            | Host path               | How it's served                                                        |
 | -------------------- | ----------------------- | ---------------------------------------------------------------------- |
 | Nitro/Octane client  | `/var/www/Octane/dist`  | nginx `location ^~ /client/` and `/nitro-client/` alias                |
-| Octane-Renderer      | `/var/www/Octane-Renderer` | Optional. Run separately (or as the commented-out `imager` service) proxied by nginx `/imaging` |
+| Avatar imager        | `/docker/Polaris-imager`| Docker container `avatar-imaging-pixinode` on port `8082` — see [Avatar Imaging](#avatar-imaging) |
 | Camera uploads       | `/var/www/Camera`       | nginx `/camera/` alias                                                 |

 **Container user & write permissions:** the CMS container runs as `www-data` (UID/GID 33) by default to match the host owner of `/var/www/Gamedata`. Ensure the other write volumes (`./public/nitro-assets`, `./public/swf`, `./storage`) are also owned by `www-data` on the host:
@@ -386,9 +386,11 @@ server {
        add_header Cache-Control "public, max-age=31536000, immutable";
    }

-    location /imaging {
-        proxy_pass http://127.0.0.1:3030;
-        add_header Cache-Control "public, max-age=3600, s-maxage=86400";
+    location /imaging/ {
+        proxy_pass http://127.0.0.1:8082/;
+        proxy_set_header Connection "";
+        proxy_http_version 1.1;
+        add_header Cache-Control "public, max-age=300";
    }

    location /health {
@@ -399,6 +401,122 @@ server {
 }
 ```

+> **`/imaging` trailing-slash gotcha:** `location /imaging` (no trailing slash) combined with `proxy_pass http://…8082/;` rewrites `/imaging/avatarimage` into `//avatarimage` and 404s. Always use `location /imaging/` + a trailing-slash `proxy_pass` so `/imaging/avatarimage` reaches the imager's `/avatarimage` route.
+
+---
+
+## Avatar Imaging
+
+Avatar images (profile figures, chat badges, forum avatars, admin previews) are rendered server-side by **Polaris-imager** ([duckietm/Polaris-imager](https://github.com/duckietm/Polaris-imager)) — a headless Nitro/Octane renderer that runs `@pixi/node` (real WebGL via `gl` + Xvfb) inside a Docker container named **`avatar-imaging-pixinode`**. It serves `GET /avatarimage?figure=…` on port **8082** and can emit PNG/APNG/GIF, gestures, actions, scenes and more.
+
+### Request flow
+
+```
+browser → Cloudflare → (page rule /origin for /imaging/*) → host:8082
+browser → nginx origin → location /imaging/ → 127.0.0.1:8082/avatarimage
+```
+
+URLs look like `https://<hotel>/imaging/avatarimage?figure=hd-180-1.ch-210-66&img_format=png&size=l&direction=4`. The public `NEXT_PUBLIC_IMAGER_URL` points at that endpoint (see `src/lib/imager.ts`). Responses carry `Cache-Control: public, max-age=300`, so browsers and Cloudflare reuse a render for 5 minutes.
+
+### Requirements (host)
+
+- Docker (daemon with `build.network: host`, same as the CMS build — this host disables Docker iptables).
+- An nginx that serves `/gamedata/` (FigureData/FigureMap/EffectMap…) and `/gamedata/bundled` (`.nitro` assets) over HTTP so the renderer can fetch them. The compose file points at `host.docker.internal:8081`.
+
+### Configuration — `/docker/Polaris-imager/.env`
+
+```dotenv
+NITRO_GAMEDATA_URL=http://host.docker.internal:8081/gamedata/config
+NITRO_ASSET_URL=http://host.docker.internal:8081/gamedata/bundled
+AVATAR_IMAGING_FPS=12
+AVATAR_IMAGING_MAX_FRAMES=60
+AVATAR_IMAGING_HOST=0.0.0.0
+AVATAR_IMAGING_PORT=8082
+AVATAR_IMAGING_RATELIMIT_MAX=600
+AVATAR_IMAGING_SCENE=1
+AVATAR_IMAGING_WARDROBE=1
+AVATAR_IMAGING_HABBO_FONTS=1
+AVATAR_IMAGING_CHAT_BUBBLES=1
+AVATAR_IMAGING_HOTEL_URL=https://<hotel>
+AVATAR_IMAGING_HOTEL_NAME=<hotel>
+```
+
+### Build & start
+
+The image is self-contained — the `Dockerfile` clones the pinned Nitro/Octane renderer and runs the full `yarn`/`vite` bundle at build time, then produces a slim Debian runtime with Xvfb + Mesa software rendering:
+
+```bash
+cd /docker/Polaris-imager
+docker compose build
+docker compose up -d        # container: avatar-imaging-pixinode
+docker logs -f avatar-imaging-pixinode   # expect: "listening on http://0.0.0.0:8082"
+```
+
+The container runs `restart: unless-stopped` and ships a healthcheck that probes `/health` (reports `{ "ready": true }` once the headless renderer has booted). Verify a render:
+
+```bash
+curl -o avatar.png 'http://127.0.0.1:8082/avatarimage?figure=hr-893-45.hd-600-1.ch-255-66.lg-280-110.sh-295-62&img_format=png&size=l'
+```
+
+A production issue that this section documents: **the `avatar-imaging-pixinode` image can be removed by a `docker image prune`**, which takes the whole site's avatars offline (502 on `/imaging/*`). Keep the image tagged and re-build it via the commands above if it ever disappears (`docker compose build && docker compose up -d`).
+
+### Figure string validation (`src/app/api/imaging/avatar/route.ts`)
+
+The CMS-side route re-validates the incoming figure against `/^[a-z]{2}-\d+(-\d+)?(\.[a-z]{2}-\d+(-\d+)?)*$/i`. Real Habbo figure strings use `type-id-color` parts separated by dots (`hd-180-1.ch-210-66`), so each part allows an optional second number. Invalid input is rejected with `400` before any render happens.
+
+---
+
+## Performance Tuning (production)
+
+The CMS shares its MySQL/MariaDB database with the emulator, so the hot wins come from caching, asset serving and database knob-tuning — not from a storage-engine migration.
+
+### 1. gzip_static pre-compression of gamedata JSON
+
+The client downloads `FurnitureData.json` (~48 MB, one file per supported language) plus other large JSON from `/gamedata/`. nginx has `gzip` and `gzip_static on` (see `/etc/nginx/nginx.conf`), but with no pre-built `.gz` file it re-compressed the 48 MB **on every request** (~0.25–0.30 s of CPU per miss).
+
+`scripts/compress-gamedata.mjs` pre-compresses every JSON >1 MB in `<Gamedata>/config` and `<Gamedata>/bundled/config` once (gzip level 9): `FurnitureData*.json` go from ~48 MB to ~2.4 MB (~95 % smaller). It is idempotent — a file is only re-compressed when the source mtime is newer than its `.gz` (e.g. after a Studio catalog export).
+
+```bash
+pnpm gamedata:compress    # manual run
+```
+
+Once the `.gz` exists, nginx `gzip_static` serves it straight from disk with `Content-Encoding: gzip` and effectively zero CPU. **Measured result on the production origin: 0.25–0.30 s → ~0.005 s per file.**
+
+The production host runs the script on a cron so newly exported catalogs are compressed shortly after they land:
+
+```bash
+crontab -e
+# */15 * * * * node /var/www/atom-nexst/scripts/compress-gamedata.mjs >> /var/www/atom-nexst/logs/gamedata-compress.log 2>&1
+```
+
+### 2. MariaDB tuning
+
+The whole `habbo` datadir is roughly 0.5 GB; the default InnoDB buffer pool (128 MB) pushed index/row reads to disk. The pool is raised to **4 GB** and the slow-query log is enabled (threshold 2 s), both live (`SET GLOBAL …`) and persisted for boot in `/etc/mysql/mariadb.conf.d/zz-cms-performance.cnf`:
+
+```ini
+[mysqld]
+innodb_buffer_pool_size = 4G
+slow_query_log = 1
+long_query_time = 2
+slow_query_log_file = /var/log/mysql/mariadb-slow.log
+```
+
+The slow-query log surfaces hot-spot SQL for a follow-up index/query audit. The CMS connection pool itself is already tuned (pool size 25, `connection_limit` in `DATABASE_URL`).
+
+### 3. Asset caching headers
+
+| Path                           | Cache-Control                                                  |
+| ------------------------------ | -------------------------------------------------------------- |
+| `/swf/**`, `/nitro-assets/**`  | `public, max-age=604800, stale-while-revalidate=2592000`       |
+| `/gamedata/**`, `/client/**`   | `public` + `expires 7–30 d` (nginx)                            |
+| `/imaging/*`                   | `public, max-age=300`                                          |
+| `/camera/**`                   | `public, max-age=31536000, immutable`                          |
+| `/_next/static/**`             | `public, max-age=31536000, immutable`                          |
+
+### 4. Cache layers (Redis + CDN)
+
+The CMS caches through `cached()` / `cachedQuery()` (`src/lib/cache.ts`): an in-memory fast path with a Redis (DragonflyDB-compatible) fallback and single-flight protection against cache stampedes. Public API routes (home, leaderboard, online count, radio, photos, …) fill Redis keys per route; verify with `redis-cli DBSIZE`. With Cloudflare in front, static and `/imaging/*` responses are additionally cached edge-side (`cf-cache-status`), cutting origin load further.
+
 ---

 ## Production Deployment (PM2)
@@ -435,6 +553,7 @@ pm2 restart next
 | `pnpm db:generate`        | Draft SQL via drizzle-kit → `drizzle/drafts/`      |
 | `pnpm db:studio`          | Drizzle Studio (dev)                               |
 | `pnpm db:introspect`      | drizzle-kit introspect (draft)                     |
+| `pnpm gamedata:compress`  | Pre-compress large gamedata JSON to `.gz` (gzip_static) |
 | `pnpm analyze`            | Build + open bundle analyzer                       |
 | `pnpm jobs:worker`        | Start background task worker                       |
 | `pnpm biome:check`        | Lint and format code                               |
@@ -456,6 +575,7 @@ pm2 restart next
 | **Streaming & Suspense**       | Next.js App Router streaming for fast page loads               |
 | **Automatic Image Optimization** | `next/image` with Sharp for resizing and WebP/AVIF          |
 | **Compression**                | Gzip compression enabled on all responses                      |
+| **gzip_static (gamedata)**     | 48 MB FurnitureData JSON served from pre-built `.gz` — no per-request CPU |
 | **PWA**                        | Service worker with skip-waiting, stale CSS chunk recovery     |
 | **Biome**                      | Fast linting and formatting (replaces ESLint + Prettier)       |

@@ -470,7 +590,8 @@ pm2 restart next
 ├── scripts/
 │   ├── apply-migrations.ts    # SQL migration runner (apply + status)
 │   ├── jobs-worker.ts         # Background task scheduler
-│   └── generate-drizzle-schema.mjs # Regen src/db/schema.ts from schema + live DB
+│   ├── generate-drizzle-schema.mjs # Regen src/db/schema.ts from schema + live DB
+│   └── compress-gamedata.mjs  # Pre-compress large gamedata JSON (gzip_static)
 ├── src/
 │   ├── db/
 │   │   ├── schema.ts          # Drizzle ORM schema (committed — runtime data layer)
@@ -577,4 +698,3 @@ pnpm biome:check  # Lint and format
 ## License

 CC BY-NC-SA 4.0. See `LICENSE` for details.
-Test aanpassing voor Gitea Actions
@@ -24,6 +24,7 @@
 		"db:migrate": "tsx scripts/apply-migrations.ts",
 		"db:migrate:status": "tsx scripts/apply-migrations.ts --status",
 		"db:studio": "drizzle-kit studio",
+		"gamedata:compress": "node scripts/compress-gamedata.mjs",
 		"hk:matrix:check": "tsx scripts/verify-housekeeping-matrix.ts",
 		"test:housekeeping": "vitest run --coverage.enabled=false src/features/housekeeping src/lib/admin-theme-source-audit.test.ts src/lib/admin/authorization-contract.test.ts",
 		"prepare": "node scripts/prepare-hooks.mjs",
@@ -0,0 +1,130 @@
+/**
+ * Pre-compress large gamedata JSON files to .gz so nginx `gzip_static`
+ * serves them via sendfile instead of re-compressing up to 48 MB on every
+ * request (see /etc/nginx/nginx.conf: `gzip_static on`).
+ *
+ * Idempotent: a file is only re-compressed when its source mtime is newer
+ * than the existing .gz (e.g. after a Studio catalog export rewrites it).
+ *
+ * Usage: node scripts/compress-gamedata.mjs
+ * Env:   GAMEDATA_ROOT (default /var/www/Gamedata)
+ *        GAMEDATA_GZIP_MIN_BYTES (default 1048576)
+ */
+
+import { createReadStream, createWriteStream, promises as fs } from "node:fs";
+import { cpus } from "node:os";
+import { join, relative } from "node:path";
+import { pipeline } from "node:stream/promises";
+import { createGzip } from "node:zlib";
+
+const GAMEDATA_ROOT = process.env.GAMEDATA_ROOT ?? "/var/www/Gamedata";
+const MIN_BYTES = Number(process.env.GAMEDATA_GZIP_MIN_BYTES ?? 1024 * 1024);
+const GZIP_LEVEL = 9;
+const TARGET_DIRS = ["config", "bundled/config"];
+const CONCURRENCY = Math.max(1, Math.min(4, cpus().length));
+
+function isCompressible(name) {
+	return (
+		name.endsWith(".json") &&
+		!name.endsWith(".gz") &&
+		!name.endsWith(".min.json")
+	);
+}
+
+async function collectCandidates() {
+	const files = [];
+	for (const dir of TARGET_DIRS) {
+		const root = join(GAMEDATA_ROOT, dir);
+		let entries;
+		try {
+			entries = await fs.readdir(root, { withFileTypes: true });
+		} catch {
+			continue;
+		}
+		for (const entry of entries) {
+			if (!entry.isFile() || !isCompressible(entry.name)) continue;
+			const full = join(root, entry.name);
+			const stat = await fs.stat(full);
+			if (stat.size < MIN_BYTES) continue;
+			files.push({ src: full, size: stat.size, mtimeMs: stat.mtimeMs });
+		}
+	}
+	return files;
+}
+
+async function needsCompression({ src, mtimeMs }) {
+	const gz = `${src}.gz`;
+	try {
+		const stat = await fs.stat(gz);
+		return stat.mtimeMs < mtimeMs;
+	} catch {
+		return true;
+	}
+}
+
+async function compressOne({ src }) {
+	const gz = `${src}.gz`;
+	const tmp = `${gz}.tmp-${process.pid}`;
+	await pipeline(
+		createReadStream(src),
+		createGzip({ level: GZIP_LEVEL }),
+		createWriteStream(tmp),
+	);
+	await fs.rename(tmp, gz);
+	try {
+		await fs.chmod(gz, 0o644);
+	} catch {
+		// Best-effort; ownership is up to the caller.
+	}
+	return gz;
+}
+
+async function main() {
+	const candidates = await collectCandidates();
+	const work = [];
+	const skipped = [];
+	for (const candidate of candidates) {
+		if (await needsCompression(candidate)) {
+			work.push(candidate);
+		} else {
+			skipped.push(candidate);
+		}
+	}
+
+	const results = [];
+	let idx = 0;
+	async function worker() {
+		while (idx < work.length) {
+			const item = work[idx++];
+			try {
+				const gz = await compressOne(item);
+				const stat = await fs.stat(gz);
+				results.push(
+					`compressed ${relative(GAMEDATA_ROOT, gz)} (${item.size} -> ${stat.size} bytes, ${Math.round((1 - stat.size / item.size) * 1000) / 10}% smaller)`,
+				);
+			} catch (err) {
+				results.push(
+					`FAILED ${relative(GAMEDATA_ROOT, item.src)}: ${err instanceof Error ? err.message : String(err)}`,
+				);
+			}
+		}
+	}
+
+	const workers = Array.from(
+		{ length: Math.min(CONCURRENCY, work.length) },
+		() => worker(),
+	);
+	await Promise.all(workers);
+
+	console.log(
+		`gamedata: ${work.length} to compress, ${skipped.length} up to date`,
+	);
+	for (const line of results) console.log(`gamedata: ${line}`);
+}
+
+main().catch((err) => {
+	console.error(
+		`gamedata: FATAL ${err instanceof Error ? err.message : String(err)}`,
+	);
+	process.exit(1);
+});
@@ -1,7 +1,7 @@
 import { type NextRequest, NextResponse } from "next/server";
 import { resolveImagerBase } from "@/lib/imager";

-const FIGURE_RE = /^([a-z]{2}-\d+)(\.[a-z]{2}-\d+)*$/i;
+const FIGURE_RE = /^[a-z]{2}-\d+(-\d+)?(\.[a-z]{2}-\d+(-\d+)?)*$/i;
 const FIGURE_MAX_LEN = 512;
 const FIGURE_MAX_PARTS = 24;
 const UPSTREAM_TIMEOUT_MS = 10_000;