--- title: "Self-hosting Matrix + Element Call with LiveKit: from zero to working (and the [not so!!] fun debugging along the way)" date: 2025-08-26 tags: ["matrix", "synapse", "element-call", "livekit", "coturn", "nginx", "webrtc", "self-hosting", "debugging"] draft: false description: "How I set up a Matrix homeserver (Synapse) with TURN and Element Call using LiveKit, plus the exact debugging steps that took it from 'waiting for media' to solid calls." cover: image: "/images/matrix-cover.png" alt: "Matrix, Signal but distributed" caption: "Distributed end-to-end encrypted chat platform" relative: true hidden: false --- > TL;DR: The call kept saying **“waiting for media”** because the browser never opened a WebSocket to LiveKit. The root cause was **duplicate `Access-Control-Allow-Origin` headers** on `/sfu/get` (CORS), which stopped the JWT response. Fixing CORS and ensuring the WS proxy worked (HTTP **101** in logs) solved it. ## What I'm building - A **Matrix homeserver** (Synapse) at `matrix.example.com` (replace with your domain). - **TURN/STUN** (coTURN) for NAT traversal. - **Element Call** backed by **LiveKit**, fronted by `rtc.example.com`. - **Nginx (host)** as the single reverse proxy for everything. - **Cloudflare** DNS (with `rtc.*` set to **DNS-only**, no orange cloud). - **UFW** firewall opened for Matrix federation, TURN, and LiveKit media ports. > I used Docker for Synapse, PostgreSQL, LiveKit and the JWT helper. I used **host** Nginx (not Nginx in Docker) to avoid port binding conflicts on 80/443/8448. --- ## Prereqs - DNS A/AAAA: - `matrix.example.com` → your server (v4/v6) - `rtc.example.com` → your server (v4/v6) - Certificates: - `matrix.example.com` and `rtc.example.com` via Let’s Encrypt on the host - Cloudflare: **DNS-only (grey cloud)** for `rtc.example.com` so WebSockets & UDP work without interference. - UFW / firewall open: - 80/tcp, 443/tcp - 8448/tcp (Matrix federation) - 3478/tcp, 3478/udp and **5349/tcp** (TURN/TLS) - **LiveKit**: 7881/tcp and **50100–50200/udp** (or your chosen range) - Docker + docker compose installed. --- ## Synapse + PostgreSQL ### 1) The PostgreSQL collation gotcha Synapse prefers the database collation **`C`**. If your Postgres cluster was initialized with `en_US.utf8`, Synapse will error like: ``` Database has incorrect collation of 'en_US.utf8'. Should be 'C' ``` **Two ways to resolve:** - **Preferred (clean)**: Re-initialize the Postgres **cluster** with `C` and `UTF-8`: ```yaml # docker-compose.yml (excerpt for Postgres) services: db: image: postgres:16 environment: POSTGRES_DB: synapse POSTGRES_USER: synapse POSTGRES_PASSWORD: POSTGRES_INITDB_ARGS: "--locale=C --encoding=UTF8 --lc-collate=C --lc-ctype=C" volumes: - ./pgdata:/var/lib/postgresql/data ``` > Requires wiping the volume and recreating the DB. - **Pragmatic (works quickly)**: In Synapse’s DB config, set `allow_unsafe_locale: true`. This bypasses the check. It’s fine for hobby use; for production, prefer the clean `C` cluster. ### 2) Start Synapse and generate config ```bash docker compose up -d db synapse # Logs docker compose logs --tail=200 synapse ``` Ensure Synapse prints your **server_name** and **public base URL** and stays up. ### 3) Create an admin user ```bash # Exec into the running Synapse container: docker compose exec synapse register_new_matrix_user \ -c /data/homeserver.yaml -u -p \ -a -k ``` > If you see “Unknown execution mode”, you probably ran the binary with the wrong entrypoint. Use `docker compose exec synapse …` against the running container. --- ## TURN (coTURN) ### 1) Avoid bad inline comments If you see errors like: ``` ERROR: Unknown boolean value: # log to journald/syslog. You can use on/off, yes/no, 1/0, true/false. ``` …it means a `#` comment is on the **same line** as a boolean directive. Move comments to their own lines. ### 2) Minimal `turnserver.conf` ```ini listening-port=3478 tls-listening-port=5349 fingerprint use-auth-secret static-auth-secret= # also set in Synapse realm=example.com # used in creds generation total-quota=0 bps-capacity=0 cli-password= no-cli cert=/etc/letsencrypt/live/turn.example.com/fullchain.pem pkey=/etc/letsencrypt/live/turn.example.com/privkey.pem # If behind NAT: # external-ip=/ ``` ### 3) Wire TURN into Synapse In `homeserver.yaml`: ```yaml turn_uris: - "turn:turn.example.com?transport=udp" - "turn:turn.example.com?transport=tcp" - "turns:turn.example.com:5349?transport=tcp" turn_shared_secret: "" turn_user_lifetime: "1d" ``` Verify the homeserver issues TURN creds: ```bash TOKEN='' curl -s -H "Authorization: Bearer $TOKEN" \ https://matrix.example.com/_matrix/client/v3/voip/turnServer | jq ``` You should see `uris`, a time-limited `username` and `password`. --- ## Nginx (host) for Synapse and federation Create `/etc/nginx/conf.d/matrix.conf`: ```nginx # Client traffic on 443 server { listen 443 ssl; listen [::]:443 ssl; server_name matrix.example.com; ssl_certificate /etc/letsencrypt/live/matrix.example.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/matrix.example.com/privkey.pem; # Proxy client & admin APIs to Synapse (container) on 8008 location ~ ^(/_matrix|/_synapse/client) { proxy_pass http://127.0.0.1:8008; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Host $host; client_max_body_size 50M; } # Advertise homeserver base + RTC focus (MSC4143) via .well-known location = /.well-known/matrix/client { default_type application/json; add_header Access-Control-Allow-Origin "*" always; return 200 '{"m.homeserver":{"base_url":"https://matrix.example.com"},"org.matrix.msc4143.rtc_foci":[{"type":"livekit","livekit_service_url":"https://rtc.example.com"}]}'; } } # Federation on 8448 server { listen 8448 ssl http2; listen [::]:8448 ssl http2; server_name matrix.example.com; ssl_certificate /etc/letsencrypt/live/matrix.example.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/matrix.example.com/privkey.pem; location / { proxy_pass http://127.0.0.1:8008; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Host $host; client_max_body_size 50M; } } ``` Reload and sanity check: ```bash sudo nginx -t && sudo systemctl reload nginx curl -s https://matrix.example.com/.well-known/matrix/client | jq ``` Also check Synapse supports RTC signaling (MSC4140) so clients actually use it: ```bash curl -s https://matrix.example.com/_matrix/client/versions | jq '.unstable_features."org.matrix.msc4140"' # expect: true ``` --- ## Element Call + LiveKit I’ll run **LiveKit** and the small **JWT helper** (Element’s `elementcall_jwt`) in Docker. LiveKit handles media; the JWT helper mints access tokens for WebSocket connects. ### 1) LiveKit config (`/etc/livekit.yaml` inside container) ```yaml port: 7880 bind_addresses: ["0.0.0.0"] rtc: tcp_port: 7881 port_range_start: 50100 port_range_end: 50200 use_external_ip: true logging: level: info turn: enabled: false keys: lk_prod_1: "REPLACE_WITH_A_64_CHAR_RANDOM_SECRET___________________________________" ``` > **Important:** the secret must be **>= 32 chars**. The default `devkey` will trigger `secret is too short` warnings and won’t work with the JWT helper. ### 2) elementcall_jwt env Run it with: - `LIVEKIT_URL=wss://rtc.example.com` _(root WS URL, **no** `/livekit/sfu` path)_ - `LIVEKIT_KEY=lk_prod_1` - `LIVEKIT_SECRET=` - `LIVEKIT_JWT_PORT=8080` (internal HTTP port the proxy will hit) - Optionally: `LIVEKIT_FULL_ACCESS_HOMESERVERS=*` during setup Check logs on start; it prints the LIVEKIT_URL it will advertise. ### 3) Nginx (host) for `rtc.example.com` Create `/etc/nginx/conf.d/rtc.conf`: ```nginx server { listen 443 ssl; listen [::]:443 ssl; server_name rtc.example.com; ssl_certificate /etc/letsencrypt/live/rtc.example.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/rtc.example.com/privkey.pem; access_log /var/log/nginx/rtc.access.log combined; # 3a) JWT endpoint with clean CORS (avoid duplicate ACAO) location = /sfu/get { proxy_hide_header Access-Control-Allow-Origin; add_header Access-Control-Allow-Origin $http_origin always; add_header Vary "Origin" always; add_header Access-Control-Allow-Methods "POST, OPTIONS" always; add_header Access-Control-Allow-Headers "Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization" always; if ($request_method = OPTIONS) { return 204; } proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto $scheme; proxy_pass http://127.0.0.1:8070/sfu/get; # elementcall_jwt } # 3b) LiveKit WS & HTTP (catch-all) location / { proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol; # "livekit" proxy_set_header Origin $http_origin; proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto $scheme; proxy_buffering off; proxy_read_timeout 3600s; proxy_pass http://127.0.0.1:7880; # livekit } } ``` Reload and basic checks: ```bash sudo nginx -t && sudo systemctl reload nginx # JWT preflight (should return a single ACAO header) curl -si -X OPTIONS https://rtc.example.com/sfu/get \ -H 'Origin: https://app.element.io' \ -H 'Access-Control-Request-Method: POST' \ -H 'Access-Control-Request-Headers: authorization, content-type' | sed -n '1,30p' # Expected: one Access-Control-Allow-Origin and 200/204 ``` > **Why I did this:** I initially had **two** `Access-Control-Allow-Origin` headers (one added by the upstream, one by Nginx). Browsers reject that with “Access-Control-Allow-Origin cannot contain more than one origin”, so the JWT response never reached the client. Fixing CORS fixed everything. --- ## The “waiting for media” debugging story (how I found it) Symptom: Element Call created rooms, but calls stayed on **“waiting for media.”** What worked: - `elementcall_jwt` could **CreateRoom** in LiveKit (seen in logs). - TURN creds endpoint returned time-limited credentials. What didn’t appear: - No **HTTP 101** lines in `rtc.access.log` → the **browser never established a WebSocket** to LiveKit. ### Step 1: prove the WS vhost works (even without auth) ```bash curl -v --http1.1 \ -H 'Connection: Upgrade' -H 'Upgrade: websocket' \ -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \ -H 'Sec-WebSocket-Version: 13' \ https://rtc.example.com/rtc -o /dev/null ``` Result: I got a `401` (expected), but importantly I saw a log line in `rtc.access.log`. So **Nginx WS proxying was fine**. ### Step 2: check the browser console The smoking gun in DevTools: ``` Access-Control-Allow-Origin cannot contain more than one origin. Fetch API cannot load https://rtc.example.com/sfu/get due to access control checks. ``` The browser refused the JWT call due to duplicated **ACAO** headers, so no token → no WebSocket connect. **Fix:** in Nginx I added: ```nginx proxy_hide_header Access-Control-Allow-Origin; add_header Access-Control-Allow-Origin $http_origin always; add_header Vary "Origin" always; ``` …and ensured no other `add_header` created duplicates. After that, `/sfu/get` succeeded and the WebSocket to `wss://rtc.example.com` immediately followed (I saw **HTTP 101** in the logs). ### Step 3: confirm LiveKit side Once the WS was up, LiveKit logs showed participants joining (not just `RoomService.CreateRoom`), and calls were established. --- ## Useful verification commands ```bash # Synapse features (expect MSC4140 true) curl -s https://matrix.example.com/_matrix/client/versions | jq '.unstable_features."org.matrix.msc4140"' # Well-known with RTC focus curl -s https://matrix.example.com/.well-known/matrix/client | jq # TURN creds (with your access token) curl -s -H "Authorization: Bearer $TOKEN" \ https://matrix.example.com/_matrix/client/v3/voip/turnServer | jq # JWT health curl -si -X POST https://rtc.example.com/sfu/get # LiveKit simple HTTP probe curl -si https://rtc.example.com | head # Nginx logs (look for 101 Switching Protocols when a call starts) sudo tail -f /var/log/nginx/rtc.access.log | grep ' 101 ' ``` --- ## Common pitfalls (I hit these so you don’t have to) - **Host Nginx vs Docker Nginx:** If you already run Nginx on the host, don’t also bind 80/443/8448 in a Docker Nginx — you’ll get `bind() ... already in use` and restart loops. Use **host** Nginx to reverse proxy to containers. - **Nginx `http2` directive:** Old Nginx may not support the `http2` directive on `listen`. Use `listen 443 ssl;` (and add `http2` if your version supports it). - **Certificate name mismatch:** Make sure `rtc.example.com`’s vhost uses a certificate **for that exact hostname** (initially I had the `matrix.*` cert on `rtc.*` and curl complained). - **Postgres collation:** Either initialize the cluster with `C` or use `allow_unsafe_locale: true` in Synapse DB config to get running quickly. - **CORS duplication on `/sfu/get`:** Only ONE `Access-Control-Allow-Origin` header. If the upstream adds it too, use `proxy_hide_header Access-Control-Allow-Origin;` on the Nginx location. - **Cloudflare:** Use **DNS-only** for `rtc.*`. Proxies can interfere with WS and UDP paths. - **Firewall:** Open the LiveKit UDP range and TURN ports on both v4 and v6. --- ## Final checklist (print me) - [ ] `https://matrix.example.com/.well-known/matrix/client` returns **both** `m.homeserver.base_url` and `org.matrix.msc4143.rtc_foci` pointing to `https://rtc.example.com`. - [ ] `/_matrix/client/versions` shows `"org.matrix.msc4140": true`. - [ ] `/sfu/get` **preflight** returns **one** `Access-Control-Allow-Origin` and **200/204**. - [ ] Starting a call creates **HTTP 101** entries to `wss://rtc.example.com` in `rtc.access.log`. - [ ] LiveKit logs show **participants joining** (not just CreateRoom). - [ ] `/voip/turnServer` returns time-limited TURN credentials. - [ ] Cloudflare set to **DNS-only** for `rtc.*`. UFW allows 7881/tcp and your LiveKit UDP range. --- ### Credits & tooling - Matrix Synapse, coTURN, LiveKit, Element Call. - `curl`, `jq`, `docker compose logs`, Nginx access logs. These are your best friends. - The debugging breakthrough was catching CORS errors in the browser console and looking for **HTTP 101** in Nginx logs. Happy calling! 🎉 --- {{< sigdl >}}