426 lines
14 KiB
Markdown
426 lines
14 KiB
Markdown
---
|
||
title: "Self-hosting Matrix + Element Call with LiveKit: from zero to working (and the [not so!!] fun debugging along the way)"
|
||
date: 2025-08-26
|
||
tags: ["matrix", "synapse", "element-call", "livekit", "coturn", "nginx", "webrtc", "self-hosting", "debugging"]
|
||
draft: false
|
||
description: "How I set up a Matrix homeserver (Synapse) with TURN and Element Call using LiveKit, plus the exact debugging steps that took it from 'waiting for media' to solid calls."
|
||
cover:
|
||
image: "/images/matrix-cover.png"
|
||
alt: "Matrix, Signal but distributed"
|
||
caption: "Distributed end-to-end encrypted chat platform"
|
||
relative: true
|
||
hidden: false
|
||
---
|
||
|
||
> TL;DR: The call kept saying **“waiting for media”** because the browser never opened a WebSocket to LiveKit. The root cause was **duplicate `Access-Control-Allow-Origin` headers** on `/sfu/get` (CORS), which stopped the JWT response. Fixing CORS and ensuring the WS proxy worked (HTTP **101** in logs) solved it.
|
||
|
||
## What I'm building
|
||
|
||
- A **Matrix homeserver** (Synapse) at `matrix.example.com` (replace with your domain).
|
||
- **TURN/STUN** (coTURN) for NAT traversal.
|
||
- **Element Call** backed by **LiveKit**, fronted by `rtc.example.com`.
|
||
- **Nginx (host)** as the single reverse proxy for everything.
|
||
- **Cloudflare** DNS (with `rtc.*` set to **DNS-only**, no orange cloud).
|
||
- **UFW** firewall opened for Matrix federation, TURN, and LiveKit media ports.
|
||
|
||
> I used Docker for Synapse, PostgreSQL, LiveKit and the JWT helper. I used **host** Nginx (not Nginx in Docker) to avoid port binding conflicts on 80/443/8448.
|
||
|
||
---
|
||
|
||
## Prereqs
|
||
|
||
- DNS A/AAAA:
|
||
- `matrix.example.com` → your server (v4/v6)
|
||
- `rtc.example.com` → your server (v4/v6)
|
||
- Certificates:
|
||
- `matrix.example.com` and `rtc.example.com` via Let’s Encrypt on the host
|
||
- Cloudflare: **DNS-only (grey cloud)** for `rtc.example.com` so WebSockets & UDP work without interference.
|
||
- UFW / firewall open:
|
||
- 80/tcp, 443/tcp
|
||
- 8448/tcp (Matrix federation)
|
||
- 3478/tcp, 3478/udp and **5349/tcp** (TURN/TLS)
|
||
- **LiveKit**: 7881/tcp and **50100–50200/udp** (or your chosen range)
|
||
- Docker + docker compose installed.
|
||
|
||
---
|
||
|
||
## Synapse + PostgreSQL
|
||
|
||
### 1) The PostgreSQL collation gotcha
|
||
|
||
Synapse prefers the database collation **`C`**. If your Postgres cluster was initialized with `en_US.utf8`, Synapse will error like:
|
||
|
||
```
|
||
Database has incorrect collation of 'en_US.utf8'. Should be 'C'
|
||
```
|
||
|
||
**Two ways to resolve:**
|
||
|
||
- **Preferred (clean)**: Re-initialize the Postgres **cluster** with `C` and `UTF-8`:
|
||
|
||
```yaml
|
||
# docker-compose.yml (excerpt for Postgres)
|
||
services:
|
||
db:
|
||
image: postgres:16
|
||
environment:
|
||
POSTGRES_DB: synapse
|
||
POSTGRES_USER: synapse
|
||
POSTGRES_PASSWORD: <strong-password>
|
||
POSTGRES_INITDB_ARGS: "--locale=C --encoding=UTF8 --lc-collate=C --lc-ctype=C"
|
||
volumes:
|
||
- ./pgdata:/var/lib/postgresql/data
|
||
```
|
||
|
||
> Requires wiping the volume and recreating the DB.
|
||
|
||
- **Pragmatic (works quickly)**: In Synapse’s DB config, set `allow_unsafe_locale: true`. This bypasses the check. It’s fine for hobby use; for production, prefer the clean `C` cluster.
|
||
|
||
### 2) Start Synapse and generate config
|
||
|
||
```bash
|
||
docker compose up -d db synapse
|
||
# Logs
|
||
docker compose logs --tail=200 synapse
|
||
```
|
||
|
||
Ensure Synapse prints your **server_name** and **public base URL** and stays up.
|
||
|
||
### 3) Create an admin user
|
||
|
||
```bash
|
||
# Exec into the running Synapse container:
|
||
docker compose exec synapse register_new_matrix_user \
|
||
-c /data/homeserver.yaml -u <username> -p <password> \
|
||
-a -k
|
||
```
|
||
|
||
> If you see “Unknown execution mode”, you probably ran the binary with the wrong entrypoint. Use `docker compose exec synapse …` against the running container.
|
||
|
||
---
|
||
|
||
## TURN (coTURN)
|
||
|
||
### 1) Avoid bad inline comments
|
||
|
||
If you see errors like:
|
||
|
||
```
|
||
ERROR: Unknown boolean value: # log to journald/syslog. You can use on/off, yes/no, 1/0, true/false.
|
||
```
|
||
|
||
…it means a `#` comment is on the **same line** as a boolean directive. Move comments to their own lines.
|
||
|
||
### 2) Minimal `turnserver.conf`
|
||
|
||
```ini
|
||
listening-port=3478
|
||
tls-listening-port=5349
|
||
fingerprint
|
||
use-auth-secret
|
||
static-auth-secret=<shared-secret> # also set in Synapse
|
||
realm=example.com # used in creds generation
|
||
total-quota=0
|
||
bps-capacity=0
|
||
cli-password=<admin-pass>
|
||
no-cli
|
||
cert=/etc/letsencrypt/live/turn.example.com/fullchain.pem
|
||
pkey=/etc/letsencrypt/live/turn.example.com/privkey.pem
|
||
# If behind NAT:
|
||
# external-ip=<public-ip>/<internal-ip>
|
||
```
|
||
|
||
### 3) Wire TURN into Synapse
|
||
|
||
In `homeserver.yaml`:
|
||
|
||
```yaml
|
||
turn_uris:
|
||
- "turn:turn.example.com?transport=udp"
|
||
- "turn:turn.example.com?transport=tcp"
|
||
- "turns:turn.example.com:5349?transport=tcp"
|
||
turn_shared_secret: "<shared-secret>"
|
||
turn_user_lifetime: "1d"
|
||
```
|
||
|
||
Verify the homeserver issues TURN creds:
|
||
|
||
```bash
|
||
TOKEN='<your matrix access token>'
|
||
curl -s -H "Authorization: Bearer $TOKEN" \
|
||
https://matrix.example.com/_matrix/client/v3/voip/turnServer | jq
|
||
```
|
||
|
||
You should see `uris`, a time-limited `username` and `password`.
|
||
|
||
---
|
||
|
||
## Nginx (host) for Synapse and federation
|
||
|
||
Create `/etc/nginx/conf.d/matrix.conf`:
|
||
|
||
```nginx
|
||
# Client traffic on 443
|
||
server {
|
||
listen 443 ssl; listen [::]:443 ssl;
|
||
server_name matrix.example.com;
|
||
|
||
ssl_certificate /etc/letsencrypt/live/matrix.example.com/fullchain.pem;
|
||
ssl_certificate_key /etc/letsencrypt/live/matrix.example.com/privkey.pem;
|
||
|
||
# Proxy client & admin APIs to Synapse (container) on 8008
|
||
location ~ ^(/_matrix|/_synapse/client) {
|
||
proxy_pass http://127.0.0.1:8008;
|
||
proxy_set_header X-Forwarded-For $remote_addr;
|
||
proxy_set_header X-Forwarded-Proto $scheme;
|
||
proxy_set_header Host $host;
|
||
client_max_body_size 50M;
|
||
}
|
||
|
||
# Advertise homeserver base + RTC focus (MSC4143) via .well-known
|
||
location = /.well-known/matrix/client {
|
||
default_type application/json;
|
||
add_header Access-Control-Allow-Origin "*" always;
|
||
return 200 '{"m.homeserver":{"base_url":"https://matrix.example.com"},"org.matrix.msc4143.rtc_foci":[{"type":"livekit","livekit_service_url":"https://rtc.example.com"}]}';
|
||
}
|
||
}
|
||
|
||
# Federation on 8448
|
||
server {
|
||
listen 8448 ssl http2; listen [::]:8448 ssl http2;
|
||
server_name matrix.example.com;
|
||
|
||
ssl_certificate /etc/letsencrypt/live/matrix.example.com/fullchain.pem;
|
||
ssl_certificate_key /etc/letsencrypt/live/matrix.example.com/privkey.pem;
|
||
|
||
location / {
|
||
proxy_pass http://127.0.0.1:8008;
|
||
proxy_set_header X-Forwarded-For $remote_addr;
|
||
proxy_set_header X-Forwarded-Proto $scheme;
|
||
proxy_set_header Host $host;
|
||
client_max_body_size 50M;
|
||
}
|
||
}
|
||
```
|
||
|
||
Reload and sanity check:
|
||
|
||
```bash
|
||
sudo nginx -t && sudo systemctl reload nginx
|
||
curl -s https://matrix.example.com/.well-known/matrix/client | jq
|
||
```
|
||
|
||
Also check Synapse supports RTC signaling (MSC4140) so clients actually use it:
|
||
|
||
```bash
|
||
curl -s https://matrix.example.com/_matrix/client/versions | jq '.unstable_features."org.matrix.msc4140"'
|
||
# expect: true
|
||
```
|
||
|
||
---
|
||
|
||
## Element Call + LiveKit
|
||
|
||
I’ll run **LiveKit** and the small **JWT helper** (Element’s `elementcall_jwt`) in Docker. LiveKit handles media; the JWT helper mints access tokens for WebSocket connects.
|
||
|
||
### 1) LiveKit config (`/etc/livekit.yaml` inside container)
|
||
|
||
```yaml
|
||
port: 7880
|
||
bind_addresses: ["0.0.0.0"]
|
||
rtc:
|
||
tcp_port: 7881
|
||
port_range_start: 50100
|
||
port_range_end: 50200
|
||
use_external_ip: true
|
||
logging:
|
||
level: info
|
||
turn:
|
||
enabled: false
|
||
keys:
|
||
lk_prod_1: "REPLACE_WITH_A_64_CHAR_RANDOM_SECRET___________________________________"
|
||
```
|
||
|
||
> **Important:** the secret must be **>= 32 chars**. The default `devkey` will trigger `secret is too short` warnings and won’t work with the JWT helper.
|
||
|
||
### 2) elementcall_jwt env
|
||
|
||
Run it with:
|
||
|
||
- `LIVEKIT_URL=wss://rtc.example.com` _(root WS URL, **no** `/livekit/sfu` path)_
|
||
- `LIVEKIT_KEY=lk_prod_1`
|
||
- `LIVEKIT_SECRET=<the long secret above>`
|
||
- `LIVEKIT_JWT_PORT=8080` (internal HTTP port the proxy will hit)
|
||
- Optionally: `LIVEKIT_FULL_ACCESS_HOMESERVERS=*` during setup
|
||
|
||
Check logs on start; it prints the LIVEKIT_URL it will advertise.
|
||
|
||
### 3) Nginx (host) for `rtc.example.com`
|
||
|
||
Create `/etc/nginx/conf.d/rtc.conf`:
|
||
|
||
```nginx
|
||
server {
|
||
listen 443 ssl; listen [::]:443 ssl;
|
||
server_name rtc.example.com;
|
||
|
||
ssl_certificate /etc/letsencrypt/live/rtc.example.com/fullchain.pem;
|
||
ssl_certificate_key /etc/letsencrypt/live/rtc.example.com/privkey.pem;
|
||
|
||
access_log /var/log/nginx/rtc.access.log combined;
|
||
|
||
# 3a) JWT endpoint with clean CORS (avoid duplicate ACAO)
|
||
location = /sfu/get {
|
||
proxy_hide_header Access-Control-Allow-Origin;
|
||
add_header Access-Control-Allow-Origin $http_origin always;
|
||
add_header Vary "Origin" always;
|
||
add_header Access-Control-Allow-Methods "POST, OPTIONS" always;
|
||
add_header Access-Control-Allow-Headers "Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization" always;
|
||
if ($request_method = OPTIONS) { return 204; }
|
||
|
||
proxy_set_header Host $host;
|
||
proxy_set_header X-Forwarded-Proto $scheme;
|
||
proxy_pass http://127.0.0.1:8070/sfu/get; # elementcall_jwt
|
||
}
|
||
|
||
# 3b) LiveKit WS & HTTP (catch-all)
|
||
location / {
|
||
proxy_http_version 1.1;
|
||
proxy_set_header Upgrade $http_upgrade;
|
||
proxy_set_header Connection "upgrade";
|
||
proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol; # "livekit"
|
||
proxy_set_header Origin $http_origin;
|
||
proxy_set_header Host $host;
|
||
proxy_set_header X-Forwarded-Proto $scheme;
|
||
proxy_buffering off;
|
||
proxy_read_timeout 3600s;
|
||
proxy_pass http://127.0.0.1:7880; # livekit
|
||
}
|
||
}
|
||
```
|
||
|
||
Reload and basic checks:
|
||
|
||
```bash
|
||
sudo nginx -t && sudo systemctl reload nginx
|
||
# JWT preflight (should return a single ACAO header)
|
||
curl -si -X OPTIONS https://rtc.example.com/sfu/get \
|
||
-H 'Origin: https://app.element.io' \
|
||
-H 'Access-Control-Request-Method: POST' \
|
||
-H 'Access-Control-Request-Headers: authorization, content-type' | sed -n '1,30p'
|
||
# Expected: one Access-Control-Allow-Origin and 200/204
|
||
```
|
||
|
||
> **Why I did this:** I initially had **two** `Access-Control-Allow-Origin` headers (one added by the upstream, one by Nginx). Browsers reject that with “Access-Control-Allow-Origin cannot contain more than one origin”, so the JWT response never reached the client. Fixing CORS fixed everything.
|
||
|
||
---
|
||
|
||
## The “waiting for media” debugging story (how I found it)
|
||
|
||
Symptom: Element Call created rooms, but calls stayed on **“waiting for media.”**
|
||
|
||
What worked:
|
||
- `elementcall_jwt` could **CreateRoom** in LiveKit (seen in logs).
|
||
- TURN creds endpoint returned time-limited credentials.
|
||
|
||
What didn’t appear:
|
||
- No **HTTP 101** lines in `rtc.access.log` → the **browser never established a WebSocket** to LiveKit.
|
||
|
||
### Step 1: prove the WS vhost works (even without auth)
|
||
|
||
```bash
|
||
curl -v --http1.1 \
|
||
-H 'Connection: Upgrade' -H 'Upgrade: websocket' \
|
||
-H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
|
||
-H 'Sec-WebSocket-Version: 13' \
|
||
https://rtc.example.com/rtc -o /dev/null
|
||
```
|
||
Result: I got a `401` (expected), but importantly I saw a log line in `rtc.access.log`. So **Nginx WS proxying was fine**.
|
||
|
||
### Step 2: check the browser console
|
||
|
||
The smoking gun in DevTools:
|
||
|
||
```
|
||
Access-Control-Allow-Origin cannot contain more than one origin.
|
||
Fetch API cannot load https://rtc.example.com/sfu/get due to access control checks.
|
||
```
|
||
|
||
The browser refused the JWT call due to duplicated **ACAO** headers, so no token → no WebSocket connect.
|
||
|
||
**Fix:** in Nginx I added:
|
||
|
||
```nginx
|
||
proxy_hide_header Access-Control-Allow-Origin;
|
||
add_header Access-Control-Allow-Origin $http_origin always;
|
||
add_header Vary "Origin" always;
|
||
```
|
||
|
||
…and ensured no other `add_header` created duplicates. After that, `/sfu/get` succeeded and the WebSocket to `wss://rtc.example.com` immediately followed (I saw **HTTP 101** in the logs).
|
||
|
||
### Step 3: confirm LiveKit side
|
||
|
||
Once the WS was up, LiveKit logs showed participants joining (not just `RoomService.CreateRoom`), and calls were established.
|
||
|
||
---
|
||
|
||
## Useful verification commands
|
||
|
||
```bash
|
||
# Synapse features (expect MSC4140 true)
|
||
curl -s https://matrix.example.com/_matrix/client/versions | jq '.unstable_features."org.matrix.msc4140"'
|
||
|
||
# Well-known with RTC focus
|
||
curl -s https://matrix.example.com/.well-known/matrix/client | jq
|
||
|
||
# TURN creds (with your access token)
|
||
curl -s -H "Authorization: Bearer $TOKEN" \
|
||
https://matrix.example.com/_matrix/client/v3/voip/turnServer | jq
|
||
|
||
# JWT health
|
||
curl -si -X POST https://rtc.example.com/sfu/get
|
||
|
||
# LiveKit simple HTTP probe
|
||
curl -si https://rtc.example.com | head
|
||
|
||
# Nginx logs (look for 101 Switching Protocols when a call starts)
|
||
sudo tail -f /var/log/nginx/rtc.access.log | grep ' 101 '
|
||
```
|
||
|
||
---
|
||
|
||
## Common pitfalls (I hit these so you don’t have to)
|
||
|
||
- **Host Nginx vs Docker Nginx:** If you already run Nginx on the host, don’t also bind 80/443/8448 in a Docker Nginx — you’ll get `bind() ... already in use` and restart loops. Use **host** Nginx to reverse proxy to containers.
|
||
- **Nginx `http2` directive:** Old Nginx may not support the `http2` directive on `listen`. Use `listen 443 ssl;` (and add `http2` if your version supports it).
|
||
- **Certificate name mismatch:** Make sure `rtc.example.com`’s vhost uses a certificate **for that exact hostname** (initially I had the `matrix.*` cert on `rtc.*` and curl complained).
|
||
- **Postgres collation:** Either initialize the cluster with `C` or use `allow_unsafe_locale: true` in Synapse DB config to get running quickly.
|
||
- **CORS duplication on `/sfu/get`:** Only ONE `Access-Control-Allow-Origin` header. If the upstream adds it too, use `proxy_hide_header Access-Control-Allow-Origin;` on the Nginx location.
|
||
- **Cloudflare:** Use **DNS-only** for `rtc.*`. Proxies can interfere with WS and UDP paths.
|
||
- **Firewall:** Open the LiveKit UDP range and TURN ports on both v4 and v6.
|
||
|
||
---
|
||
|
||
## Final checklist (print me)
|
||
|
||
- [ ] `https://matrix.example.com/.well-known/matrix/client` returns **both** `m.homeserver.base_url` and `org.matrix.msc4143.rtc_foci` pointing to `https://rtc.example.com`.
|
||
- [ ] `/_matrix/client/versions` shows `"org.matrix.msc4140": true`.
|
||
- [ ] `/sfu/get` **preflight** returns **one** `Access-Control-Allow-Origin` and **200/204**.
|
||
- [ ] Starting a call creates **HTTP 101** entries to `wss://rtc.example.com` in `rtc.access.log`.
|
||
- [ ] LiveKit logs show **participants joining** (not just CreateRoom).
|
||
- [ ] `/voip/turnServer` returns time-limited TURN credentials.
|
||
- [ ] Cloudflare set to **DNS-only** for `rtc.*`. UFW allows 7881/tcp and your LiveKit UDP range.
|
||
|
||
---
|
||
|
||
### Credits & tooling
|
||
|
||
- Matrix Synapse, coTURN, LiveKit, Element Call.
|
||
- `curl`, `jq`, `docker compose logs`, Nginx access logs. These are your best friends.
|
||
- The debugging breakthrough was catching CORS errors in the browser console and looking for **HTTP 101** in Nginx logs.
|
||
|
||
Happy calling! 🎉
|
||
|
||
---
|
||
|
||
{{< sigdl >}}
|