---
title: "Self-hosting Matrix + Element Call with LiveKit: from zero to working (and the [not so!!] fun debugging along the way)"
date: 2025-08-26
tags: ["matrix", "synapse", "element-call", "livekit", "coturn", "nginx", "webrtc", "self-hosting", "debugging"]
draft: false
description: "How I set up a Matrix homeserver (Synapse) with TURN and Element Call using LiveKit, plus the exact debugging steps that took it from 'waiting for media' to solid calls."
cover:
  image: "/images/matrix-cover.png"
  alt: "Matrix, Signal but distributed"
  caption: "Distributed end-to-end encrypted chat platform"
  relative: true
  hidden: false
---

> TL;DR: The call kept saying **“waiting for media”** because the browser never opened a WebSocket to LiveKit. The root cause was **duplicate `Access-Control-Allow-Origin` headers** on `/sfu/get` (CORS), which stopped the JWT response. Fixing CORS and ensuring the WS proxy worked (HTTP **101** in logs) solved it.

## What I'm building

- A **Matrix homeserver** (Synapse) at `matrix.example.com` (replace with your domain).
- **TURN/STUN** (coTURN) for NAT traversal.
- **Element Call** backed by **LiveKit**, fronted by `rtc.example.com`.
- **Nginx (host)** as the single reverse proxy for everything.
- **Cloudflare** DNS (with `rtc.*` set to **DNS-only**, no orange cloud).
- **UFW** firewall opened for Matrix federation, TURN, and LiveKit media ports.

> I used Docker for Synapse, PostgreSQL, LiveKit and the JWT helper. I used **host** Nginx (not Nginx in Docker) to avoid port binding conflicts on 80/443/8448.

---

## Prereqs

- DNS A/AAAA:
  - `matrix.example.com` → your server (v4/v6)
  - `rtc.example.com` → your server (v4/v6)
- Certificates:
  - `matrix.example.com` and `rtc.example.com` via Let’s Encrypt on the host
- Cloudflare: **DNS-only (grey cloud)** for `rtc.example.com` so WebSockets & UDP work without interference.
- UFW / firewall open:
  - 80/tcp, 443/tcp
  - 8448/tcp (Matrix federation)
  - 3478/tcp, 3478/udp and **5349/tcp** (TURN/TLS)
  - **LiveKit**: 7881/tcp and **50100–50200/udp** (or your chosen range)
- Docker + docker compose installed.

---

## Synapse + PostgreSQL

### 1) The PostgreSQL collation gotcha

Synapse prefers the database collation **`C`**. If your Postgres cluster was initialized with `en_US.utf8`, Synapse will error like:

```
Database has incorrect collation of 'en_US.utf8'. Should be 'C'
```

**Two ways to resolve:**

- **Preferred (clean)**: Re-initialize the Postgres **cluster** with `C` and `UTF-8`:

```yaml
# docker-compose.yml (excerpt for Postgres)
services:
  db:
    image: postgres:16
    environment:
      POSTGRES_DB: synapse
      POSTGRES_USER: synapse
      POSTGRES_PASSWORD: <strong-password>
      POSTGRES_INITDB_ARGS: "--locale=C --encoding=UTF8 --lc-collate=C --lc-ctype=C"
    volumes:
      - ./pgdata:/var/lib/postgresql/data
```

> Requires wiping the volume and recreating the DB.

- **Pragmatic (works quickly)**: In Synapse’s DB config, set `allow_unsafe_locale: true`. This bypasses the check. It’s fine for hobby use; for production, prefer the clean `C` cluster.

### 2) Start Synapse and generate config

```bash
docker compose up -d db synapse
# Logs
docker compose logs --tail=200 synapse
```

Ensure Synapse prints your **server_name** and **public base URL** and stays up.

### 3) Create an admin user

```bash
# Exec into the running Synapse container:
docker compose exec synapse register_new_matrix_user \
  -c /data/homeserver.yaml -u <username> -p <password> \
  -a -k
```

> If you see “Unknown execution mode”, you probably ran the binary with the wrong entrypoint. Use `docker compose exec synapse …` against the running container.

---

## TURN (coTURN)

### 1) Avoid bad inline comments

If you see errors like:

```
ERROR: Unknown boolean value: # log to journald/syslog. You can use on/off, yes/no, 1/0, true/false.
```

…it means a `#` comment is on the **same line** as a boolean directive. Move comments to their own lines.

### 2) Minimal `turnserver.conf`

```ini
listening-port=3478
tls-listening-port=5349
fingerprint
use-auth-secret
static-auth-secret=<shared-secret>   # also set in Synapse
realm=example.com                    # used in creds generation
total-quota=0
bps-capacity=0
cli-password=<admin-pass>
no-cli
cert=/etc/letsencrypt/live/turn.example.com/fullchain.pem
pkey=/etc/letsencrypt/live/turn.example.com/privkey.pem
# If behind NAT:
# external-ip=<public-ip>/<internal-ip>
```

### 3) Wire TURN into Synapse

In `homeserver.yaml`:

```yaml
turn_uris:
  - "turn:turn.example.com?transport=udp"
  - "turn:turn.example.com?transport=tcp"
  - "turns:turn.example.com:5349?transport=tcp"
turn_shared_secret: "<shared-secret>"
turn_user_lifetime: "1d"
```

Verify the homeserver issues TURN creds:

```bash
TOKEN='<your matrix access token>'
curl -s -H "Authorization: Bearer $TOKEN" \
  https://matrix.example.com/_matrix/client/v3/voip/turnServer | jq
```

You should see `uris`, a time-limited `username` and `password`.

---

## Nginx (host) for Synapse and federation

Create `/etc/nginx/conf.d/matrix.conf`:

```nginx
# Client traffic on 443
server {
  listen 443 ssl;  listen [::]:443 ssl;
  server_name matrix.example.com;

  ssl_certificate     /etc/letsencrypt/live/matrix.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/matrix.example.com/privkey.pem;

  # Proxy client & admin APIs to Synapse (container) on 8008
  location ~ ^(/_matrix|/_synapse/client) {
    proxy_pass http://127.0.0.1:8008;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Host $host;
    client_max_body_size 50M;
  }

  # Advertise homeserver base + RTC focus (MSC4143) via .well-known
  location = /.well-known/matrix/client {
    default_type application/json;
    add_header Access-Control-Allow-Origin "*" always;
    return 200 '{"m.homeserver":{"base_url":"https://matrix.example.com"},"org.matrix.msc4143.rtc_foci":[{"type":"livekit","livekit_service_url":"https://rtc.example.com"}]}';
  }
}

# Federation on 8448
server {
  listen 8448 ssl http2;  listen [::]:8448 ssl http2;
  server_name matrix.example.com;

  ssl_certificate     /etc/letsencrypt/live/matrix.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/matrix.example.com/privkey.pem;

  location / {
    proxy_pass http://127.0.0.1:8008;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Host $host;
    client_max_body_size 50M;
  }
}
```

Reload and sanity check:

```bash
sudo nginx -t && sudo systemctl reload nginx
curl -s https://matrix.example.com/.well-known/matrix/client | jq
```

Also check Synapse supports RTC signaling (MSC4140) so clients actually use it:

```bash
curl -s https://matrix.example.com/_matrix/client/versions | jq '.unstable_features."org.matrix.msc4140"'
# expect: true
```

---

## Element Call + LiveKit

I’ll run **LiveKit** and the small **JWT helper** (Element’s `elementcall_jwt`) in Docker. LiveKit handles media; the JWT helper mints access tokens for WebSocket connects.

### 1) LiveKit config (`/etc/livekit.yaml` inside container)

```yaml
port: 7880
bind_addresses: ["0.0.0.0"]
rtc:
  tcp_port: 7881
  port_range_start: 50100
  port_range_end: 50200
  use_external_ip: true
logging:
  level: info
turn:
  enabled: false
keys:
  lk_prod_1: "REPLACE_WITH_A_64_CHAR_RANDOM_SECRET___________________________________"
```

> **Important:** the secret must be **>= 32 chars**. The default `devkey` will trigger `secret is too short` warnings and won’t work with the JWT helper.

### 2) elementcall_jwt env

Run it with:

- `LIVEKIT_URL=wss://rtc.example.com`   _(root WS URL, **no** `/livekit/sfu` path)_
- `LIVEKIT_KEY=lk_prod_1`
- `LIVEKIT_SECRET=<the long secret above>`
- `LIVEKIT_JWT_PORT=8080` (internal HTTP port the proxy will hit)
- Optionally: `LIVEKIT_FULL_ACCESS_HOMESERVERS=*` during setup

Check logs on start; it prints the LIVEKIT_URL it will advertise.

### 3) Nginx (host) for `rtc.example.com`

Create `/etc/nginx/conf.d/rtc.conf`:

```nginx
server {
  listen 443 ssl;  listen [::]:443 ssl;
  server_name rtc.example.com;

  ssl_certificate     /etc/letsencrypt/live/rtc.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/rtc.example.com/privkey.pem;

  access_log /var/log/nginx/rtc.access.log combined;

  # 3a) JWT endpoint with clean CORS (avoid duplicate ACAO)
  location = /sfu/get {
    proxy_hide_header Access-Control-Allow-Origin;
    add_header Access-Control-Allow-Origin $http_origin always;
    add_header Vary "Origin" always;
    add_header Access-Control-Allow-Methods "POST, OPTIONS" always;
    add_header Access-Control-Allow-Headers "Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization" always;
    if ($request_method = OPTIONS) { return 204; }

    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_pass http://127.0.0.1:8070/sfu/get;  # elementcall_jwt
  }

  # 3b) LiveKit WS & HTTP (catch-all)
  location / {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;  # "livekit"
    proxy_set_header Origin $http_origin;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_buffering off;
    proxy_read_timeout 3600s;
    proxy_pass http://127.0.0.1:7880;         # livekit
  }
}
```

Reload and basic checks:

```bash
sudo nginx -t && sudo systemctl reload nginx
# JWT preflight (should return a single ACAO header)
curl -si -X OPTIONS https://rtc.example.com/sfu/get \
  -H 'Origin: https://app.element.io' \
  -H 'Access-Control-Request-Method: POST' \
  -H 'Access-Control-Request-Headers: authorization, content-type' | sed -n '1,30p'
# Expected: one Access-Control-Allow-Origin and 200/204
```

> **Why I did this:** I initially had **two** `Access-Control-Allow-Origin` headers (one added by the upstream, one by Nginx). Browsers reject that with “Access-Control-Allow-Origin cannot contain more than one origin”, so the JWT response never reached the client. Fixing CORS fixed everything.

---

## The “waiting for media” debugging story (how I found it)

Symptom: Element Call created rooms, but calls stayed on **“waiting for media.”**

What worked:
- `elementcall_jwt` could **CreateRoom** in LiveKit (seen in logs).
- TURN creds endpoint returned time-limited credentials.

What didn’t appear:
- No **HTTP 101** lines in `rtc.access.log` → the **browser never established a WebSocket** to LiveKit.

### Step 1: prove the WS vhost works (even without auth)

```bash
curl -v --http1.1 \
  -H 'Connection: Upgrade' -H 'Upgrade: websocket' \
  -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
  -H 'Sec-WebSocket-Version: 13' \
  https://rtc.example.com/rtc -o /dev/null
```
Result: I got a `401` (expected), but importantly I saw a log line in `rtc.access.log`. So **Nginx WS proxying was fine**.

### Step 2: check the browser console

The smoking gun in DevTools:

```
Access-Control-Allow-Origin cannot contain more than one origin.
Fetch API cannot load https://rtc.example.com/sfu/get due to access control checks.
```

The browser refused the JWT call due to duplicated **ACAO** headers, so no token → no WebSocket connect.

**Fix:** in Nginx I added:

```nginx
proxy_hide_header Access-Control-Allow-Origin;
add_header Access-Control-Allow-Origin $http_origin always;
add_header Vary "Origin" always;
```

…and ensured no other `add_header` created duplicates. After that, `/sfu/get` succeeded and the WebSocket to `wss://rtc.example.com` immediately followed (I saw **HTTP 101** in the logs).

### Step 3: confirm LiveKit side

Once the WS was up, LiveKit logs showed participants joining (not just `RoomService.CreateRoom`), and calls were established.

---

## Useful verification commands

```bash
# Synapse features (expect MSC4140 true)
curl -s https://matrix.example.com/_matrix/client/versions | jq '.unstable_features."org.matrix.msc4140"'

# Well-known with RTC focus
curl -s https://matrix.example.com/.well-known/matrix/client | jq

# TURN creds (with your access token)
curl -s -H "Authorization: Bearer $TOKEN" \
  https://matrix.example.com/_matrix/client/v3/voip/turnServer | jq

# JWT health
curl -si -X POST https://rtc.example.com/sfu/get

# LiveKit simple HTTP probe
curl -si https://rtc.example.com | head

# Nginx logs (look for 101 Switching Protocols when a call starts)
sudo tail -f /var/log/nginx/rtc.access.log | grep ' 101 '
```

---

## Common pitfalls (I hit these so you don’t have to)

- **Host Nginx vs Docker Nginx:** If you already run Nginx on the host, don’t also bind 80/443/8448 in a Docker Nginx — you’ll get `bind() ... already in use` and restart loops. Use **host** Nginx to reverse proxy to containers.
- **Nginx `http2` directive:** Old Nginx may not support the `http2` directive on `listen`. Use `listen 443 ssl;` (and add `http2` if your version supports it).
- **Certificate name mismatch:** Make sure `rtc.example.com`’s vhost uses a certificate **for that exact hostname** (initially I had the `matrix.*` cert on `rtc.*` and curl complained).
- **Postgres collation:** Either initialize the cluster with `C` or use `allow_unsafe_locale: true` in Synapse DB config to get running quickly.
- **CORS duplication on `/sfu/get`:** Only ONE `Access-Control-Allow-Origin` header. If the upstream adds it too, use `proxy_hide_header Access-Control-Allow-Origin;` on the Nginx location.
- **Cloudflare:** Use **DNS-only** for `rtc.*`. Proxies can interfere with WS and UDP paths.
- **Firewall:** Open the LiveKit UDP range and TURN ports on both v4 and v6.

---

## Final checklist (print me)

- [ ] `https://matrix.example.com/.well-known/matrix/client` returns **both** `m.homeserver.base_url` and `org.matrix.msc4143.rtc_foci` pointing to `https://rtc.example.com`.
- [ ] `/_matrix/client/versions` shows `"org.matrix.msc4140": true`.
- [ ] `/sfu/get` **preflight** returns **one** `Access-Control-Allow-Origin` and **200/204**.
- [ ] Starting a call creates **HTTP 101** entries to `wss://rtc.example.com` in `rtc.access.log`.
- [ ] LiveKit logs show **participants joining** (not just CreateRoom).
- [ ] `/voip/turnServer` returns time-limited TURN credentials.
- [ ] Cloudflare set to **DNS-only** for `rtc.*`. UFW allows 7881/tcp and your LiveKit UDP range.

---

### Credits & tooling

- Matrix Synapse, coTURN, LiveKit, Element Call.
- `curl`, `jq`, `docker compose logs`, Nginx access logs. These are your best friends.
- The debugging breakthrough was catching CORS errors in the browser console and looking for **HTTP 101** in Nginx logs.

Happy calling! 🎉

---

{{< sigdl >}}
