mysite/public-onion/sources/posts/matrix_setup.md

title	date	tags	draft	description	cover
Self-hosting Matrix + Element Call with LiveKit: from zero to working (and the [not so!!] fun debugging along the way)	2025-08-26	matrix synapse element-call livekit coturn nginx webrtc self-hosting debugging	false	How I set up a Matrix homeserver (Synapse) with TURN and Element Call using LiveKit, plus the exact debugging steps that took it from 'waiting for media' to solid calls.	image alt caption relative hidden /images/matrix-cover.png Matrix, Signal but distributed Distributed end-to-end encrypted chat platform true false

TL;DR: The call kept saying “waiting for media” because the browser never opened a WebSocket to LiveKit. The root cause was duplicate Access-Control-Allow-Origin headers on /sfu/get (CORS), which stopped the JWT response. Fixing CORS and ensuring the WS proxy worked (HTTP 101 in logs) solved it.

What I'm building

• A Matrix homeserver (Synapse) at matrix.example.com (replace with your domain).
• TURN/STUN (coTURN) for NAT traversal.
• Element Call backed by LiveKit, fronted by rtc.example.com.
• Nginx (host) as the single reverse proxy for everything.
• Cloudflare DNS (with rtc.* set to DNS-only, no orange cloud).
• UFW firewall opened for Matrix federation, TURN, and LiveKit media ports.

I used Docker for Synapse, PostgreSQL, LiveKit and the JWT helper. I used host Nginx (not Nginx in Docker) to avoid port binding conflicts on 80/443/8448.

---

Prereqs

• DNS A/AAAA:
  • matrix.example.com → your server (v4/v6)
  • rtc.example.com → your server (v4/v6)
• Certificates:
  • matrix.example.com and rtc.example.com via Let’s Encrypt on the host
• Cloudflare: DNS-only (grey cloud) for rtc.example.com so WebSockets & UDP work without interference.
• UFW / firewall open:
  • 80/tcp, 443/tcp
  • 8448/tcp (Matrix federation)
  • 3478/tcp, 3478/udp and 5349/tcp (TURN/TLS)
  • LiveKit: 7881/tcp and 50100–50200/udp (or your chosen range)
• Docker + docker compose installed.

---

Synapse + PostgreSQL

1) The PostgreSQL collation gotcha

Synapse prefers the database collation C. If your Postgres cluster was initialized with en_US.utf8, Synapse will error like:

Database has incorrect collation of 'en_US.utf8'. Should be 'C'

Two ways to resolve:

• Preferred (clean): Re-initialize the Postgres cluster with C and UTF-8:

# docker-compose.yml (excerpt for Postgres)
services:
  db:
    image: postgres:16
    environment:
      POSTGRES_DB: synapse
      POSTGRES_USER: synapse
      POSTGRES_PASSWORD: <strong-password>
      POSTGRES_INITDB_ARGS: "--locale=C --encoding=UTF8 --lc-collate=C --lc-ctype=C"
    volumes:
      - ./pgdata:/var/lib/postgresql/data

Requires wiping the volume and recreating the DB.

• Pragmatic (works quickly): In Synapse’s DB config, set allow_unsafe_locale: true. This bypasses the check. It’s fine for hobby use; for production, prefer the clean C cluster.

2) Start Synapse and generate config

docker compose up -d db synapse
# Logs
docker compose logs --tail=200 synapse

Ensure Synapse prints your server_name and public base URL and stays up.

3) Create an admin user

# Exec into the running Synapse container:
docker compose exec synapse register_new_matrix_user \
  -c /data/homeserver.yaml -u <username> -p <password> \
  -a -k

If you see “Unknown execution mode”, you probably ran the binary with the wrong entrypoint. Use docker compose exec synapse … against the running container.

---

TURN (coTURN)

1) Avoid bad inline comments

If you see errors like:

ERROR: Unknown boolean value: # log to journald/syslog. You can use on/off, yes/no, 1/0, true/false.

…it means a # comment is on the same line as a boolean directive. Move comments to their own lines.

2) Minimal turnserver.conf

listening-port=3478
tls-listening-port=5349
fingerprint
use-auth-secret
static-auth-secret=<shared-secret>   # also set in Synapse
realm=example.com                    # used in creds generation
total-quota=0
bps-capacity=0
cli-password=<admin-pass>
no-cli
cert=/etc/letsencrypt/live/turn.example.com/fullchain.pem
pkey=/etc/letsencrypt/live/turn.example.com/privkey.pem
# If behind NAT:
# external-ip=<public-ip>/<internal-ip>

3) Wire TURN into Synapse

In homeserver.yaml:

turn_uris:
  - "turn:turn.example.com?transport=udp"
  - "turn:turn.example.com?transport=tcp"
  - "turns:turn.example.com:5349?transport=tcp"
turn_shared_secret: "<shared-secret>"
turn_user_lifetime: "1d"

Verify the homeserver issues TURN creds:

TOKEN='<your matrix access token>'
curl -s -H "Authorization: Bearer $TOKEN" \
  https://matrix.example.com/_matrix/client/v3/voip/turnServer | jq

You should see uris, a time-limited username and password.

---

Nginx (host) for Synapse and federation

Create /etc/nginx/conf.d/matrix.conf:

# Client traffic on 443
server {
  listen 443 ssl;  listen [::]:443 ssl;
  server_name matrix.example.com;

  ssl_certificate     /etc/letsencrypt/live/matrix.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/matrix.example.com/privkey.pem;

  # Proxy client & admin APIs to Synapse (container) on 8008
  location ~ ^(/_matrix|/_synapse/client) {
    proxy_pass http://127.0.0.1:8008;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Host $host;
    client_max_body_size 50M;
  }

  # Advertise homeserver base + RTC focus (MSC4143) via .well-known
  location = /.well-known/matrix/client {
    default_type application/json;
    add_header Access-Control-Allow-Origin "*" always;
    return 200 '{"m.homeserver":{"base_url":"https://matrix.example.com"},"org.matrix.msc4143.rtc_foci":[{"type":"livekit","livekit_service_url":"https://rtc.example.com"}]}';
  }
}

# Federation on 8448
server {
  listen 8448 ssl http2;  listen [::]:8448 ssl http2;
  server_name matrix.example.com;

  ssl_certificate     /etc/letsencrypt/live/matrix.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/matrix.example.com/privkey.pem;

  location / {
    proxy_pass http://127.0.0.1:8008;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Host $host;
    client_max_body_size 50M;
  }
}

Reload and sanity check:

sudo nginx -t && sudo systemctl reload nginx
curl -s https://matrix.example.com/.well-known/matrix/client | jq

Also check Synapse supports RTC signaling (MSC4140) so clients actually use it:

curl -s https://matrix.example.com/_matrix/client/versions | jq '.unstable_features."org.matrix.msc4140"'
# expect: true

---

Element Call + LiveKit

I’ll run LiveKit and the small JWT helper (Element’s elementcall_jwt) in Docker. LiveKit handles media; the JWT helper mints access tokens for WebSocket connects.

1) LiveKit config (/etc/livekit.yaml inside container)

port: 7880
bind_addresses: ["0.0.0.0"]
rtc:
  tcp_port: 7881
  port_range_start: 50100
  port_range_end: 50200
  use_external_ip: true
logging:
  level: info
turn:
  enabled: false
keys:
  lk_prod_1: "REPLACE_WITH_A_64_CHAR_RANDOM_SECRET___________________________________"

Important: the secret must be >= 32 chars. The default devkey will trigger secret is too short warnings and won’t work with the JWT helper.

2) elementcall_jwt env

Run it with:

• LIVEKIT_URL=wss://rtc.example.com (root WS URL, no /livekit/sfu path)
• LIVEKIT_KEY=lk_prod_1
• LIVEKIT_SECRET=<the long secret above>
• LIVEKIT_JWT_PORT=8080 (internal HTTP port the proxy will hit)
• Optionally: LIVEKIT_FULL_ACCESS_HOMESERVERS=* during setup

Check logs on start; it prints the LIVEKIT_URL it will advertise.

3) Nginx (host) for rtc.example.com

Create /etc/nginx/conf.d/rtc.conf:

server {
  listen 443 ssl;  listen [::]:443 ssl;
  server_name rtc.example.com;

  ssl_certificate     /etc/letsencrypt/live/rtc.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/rtc.example.com/privkey.pem;

  access_log /var/log/nginx/rtc.access.log combined;

  # 3a) JWT endpoint with clean CORS (avoid duplicate ACAO)
  location = /sfu/get {
    proxy_hide_header Access-Control-Allow-Origin;
    add_header Access-Control-Allow-Origin $http_origin always;
    add_header Vary "Origin" always;
    add_header Access-Control-Allow-Methods "POST, OPTIONS" always;
    add_header Access-Control-Allow-Headers "Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization" always;
    if ($request_method = OPTIONS) { return 204; }

    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_pass http://127.0.0.1:8070/sfu/get;  # elementcall_jwt
  }

  # 3b) LiveKit WS & HTTP (catch-all)
  location / {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;  # "livekit"
    proxy_set_header Origin $http_origin;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_buffering off;
    proxy_read_timeout 3600s;
    proxy_pass http://127.0.0.1:7880;         # livekit
  }
}

Reload and basic checks:

sudo nginx -t && sudo systemctl reload nginx
# JWT preflight (should return a single ACAO header)
curl -si -X OPTIONS https://rtc.example.com/sfu/get \
  -H 'Origin: https://app.element.io' \
  -H 'Access-Control-Request-Method: POST' \
  -H 'Access-Control-Request-Headers: authorization, content-type' | sed -n '1,30p'
# Expected: one Access-Control-Allow-Origin and 200/204

Why I did this: I initially had two Access-Control-Allow-Origin headers (one added by the upstream, one by Nginx). Browsers reject that with “Access-Control-Allow-Origin cannot contain more than one origin”, so the JWT response never reached the client. Fixing CORS fixed everything.

---

The “waiting for media” debugging story (how I found it)

Symptom: Element Call created rooms, but calls stayed on “waiting for media.”

What worked:

• elementcall_jwt could CreateRoom in LiveKit (seen in logs).
• TURN creds endpoint returned time-limited credentials.

What didn’t appear:

• No HTTP 101 lines in rtc.access.log → the browser never established a WebSocket to LiveKit.

Step 1: prove the WS vhost works (even without auth)

curl -v --http1.1 \
  -H 'Connection: Upgrade' -H 'Upgrade: websocket' \
  -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
  -H 'Sec-WebSocket-Version: 13' \
  https://rtc.example.com/rtc -o /dev/null

Result: I got a 401 (expected), but importantly I saw a log line in rtc.access.log. So Nginx WS proxying was fine.

Step 2: check the browser console

The smoking gun in DevTools:

Access-Control-Allow-Origin cannot contain more than one origin.
Fetch API cannot load https://rtc.example.com/sfu/get due to access control checks.

The browser refused the JWT call due to duplicated ACAO headers, so no token → no WebSocket connect.

Fix: in Nginx I added:

proxy_hide_header Access-Control-Allow-Origin;
add_header Access-Control-Allow-Origin $http_origin always;
add_header Vary "Origin" always;

…and ensured no other add_header created duplicates. After that, /sfu/get succeeded and the WebSocket to wss://rtc.example.com immediately followed (I saw HTTP 101 in the logs).

Step 3: confirm LiveKit side

Once the WS was up, LiveKit logs showed participants joining (not just RoomService.CreateRoom), and calls were established.

---

Useful verification commands

# Synapse features (expect MSC4140 true)
curl -s https://matrix.example.com/_matrix/client/versions | jq '.unstable_features."org.matrix.msc4140"'

# Well-known with RTC focus
curl -s https://matrix.example.com/.well-known/matrix/client | jq

# TURN creds (with your access token)
curl -s -H "Authorization: Bearer $TOKEN" \
  https://matrix.example.com/_matrix/client/v3/voip/turnServer | jq

# JWT health
curl -si -X POST https://rtc.example.com/sfu/get

# LiveKit simple HTTP probe
curl -si https://rtc.example.com | head

# Nginx logs (look for 101 Switching Protocols when a call starts)
sudo tail -f /var/log/nginx/rtc.access.log | grep ' 101 '

---

Common pitfalls (I hit these so you don’t have to)

• Host Nginx vs Docker Nginx: If you already run Nginx on the host, don’t also bind 80/443/8448 in a Docker Nginx — you’ll get bind() ... already in use and restart loops. Use host Nginx to reverse proxy to containers.
• Nginx http2 directive: Old Nginx may not support the http2 directive on listen. Use listen 443 ssl; (and add http2 if your version supports it).
• Certificate name mismatch: Make sure rtc.example.com’s vhost uses a certificate for that exact hostname (initially I had the matrix.* cert on rtc.* and curl complained).
• Postgres collation: Either initialize the cluster with C or use allow_unsafe_locale: true in Synapse DB config to get running quickly.
• CORS duplication on /sfu/get: Only ONE Access-Control-Allow-Origin header. If the upstream adds it too, use proxy_hide_header Access-Control-Allow-Origin; on the Nginx location.
• Cloudflare: Use DNS-only for rtc.*. Proxies can interfere with WS and UDP paths.
• Firewall: Open the LiveKit UDP range and TURN ports on both v4 and v6.

---

Final checklist (print me)

• https://matrix.example.com/.well-known/matrix/client returns both m.homeserver.base_url and org.matrix.msc4143.rtc_foci pointing to https://rtc.example.com.
• /_matrix/client/versions shows "org.matrix.msc4140": true.
• /sfu/get preflight returns one Access-Control-Allow-Origin and 200/204.
• Starting a call creates HTTP 101 entries to wss://rtc.example.com in rtc.access.log.
• LiveKit logs show participants joining (not just CreateRoom).
• /voip/turnServer returns time-limited TURN credentials.
• Cloudflare set to DNS-only for rtc.*. UFW allows 7881/tcp and your LiveKit UDP range.

---

Credits & tooling

• Matrix Synapse, coTURN, LiveKit, Element Call.
• curl, jq, docker compose logs, Nginx access logs. These are your best friends.
• The debugging breakthrough was catching CORS errors in the browser console and looking for HTTP 101 in Nginx logs.

Happy calling! 🎉

---

{{< sigdl >}}