Initial import of Hugo site to Forgejo
This commit is contained in:
commit
cb1ba0317f
1259 changed files with 236349 additions and 0 deletions
426
public-clearnet/sources/posts/matrix_setup.md
Normal file
426
public-clearnet/sources/posts/matrix_setup.md
Normal file
|
|
@ -0,0 +1,426 @@
|
|||
---
|
||||
title: "Self-hosting Matrix + Element Call with LiveKit: from zero to working (and the [not so!!] fun debugging along the way)"
|
||||
date: 2025-08-26
|
||||
tags: ["matrix", "synapse", "element-call", "livekit", "coturn", "nginx", "webrtc", "self-hosting", "debugging"]
|
||||
draft: false
|
||||
description: "How I set up a Matrix homeserver (Synapse) with TURN and Element Call using LiveKit, plus the exact debugging steps that took it from 'waiting for media' to solid calls."
|
||||
cover:
|
||||
image: "/images/matrix-cover.png"
|
||||
alt: "Matrix, Signal but distributed"
|
||||
caption: "Distributed end-to-end encrypted chat platform"
|
||||
relative: true
|
||||
hidden: false
|
||||
---
|
||||
|
||||
> TL;DR: The call kept saying **“waiting for media”** because the browser never opened a WebSocket to LiveKit. The root cause was **duplicate `Access-Control-Allow-Origin` headers** on `/sfu/get` (CORS), which stopped the JWT response. Fixing CORS and ensuring the WS proxy worked (HTTP **101** in logs) solved it.
|
||||
|
||||
## What I'm building
|
||||
|
||||
- A **Matrix homeserver** (Synapse) at `matrix.example.com` (replace with your domain).
|
||||
- **TURN/STUN** (coTURN) for NAT traversal.
|
||||
- **Element Call** backed by **LiveKit**, fronted by `rtc.example.com`.
|
||||
- **Nginx (host)** as the single reverse proxy for everything.
|
||||
- **Cloudflare** DNS (with `rtc.*` set to **DNS-only**, no orange cloud).
|
||||
- **UFW** firewall opened for Matrix federation, TURN, and LiveKit media ports.
|
||||
|
||||
> I used Docker for Synapse, PostgreSQL, LiveKit and the JWT helper. I used **host** Nginx (not Nginx in Docker) to avoid port binding conflicts on 80/443/8448.
|
||||
|
||||
---
|
||||
|
||||
## Prereqs
|
||||
|
||||
- DNS A/AAAA:
|
||||
- `matrix.example.com` → your server (v4/v6)
|
||||
- `rtc.example.com` → your server (v4/v6)
|
||||
- Certificates:
|
||||
- `matrix.example.com` and `rtc.example.com` via Let’s Encrypt on the host
|
||||
- Cloudflare: **DNS-only (grey cloud)** for `rtc.example.com` so WebSockets & UDP work without interference.
|
||||
- UFW / firewall open:
|
||||
- 80/tcp, 443/tcp
|
||||
- 8448/tcp (Matrix federation)
|
||||
- 3478/tcp, 3478/udp and **5349/tcp** (TURN/TLS)
|
||||
- **LiveKit**: 7881/tcp and **50100–50200/udp** (or your chosen range)
|
||||
- Docker + docker compose installed.
|
||||
|
||||
---
|
||||
|
||||
## Synapse + PostgreSQL
|
||||
|
||||
### 1) The PostgreSQL collation gotcha
|
||||
|
||||
Synapse prefers the database collation **`C`**. If your Postgres cluster was initialized with `en_US.utf8`, Synapse will error like:
|
||||
|
||||
```
|
||||
Database has incorrect collation of 'en_US.utf8'. Should be 'C'
|
||||
```
|
||||
|
||||
**Two ways to resolve:**
|
||||
|
||||
- **Preferred (clean)**: Re-initialize the Postgres **cluster** with `C` and `UTF-8`:
|
||||
|
||||
```yaml
|
||||
# docker-compose.yml (excerpt for Postgres)
|
||||
services:
|
||||
db:
|
||||
image: postgres:16
|
||||
environment:
|
||||
POSTGRES_DB: synapse
|
||||
POSTGRES_USER: synapse
|
||||
POSTGRES_PASSWORD: <strong-password>
|
||||
POSTGRES_INITDB_ARGS: "--locale=C --encoding=UTF8 --lc-collate=C --lc-ctype=C"
|
||||
volumes:
|
||||
- ./pgdata:/var/lib/postgresql/data
|
||||
```
|
||||
|
||||
> Requires wiping the volume and recreating the DB.
|
||||
|
||||
- **Pragmatic (works quickly)**: In Synapse’s DB config, set `allow_unsafe_locale: true`. This bypasses the check. It’s fine for hobby use; for production, prefer the clean `C` cluster.
|
||||
|
||||
### 2) Start Synapse and generate config
|
||||
|
||||
```bash
|
||||
docker compose up -d db synapse
|
||||
# Logs
|
||||
docker compose logs --tail=200 synapse
|
||||
```
|
||||
|
||||
Ensure Synapse prints your **server_name** and **public base URL** and stays up.
|
||||
|
||||
### 3) Create an admin user
|
||||
|
||||
```bash
|
||||
# Exec into the running Synapse container:
|
||||
docker compose exec synapse register_new_matrix_user \
|
||||
-c /data/homeserver.yaml -u <username> -p <password> \
|
||||
-a -k
|
||||
```
|
||||
|
||||
> If you see “Unknown execution mode”, you probably ran the binary with the wrong entrypoint. Use `docker compose exec synapse …` against the running container.
|
||||
|
||||
---
|
||||
|
||||
## TURN (coTURN)
|
||||
|
||||
### 1) Avoid bad inline comments
|
||||
|
||||
If you see errors like:
|
||||
|
||||
```
|
||||
ERROR: Unknown boolean value: # log to journald/syslog. You can use on/off, yes/no, 1/0, true/false.
|
||||
```
|
||||
|
||||
…it means a `#` comment is on the **same line** as a boolean directive. Move comments to their own lines.
|
||||
|
||||
### 2) Minimal `turnserver.conf`
|
||||
|
||||
```ini
|
||||
listening-port=3478
|
||||
tls-listening-port=5349
|
||||
fingerprint
|
||||
use-auth-secret
|
||||
static-auth-secret=<shared-secret> # also set in Synapse
|
||||
realm=example.com # used in creds generation
|
||||
total-quota=0
|
||||
bps-capacity=0
|
||||
cli-password=<admin-pass>
|
||||
no-cli
|
||||
cert=/etc/letsencrypt/live/turn.example.com/fullchain.pem
|
||||
pkey=/etc/letsencrypt/live/turn.example.com/privkey.pem
|
||||
# If behind NAT:
|
||||
# external-ip=<public-ip>/<internal-ip>
|
||||
```
|
||||
|
||||
### 3) Wire TURN into Synapse
|
||||
|
||||
In `homeserver.yaml`:
|
||||
|
||||
```yaml
|
||||
turn_uris:
|
||||
- "turn:turn.example.com?transport=udp"
|
||||
- "turn:turn.example.com?transport=tcp"
|
||||
- "turns:turn.example.com:5349?transport=tcp"
|
||||
turn_shared_secret: "<shared-secret>"
|
||||
turn_user_lifetime: "1d"
|
||||
```
|
||||
|
||||
Verify the homeserver issues TURN creds:
|
||||
|
||||
```bash
|
||||
TOKEN='<your matrix access token>'
|
||||
curl -s -H "Authorization: Bearer $TOKEN" \
|
||||
https://matrix.example.com/_matrix/client/v3/voip/turnServer | jq
|
||||
```
|
||||
|
||||
You should see `uris`, a time-limited `username` and `password`.
|
||||
|
||||
---
|
||||
|
||||
## Nginx (host) for Synapse and federation
|
||||
|
||||
Create `/etc/nginx/conf.d/matrix.conf`:
|
||||
|
||||
```nginx
|
||||
# Client traffic on 443
|
||||
server {
|
||||
listen 443 ssl; listen [::]:443 ssl;
|
||||
server_name matrix.example.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/matrix.example.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/matrix.example.com/privkey.pem;
|
||||
|
||||
# Proxy client & admin APIs to Synapse (container) on 8008
|
||||
location ~ ^(/_matrix|/_synapse/client) {
|
||||
proxy_pass http://127.0.0.1:8008;
|
||||
proxy_set_header X-Forwarded-For $remote_addr;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Host $host;
|
||||
client_max_body_size 50M;
|
||||
}
|
||||
|
||||
# Advertise homeserver base + RTC focus (MSC4143) via .well-known
|
||||
location = /.well-known/matrix/client {
|
||||
default_type application/json;
|
||||
add_header Access-Control-Allow-Origin "*" always;
|
||||
return 200 '{"m.homeserver":{"base_url":"https://matrix.example.com"},"org.matrix.msc4143.rtc_foci":[{"type":"livekit","livekit_service_url":"https://rtc.example.com"}]}';
|
||||
}
|
||||
}
|
||||
|
||||
# Federation on 8448
|
||||
server {
|
||||
listen 8448 ssl http2; listen [::]:8448 ssl http2;
|
||||
server_name matrix.example.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/matrix.example.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/matrix.example.com/privkey.pem;
|
||||
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:8008;
|
||||
proxy_set_header X-Forwarded-For $remote_addr;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Host $host;
|
||||
client_max_body_size 50M;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Reload and sanity check:
|
||||
|
||||
```bash
|
||||
sudo nginx -t && sudo systemctl reload nginx
|
||||
curl -s https://matrix.example.com/.well-known/matrix/client | jq
|
||||
```
|
||||
|
||||
Also check Synapse supports RTC signaling (MSC4140) so clients actually use it:
|
||||
|
||||
```bash
|
||||
curl -s https://matrix.example.com/_matrix/client/versions | jq '.unstable_features."org.matrix.msc4140"'
|
||||
# expect: true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Element Call + LiveKit
|
||||
|
||||
I’ll run **LiveKit** and the small **JWT helper** (Element’s `elementcall_jwt`) in Docker. LiveKit handles media; the JWT helper mints access tokens for WebSocket connects.
|
||||
|
||||
### 1) LiveKit config (`/etc/livekit.yaml` inside container)
|
||||
|
||||
```yaml
|
||||
port: 7880
|
||||
bind_addresses: ["0.0.0.0"]
|
||||
rtc:
|
||||
tcp_port: 7881
|
||||
port_range_start: 50100
|
||||
port_range_end: 50200
|
||||
use_external_ip: true
|
||||
logging:
|
||||
level: info
|
||||
turn:
|
||||
enabled: false
|
||||
keys:
|
||||
lk_prod_1: "REPLACE_WITH_A_64_CHAR_RANDOM_SECRET___________________________________"
|
||||
```
|
||||
|
||||
> **Important:** the secret must be **>= 32 chars**. The default `devkey` will trigger `secret is too short` warnings and won’t work with the JWT helper.
|
||||
|
||||
### 2) elementcall_jwt env
|
||||
|
||||
Run it with:
|
||||
|
||||
- `LIVEKIT_URL=wss://rtc.example.com` _(root WS URL, **no** `/livekit/sfu` path)_
|
||||
- `LIVEKIT_KEY=lk_prod_1`
|
||||
- `LIVEKIT_SECRET=<the long secret above>`
|
||||
- `LIVEKIT_JWT_PORT=8080` (internal HTTP port the proxy will hit)
|
||||
- Optionally: `LIVEKIT_FULL_ACCESS_HOMESERVERS=*` during setup
|
||||
|
||||
Check logs on start; it prints the LIVEKIT_URL it will advertise.
|
||||
|
||||
### 3) Nginx (host) for `rtc.example.com`
|
||||
|
||||
Create `/etc/nginx/conf.d/rtc.conf`:
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 443 ssl; listen [::]:443 ssl;
|
||||
server_name rtc.example.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/rtc.example.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/rtc.example.com/privkey.pem;
|
||||
|
||||
access_log /var/log/nginx/rtc.access.log combined;
|
||||
|
||||
# 3a) JWT endpoint with clean CORS (avoid duplicate ACAO)
|
||||
location = /sfu/get {
|
||||
proxy_hide_header Access-Control-Allow-Origin;
|
||||
add_header Access-Control-Allow-Origin $http_origin always;
|
||||
add_header Vary "Origin" always;
|
||||
add_header Access-Control-Allow-Methods "POST, OPTIONS" always;
|
||||
add_header Access-Control-Allow-Headers "Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization" always;
|
||||
if ($request_method = OPTIONS) { return 204; }
|
||||
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_pass http://127.0.0.1:8070/sfu/get; # elementcall_jwt
|
||||
}
|
||||
|
||||
# 3b) LiveKit WS & HTTP (catch-all)
|
||||
location / {
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol; # "livekit"
|
||||
proxy_set_header Origin $http_origin;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_buffering off;
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_pass http://127.0.0.1:7880; # livekit
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Reload and basic checks:
|
||||
|
||||
```bash
|
||||
sudo nginx -t && sudo systemctl reload nginx
|
||||
# JWT preflight (should return a single ACAO header)
|
||||
curl -si -X OPTIONS https://rtc.example.com/sfu/get \
|
||||
-H 'Origin: https://app.element.io' \
|
||||
-H 'Access-Control-Request-Method: POST' \
|
||||
-H 'Access-Control-Request-Headers: authorization, content-type' | sed -n '1,30p'
|
||||
# Expected: one Access-Control-Allow-Origin and 200/204
|
||||
```
|
||||
|
||||
> **Why I did this:** I initially had **two** `Access-Control-Allow-Origin` headers (one added by the upstream, one by Nginx). Browsers reject that with “Access-Control-Allow-Origin cannot contain more than one origin”, so the JWT response never reached the client. Fixing CORS fixed everything.
|
||||
|
||||
---
|
||||
|
||||
## The “waiting for media” debugging story (how I found it)
|
||||
|
||||
Symptom: Element Call created rooms, but calls stayed on **“waiting for media.”**
|
||||
|
||||
What worked:
|
||||
- `elementcall_jwt` could **CreateRoom** in LiveKit (seen in logs).
|
||||
- TURN creds endpoint returned time-limited credentials.
|
||||
|
||||
What didn’t appear:
|
||||
- No **HTTP 101** lines in `rtc.access.log` → the **browser never established a WebSocket** to LiveKit.
|
||||
|
||||
### Step 1: prove the WS vhost works (even without auth)
|
||||
|
||||
```bash
|
||||
curl -v --http1.1 \
|
||||
-H 'Connection: Upgrade' -H 'Upgrade: websocket' \
|
||||
-H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
|
||||
-H 'Sec-WebSocket-Version: 13' \
|
||||
https://rtc.example.com/rtc -o /dev/null
|
||||
```
|
||||
Result: I got a `401` (expected), but importantly I saw a log line in `rtc.access.log`. So **Nginx WS proxying was fine**.
|
||||
|
||||
### Step 2: check the browser console
|
||||
|
||||
The smoking gun in DevTools:
|
||||
|
||||
```
|
||||
Access-Control-Allow-Origin cannot contain more than one origin.
|
||||
Fetch API cannot load https://rtc.example.com/sfu/get due to access control checks.
|
||||
```
|
||||
|
||||
The browser refused the JWT call due to duplicated **ACAO** headers, so no token → no WebSocket connect.
|
||||
|
||||
**Fix:** in Nginx I added:
|
||||
|
||||
```nginx
|
||||
proxy_hide_header Access-Control-Allow-Origin;
|
||||
add_header Access-Control-Allow-Origin $http_origin always;
|
||||
add_header Vary "Origin" always;
|
||||
```
|
||||
|
||||
…and ensured no other `add_header` created duplicates. After that, `/sfu/get` succeeded and the WebSocket to `wss://rtc.example.com` immediately followed (I saw **HTTP 101** in the logs).
|
||||
|
||||
### Step 3: confirm LiveKit side
|
||||
|
||||
Once the WS was up, LiveKit logs showed participants joining (not just `RoomService.CreateRoom`), and calls were established.
|
||||
|
||||
---
|
||||
|
||||
## Useful verification commands
|
||||
|
||||
```bash
|
||||
# Synapse features (expect MSC4140 true)
|
||||
curl -s https://matrix.example.com/_matrix/client/versions | jq '.unstable_features."org.matrix.msc4140"'
|
||||
|
||||
# Well-known with RTC focus
|
||||
curl -s https://matrix.example.com/.well-known/matrix/client | jq
|
||||
|
||||
# TURN creds (with your access token)
|
||||
curl -s -H "Authorization: Bearer $TOKEN" \
|
||||
https://matrix.example.com/_matrix/client/v3/voip/turnServer | jq
|
||||
|
||||
# JWT health
|
||||
curl -si -X POST https://rtc.example.com/sfu/get
|
||||
|
||||
# LiveKit simple HTTP probe
|
||||
curl -si https://rtc.example.com | head
|
||||
|
||||
# Nginx logs (look for 101 Switching Protocols when a call starts)
|
||||
sudo tail -f /var/log/nginx/rtc.access.log | grep ' 101 '
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common pitfalls (I hit these so you don’t have to)
|
||||
|
||||
- **Host Nginx vs Docker Nginx:** If you already run Nginx on the host, don’t also bind 80/443/8448 in a Docker Nginx — you’ll get `bind() ... already in use` and restart loops. Use **host** Nginx to reverse proxy to containers.
|
||||
- **Nginx `http2` directive:** Old Nginx may not support the `http2` directive on `listen`. Use `listen 443 ssl;` (and add `http2` if your version supports it).
|
||||
- **Certificate name mismatch:** Make sure `rtc.example.com`’s vhost uses a certificate **for that exact hostname** (initially I had the `matrix.*` cert on `rtc.*` and curl complained).
|
||||
- **Postgres collation:** Either initialize the cluster with `C` or use `allow_unsafe_locale: true` in Synapse DB config to get running quickly.
|
||||
- **CORS duplication on `/sfu/get`:** Only ONE `Access-Control-Allow-Origin` header. If the upstream adds it too, use `proxy_hide_header Access-Control-Allow-Origin;` on the Nginx location.
|
||||
- **Cloudflare:** Use **DNS-only** for `rtc.*`. Proxies can interfere with WS and UDP paths.
|
||||
- **Firewall:** Open the LiveKit UDP range and TURN ports on both v4 and v6.
|
||||
|
||||
---
|
||||
|
||||
## Final checklist (print me)
|
||||
|
||||
- [ ] `https://matrix.example.com/.well-known/matrix/client` returns **both** `m.homeserver.base_url` and `org.matrix.msc4143.rtc_foci` pointing to `https://rtc.example.com`.
|
||||
- [ ] `/_matrix/client/versions` shows `"org.matrix.msc4140": true`.
|
||||
- [ ] `/sfu/get` **preflight** returns **one** `Access-Control-Allow-Origin` and **200/204**.
|
||||
- [ ] Starting a call creates **HTTP 101** entries to `wss://rtc.example.com` in `rtc.access.log`.
|
||||
- [ ] LiveKit logs show **participants joining** (not just CreateRoom).
|
||||
- [ ] `/voip/turnServer` returns time-limited TURN credentials.
|
||||
- [ ] Cloudflare set to **DNS-only** for `rtc.*`. UFW allows 7881/tcp and your LiveKit UDP range.
|
||||
|
||||
---
|
||||
|
||||
### Credits & tooling
|
||||
|
||||
- Matrix Synapse, coTURN, LiveKit, Element Call.
|
||||
- `curl`, `jq`, `docker compose logs`, Nginx access logs. These are your best friends.
|
||||
- The debugging breakthrough was catching CORS errors in the browser console and looking for **HTTP 101** in Nginx logs.
|
||||
|
||||
Happy calling! 🎉
|
||||
|
||||
---
|
||||
|
||||
{{< sigdl >}}
|
||||
Loading…
Add table
Add a link
Reference in a new issue