nerd lab

How mochela.it is built, with the names of things. For anyone who wants to check, copy, criticise or set up a similar server.

What is written here is really running; what is not running yet is marked as planned.

Source code

• All the code (Ansible playbooks and roles, laptop scripts, operations guide) is published at mochela.it/codice, as static, read-only pages: files and history. The full repository is available on request at forgy@mochela.it, as a git bundle signed with OpenPGP.
• The working repository lives on our own Forgejo, on mx2: it listens only on 127.0.0.1, and the administrator reaches it through the admin tunnel. Nobody on the Internet talks to Forgejo: the public pages are generated by the laptop at every release.
• Licences: AGPL-3.0-or-later for the code and CC BY-SA 4.0 for the texts, declared file by file following REUSE. Every release is a cleaned copy (no Vault, personal variables, node addresses or provider names), checked with gitleaks.
• Written largely with an AI assistant (Claude, by Anthropic), under the direction of the person who runs mochela, and tested on the production cluster: how it was made.

Nodes and network

• KVM VPS running Debian 13 (trixie); packages from a European mirror.
• Full-mesh WireGuard private network between the three nodes. Database, storage, coordination and checks travel only through the tunnel; none of those ports is open to the Internet. Every pair of nodes has an additional shared key (PresharedKey, 256-bit): whoever records the traffic today will not be able to open it tomorrow with a quantum computer active
• nftables firewall with an inbound drop policy. Open: 22 (SSH), 80 and 443 (web); on the mail nodes also 25, 465, 587, 993 (mail) and 5223 (SimpleX); on mx1 also 1194 TCP and UDP (Galène's TURN) and UDP 20000-20999 (video call audio and video).
• Outgoing mail over IPv4 only, with a PTR matching the node name IPv6 planned

Replicated data

• etcd with three members: the quorum. It decides which node is the database primary; it survives the loss of any one node.
• PostgreSQL managed by Patroni: one primary and two replicas, automatic failover if the primary goes down. On each mail node HAProxy always sends Stalwart to the current primary.
• Garage (S3-compatible object storage, by Deuxfleurs): message bodies and attachments, one copy per zone, three zones, one per provider.
• Valkey with three Sentinels: temporary data and coordination between the two Stalwart instances.

Mail

• Stalwart 0.16 on mx1 and mx2, both active, as a single server: configuration in PostgreSQL, messages in Garage, coordination in Valkey. SMTP, IMAP, JMAP.
• Client ports: IMAP 993 and submission 465, both with implicit TLS; JMAP and /account over HTTPS. No POP3.
• SRV records (RFC 6186, RFC 8314) for _imaps, _submissions and _jmap; autoconfiguration file in /.well-known/autoconfig/.
• SPF, DKIM with dual signatures (RSA and Ed25519), DMARC with p=reject: mail pretending to come from mochela.it without a valid signature is rejected. Aggregate reports (rua) to postmaster@: who sent mail in the name of mochela.it, day by day.
• In mail sent by an authenticated client the server does not add the Received line with the client's IP.
• Stalwart logs: one file per hour. Every hour a timer removes from closed files the IP addresses and the local part of email addresses (the domain stays: xxx@gmail.com), and deletes files after seven days. Only the cluster's internal addresses stay in clear. A stalwart-log command for searches; the monitor raises an alert if a file with addresses is older than two hours.
• Encryption on arrival with the user's public OpenPGP key (a Stalwart feature, optional for each account). Keys published through the Web Key Directory, direct method on mochela.it, only after checking that they come from the mailbox concerned; the published file keeps only the @mochela.it uid (export-minimal).
• Anti-spam: only European IP-type DNSBLs (Spamhaus, blocklist.de, VirusFree), which receive only the address of the sending server. Lists on domains, URLs and addresses switched off; Pyzor switched off (it would send message fingerprints); the LLM classifier switched off. The local classifier teaches itself that a mail is spam only if the sending server is on at least three lists.
• DANE outbound: towards servers that publish DNSSEC-signed TLSA records, mail leaves only over TLS and with the matching certificate, retries after an error included (no fallback strategy turns DANE or MTA-STS off). DNSSEC validated by Stalwart through the node's Unbound, over TCP. Inbound: mochela.it zone signed with DNSSEC and 3 1 1 TLSA records for _25._tcp.posta; a single certificate on port 25, same key across renewals (reuseKey), and a check every five minutes that the key presented matches the published records.
• A single MX, posta.mochela.it, pointing to mx1 and mx2: the same name as the port 25 certificate (the free Actalis plan gives one name per certificate). The node names remain for reverse DNS and the greeting of outgoing mail.
• MTA-STS (RFC 8461) for senders that do not use DANE, such as Gmail and Microsoft: policy on mta-sts.mochela.it in enforce mode since 8 October 2026 (testing from 1 October): senders that honour it never deliver to us in clear text active
• TLS-RPT (RFC 8460): servers that write to us send a daily report on their encrypted connections to us. Reports go to postmaster@ and are read only here.
• /.well-known/security.txt (RFC 9116): who to contact about a security issue.
• External check with MECSA (European Commission Joint Research Centre), 1 October 2026: 100% on STARTTLS, X.509, SPF, DKIM, DMARC, DNSSEC, DANE and MTA-STS. Report a96b6500a4131f74d944a6a05f6e4720 (how to check it).

End-to-end and zero-access encryption

Two mechanisms, both with OpenPGP, that complement each other.

• With OpenPGP between users the content is zero-knowledge for the server: body and attachments, and the subject if the program uses protected headers (in Thunderbird: Encrypt subject).
• The metadata stays visible: sender, recipients, date, size, the servers along the way. It is needed to deliver mail, and no mail encryption hides it.
• Public keys are found through the Web Key Directory (/.well-known/openpgpkey/, direct method): Thunderbird and GnuPG look them up by themselves. Secret keys never pass through the servers.
• With people who do not use OpenPGP there is no end-to-end encryption: Stalwart does not have the "link and password" message some commercial services offer. For those messages, encryption on arrival remains.

SimpleX

• Two independent SMP servers, smp1.mochela.it on mx1 and smp2.mochela.it on mx2, port 5223. SMP is not clustered: redundancy comes from the app, which uses several servers together, and with private routing the recipient's server does not see the sender's IP.
• No users on the server: no accounts, numbers or names. The server sees only anonymous queues.
• Queue creation protected by a password, given to people with a mochela mailbox; anyone writing to a mochela user does not need it.
• Each server's CA key kept off the nodes: only the online key stays on the servers.
• Queues saved to disk, undelivered messages deleted after 21 days, daily statistics off, no built-in web server.
• Official build (simplexmq), with its SHA-256 fingerprint pinned in the playbook.

Video calls (Galène)

• Galène 1.2 (an SFU written in Go, IRIF / Université Paris Cité) on mx1, behind nginx on riunioni.mochela.it with an Actalis certificate. Galène listens on 127.0.0.1 only; from the Internet only HTTPS (pages and WebSocket), audio and video over UDP 20000-20999 and its TURN on 1194, TCP and UDP, for people behind networks that block UDP.
• Encryption: audio and video in DTLS-SRTP between each browser and the server, which decrypts and re-encrypts them to forward them to the others (that is how an SFU works). It is not end-to-end encryption, as the Galène website also says. File transfer, on the other hand, is: it goes from browser to browser. For an end-to-end call between two people: SimpleX.
• Rooms: every meeting can have a random address (sub-rooms created on first entry, gone when empty). Locked until an operator joins (autolock). Guests join with a time-limited invitation link, no account needed. Passwords stored only as PBKDF2-SHA256 hashes; Galène's administration interface is closed, rooms are written by the playbook.
• Recordings only by an operator (/record), only in rooms where they are enabled and after telling the participants: the server writes them to a fixed-size space in the encrypted volume of mx1, lets only the room's operators download them, keeps them out of the backups and deletes them after five days. Chat in memory only, deleted after four hours.
• IP addresses: nginx keeps no access log and does not pass the client's address to Galène, so Galène only sees 127.0.0.1; Galène log lines containing an IP address are dropped by journald (LogFilterPatterns). Audio and video, however, reach mx1 directly from each participant's address, for as long as the call lasts.
• Built on the administration laptop from the official source, pinned version and source fingerprint in the repository; unprivileged systemd service, with a read-only file system except for its own data directory.

Web and certificates

• nginx. The site is static: no JavaScript, no cookies, no external resources, enforced by a Content-Security-Policy that forbids everything except the site's own style sheets, fonts and images. HSTS, Referrer-Policy: no-referrer.
• TLS 1.2 with ECDHE ciphers only (forward secrecy), TLS 1.3; hybrid post-quantum key exchange X25519MLKEM768 (OpenSSL 3.5), with X25519 for clients that do not know it. Applies to the website, the mail pages and the video calls active
• Access log with 0.0.0.0 in place of the IP, kept for seven days. nginx error log: IP addresses and names removed after one hour, then kept seven days.
• On the mail nodes nginx serves HTTPS for posta.mochela.it and passes to Stalwart only the pages meant for users: /admin answers 404 from the Internet (administration goes through an SSH tunnel). Stalwart receives the client IP in X-Forwarded-For, written by nginx, for its blocking of login attempts.
• Certificates from Actalis (Italy) via ACME with HTTP-01 validation: nginx's with certbot, one per node; those for IMAP and SMTP requested and renewed by Stalwart. Validation works whichever node Actalis asks: nginx looks for the answer locally and then on the other node, over WireGuard.

DNS and time

• Unbound on every node, as a recursive resolver: queries go from the node to the authoritative servers, with no third-party public resolvers.
• chrony with NTS (Network Time Security, authenticated time) from European servers.
• mochela.it zone at INWX (Germany), signed with DNSSEC, with a CAA record that allows only Actalis.

System security

• SSH with keys only; the private key has never been on a server.
• Administration (SSH and the Stalwart panel) only inside a dedicated WireGuard tunnel between the laptop and the nodes, with an additional shared key (PresharedKey); port 22 closed to the Internet since 14 October 2026, with the provider's console as the emergency way back in
• Automatic security updates (unattended-upgrades); the cluster packages are excluded and updated with the playbook, one node at a time. Manual reboots, one at a time.
• Secrets (service passwords, API keys) encrypted with Ansible Vault, never in clear in the repository.
• Data in a LUKS2 volume on every node (AES-XTS 512-bit, passphrase protected with Argon2id): database, message storage, etcd, Valkey, Stalwart logs, SimpleX queues. Passphrase only in the Vault and on paper, never on the nodes. After a reboot the data services do not start (systemd condition) until the volume is unlocked from the laptop, or from the provider's console; Sentinel stays outside and keeps voting. Active on all three nodes since October 2026 active
• The whole configuration is an Ansible playbook: a new node is installed and joins the cluster in one run. A monthly dry run (--check) confirms that nothing has been changed by hand.
• The playbook, the operations guide and the scripts published as free software (AGPL-3.0 for the code, CC BY-SA 4.0 for the texts), with no secrets and no data: source code active. An interactive installer, so that anyone can build the same cluster on their own servers planned

Checks and alerts

• On every node a systemd timer runs a check script every five minutes: active services, disk, time, certificates, database replication, state of etcd, Garage and Sentinel.
• WKD integrity: the node without the website downloads the published keys from mochela.it and compares their SHA-256 with that of the files in the repository. A key changed on the server triggers an alert.
• The nodes check each other over WireGuard; cluster checks are reported by a single "spokesperson", the first reachable node.
• Alerts only on state changes (failure, recovery), via SMTP with TLS to an external mailbox: they do not go through the mail system they are checking.

Backup

• Every night the mail nodes export the database (pg_dump), the messages (a copy of the Garage storage) and the Stalwart configuration.
• The admin laptop fetches the export with rsync over SSH, with a dedicated key that the server restricts to reading that folder only (rrsync -ro).
• In the same archive, every version of the programs that do not come from Debian (Stalwart, stalwart-cli, SimpleX, Garage; Galène as source, built on the laptop), with its source code and its SHA-256 fingerprint under git: the laptop downloads them once and copies them to the nodes, which never connect to GitHub.
• Borg repository (repokey-blake2 encryption, zstd compression) on a LUKS2 disk that stays disconnected. Retention: 7 daily, 4 weekly, 6 monthly. second disk elsewhere

Software

All free software.

Found something that doesn't add up? Write to info@mochela.it: corrections are welcome.