Refrain is intended to be a lightweight, self-hostable alternative to services like Discord and TeamSpeak. It enables communities to communicate with each other, over text, voice, or video. These are basic needs any community has, and there is a fairly crowded field of offerings out there, including some open-source ones like Stoat. So what makes Refrain different?
Honestly, if you are a normal user, not much. In fact, it's currently worse than a lot of the alternatives in terms of features. The goal of this project isn't necessarily to out-compete on features, it's to make self-hosting simple and to scale. Refrain is something that can run on old hardware to host a few reasonably-large friend groups (a couple dozen people in each) with nothing more than:
- A TLS cert from Let's Encrypt
- An IP address
- A single open TCP/UDP port
Note
A key feature of Refrain is that the multimedia sessions are not operated over WebRTC. If you've ever had to host a WebRTC service, then you're probably familiar with the Voltron-like appendages that make it up (ICE, STUN, TURN, DTLS, RTP/SRTP, etc.) WebRTC is notoriously finicky to operate because, at its heart, it is built to set up P2P connections between browsers. Most server-based implementations have to effectively pretend that the server is just another browser-like node in the network.
Client-server systems are not like this, and it's very annoying to have to set up ICE/STUN/TURN services (plus the large number of ports they need) or rely on corporate offerings from companies like Google to run. Refrain uses WebTransport to transmit media session data, which allows clients to treat it like any other HTTP service and for operators to scale it the same way.
When your friend-groups grow or turn into multiple shared community spaces, and you need to upgrade, Refrain is intended to be able to grow with you:
- Scale vertically by upgrading hardware. Refrain scales with CPU cores and network bandwidth.
- Scale horizontally with more devices. All Refrain services can be run on multiple devices to distribute load.
- Combine multiple servers together without uprooting anyone through server federation
The reality of self-hosting is that you tend to have to make do with what you have rather than build something ideal. Since Refrain is easy to break apart and consumes few resources, you can easily cobble together something that works out of what you've got lying around!
🔗Where can I run it?
The server has been tested running on a Raspberry Pi 4 with up to a couple dozen users, all the way up to large multi-CPU servers with thousands of users! All it needs to run is a Linux OS and the following dependencies:
- Postgres: home-of-record for all entities
- RabbitMQ: used for event production and consumption
- S3-compatible store: stores file uploads. There are dozens of these, pick your favorite:
The entire thing can be easily started in a Docker Compose stack, or scaled up to a full distributed Kubernetes deployment.
🔗How does it work?
Refrain's server runs as a collection of microservices serving HTTP/3 traffic to clients. All traffic is done over HTTP, including voice/video thanks to the WebTransport standard! This means that you can easily serve all traffic with a single TCP/UDP port exposed, and traffic can be routed and load-balanced like any other HTTP service.
Note
One caveat that HTTP/3 has is that TLS is absolutely mandatory. No HTTP/3 client will work without it. This is generally a good thing, but can be kind of a pain when self-hosting because TLS includes hostname verification. Let's Encrypt can provide you a TLS cert, but you have to own a public domain name for it to check against.
Refrain can autogenerate self-signed certs for you, which does require the mandatory TLS scaremongering consent if you access the UI in your browser. However, if you access the server with the desktop client, this will not be necessary!
So how do the microservices work? Well, they are all Rust modules that perform various functions:
- refrain-server: An HTTP/3 frontend, routes traffic to the appropriate components based on request path. Also hosts the web UI static content.
- refrain-api: A standard RESTful API, including an OIDC authentication system
- refrain-subscribe: Allows users to subscribe to changes in the Refrain server. Operates over WebTransport reliable streams.
- refrain-voice: Hosts voice conference sessions for voice-capable community channels. Operates over WebTransport unreliable streams.
- refrain-video: Hosts video conference and screensharing sessions for video-capable community channels. Operates over WebTransport unreliable streams.
The clients listed below are the official ones compiled by this repo:
- refrain-webui: A Vue3 SPA, statically hosted by refrain-server
- refrain-desktop: An Electron app, provides the web UI locally with some extra bells and whistles.
- refrain-cli: A TUI/CLI app, useful for scripting and for being a super cool hacker
All of these services coordinate through the dependencies listed in the previous section. The following sections detail the very-high-level architecture.
🔗Postgres
Postgres is the home-of-record for almost every entity in Refrain. The main exception is file uploads/images, as those are stored in S3. Postgres also serves as a coordination point between services when they are running on multiple hosts.
Architecturally, this does appear to get a lot of load. However, most database interactions are fairly lightweight reads (and are automatically cached by Postgres to avoid expensive disk I/O) or are updates to non-durable tables for user presence tracking. Unless you have communities posting huge numbers of chat messages per second, it's unlikely that the system will cause much load on Postgres.
🔗RabbitMQ
RabbitMQ is used as the event bus for Refrain. Most events today are emitted for the sake of the event subscriber client, but there are some point-to-point queues that allow media servers to communicate with each other. This is currently used today to evict users from existing sessions when they move to another one, but may be used for more in the future.
🔗S3
S3 today is used exclusively for storing binary blobs, either for channel/user avatars or from user-submitted uploads. No advanced S3 features are used here, it's just a dumb put/get store for large binary content.
🔗How does the voice/video stuff work?
This is embarrassingly simple, actually: the voice/video services just schlep bytes around. Clients join a session and begin sending audio/video data in an already-encoded form ( Opus for audio, VP9 for video) to the server of a WebTransport connection. Each transmission is effectively a "frame" of data; a short segment of audio or a key/delta frame of video. WebTransport has this amazingly useful notion called "unreliable transport," which is W3C-speak for "datagrams."
If you didn't know, HTTP/3 is built on UDP (datagrams) which are unordered and have no delivery guarantees, but HTTP/3 has reliability baked into its protocol. When the application sitting on top of HTTP/3 requests data, it will always get that data in order. Under the hood, HTTP/3 is storing out-of-order datagrams as they arrive and re-ordering them when all the pieces appear/asking the server to pretty-please send a message again if a packet gets dropped.
WebTransport allows you to effectively turn that ordering off, and just blast datagrams to and fro with no reliability guarantees. Unreliable datagrams are perfect for live audio/video because we absolutely do not want to wait for frames that get lost/take too long to show up. If we did, the stream would stutter and the player would start falling behind the source.
Because of this utility, the voice/video services just facilitate broadcasting the bytes sent by a client to other clients. A voice/video session is a temporary tracker indicating what connections should be paying attention to which other connections. A voice session is a conference, so all participants send to and receive from each other. Therefore, the majority of what these services do is just...copy bytes around!
There's more to it, obviously, we have to track state, optimize the broadcast paths, and deal with the fact that the session is inherently stateful. The clients also need to deal with data dropping out because the standard HTML elements don't handle that very well. However, the big fancy thing that these backend services do is not actually anything to do with audio or video, it's just schlepping bytes around.
🔗How do I run it?
As stated above, all of these services are built to be run on different hosts and to scale independently. This allows compute to be thrown at scaling issues as the server grows.
- The api service is stateless and more instances can be added to handle high API traffic
- The subscribe service is stateful in that each connection made to it is bound to a single host, but adding more instances to handle high subscriber traffic is simple.
- Individual voice/video sessions must have all their participants on the same host, but having multiple voice/video instances means you can host more sessions in parallel.
- The refrain-server module handles stateful routing for WebTransport traffic, ensuring that voice/video participants always wind up on the same host when they join a session in progress. This module is itself stateless and more instances can be added to handle high traffic.
If you wanted to, you could run many instances of each of these services across a fleet of hosts!
Note
Today, you will have to have multiple IPs to do DNS round-robin (or something more intelligent) to scale the router component. There is currently no support for WebTransport reverse proxying in any of the major vendors, but Caddy has an experimental PR to add it soon!
This is really only necessary when you have massive amounts of media traffic, enough to saturate the CPU of the router component which is simply proxying the connections to the backend services behind it. This happens with large numbers of people (>50) in the same channel, talking or sharing video at the same time.
All that being said, these individual services are actually run from the same binary, just with different commands. As such, Refrain can run all of them in a single process, so if you are just getting started and don't need a lot of scaling, you can run the server with a single command:
> refrain-server standalone
This will work well for quite a lot of load! When you need to start scaling up, you can decompose the single process into multiple processes across multiple hosts. You don't necessarily need to go from "one host on a RPI4" to "a fleet of hosts in AWS" in one step, you can break out the pieces that are getting a lot of load incrementally.
More info on how to deploy is available in the hosting a server page.
🔗What is Refrain not?
Refrain is not a platform, nor does it intend to be. While it's fair to compare it with Discord and TeamSpeak, these are both fairly large services with a lot of features and a lot of capital behind them. They are both pursuing a platform model (TS does offer a self-hosted option, but it is heavily hamstrung) because that locks their users in and guarantees that sweet annually-recurring revenue.
Platforms are a honeypot, and their bait is features. At their core, Discord and TeamSpeak facilitate a community's basic communication needs, but they add little bells and whistles outside of that core that entice people to commit more and more to them. After a certain point, their users are so deeply entrenched in the platform that leaving is impossible. When the platforms inevitably enshittify, these users are simply stuck and their community suffers.
Refrain is not a money-making venture. Aside from the fact that the market is crowded, I (author here, hi!) have zero interest in turning this into a proper business. Making money off of people's basic need to communicate is gross. This project is intended to be something everyone can run on their own, and while I won't say I'll never take donations, the AGPLv3 license on this project is intended to guard against any bait-and-switch like Hashicorp or Minio happening in the future. I'm not a lawyer though, so if you know of a better way, please email me!
Refrain is not going to have every feature from every competing platform. Matching all of these features is not feasible for any project without a sufficiently-large amount of capital, so Refrain will refrain from doing this. Instead, it will focus on the basic needs that all communities have and work to improve the self-hosting features over time. You don't need to have a jillion emojis or shitty little games in the thing that you use to talk to your friends. You need a tool that makes communication easy.