Federation in Refrain is based on shared trust and data sovereignty, which allows users from one server to log into and perform actions on another without requiring that data to be broadcasted. This is very much an intentional design decision, as any kind of API/media proxying that could be done by a user's home server simply doubles the network bandwidth burden of both the home server and the target federated server. Any kind of shared eventing means that a particularly active server sends quite a lot of (possibly sensitive) data to all other federated servers.
If federated users are given access to see the data, that's between the server/community admin and their god, but we don't want that data blasted across the federation for anyone to pick up. The local server admin decides what local and federated users can do by default, and community admins decide what local and federated users are permitted to do. Data and access control stays local, the only thing federation directly does is give users the ability to access the federated servers.
Note
This is where end-to-end encryption is sometimes pointed to as a silver bullet. "If all your data is encrypted, who cares where it ends up?" one might say. This is a categorically bad idea; if the key to that data or the encryption algorithm is ever compromised, the data sitting out on other servers is vulnerable. It is much safer to simply keep your data in its home than to try to come up with a fancy way of distributing it safely.
So, clients in Refrain are a bit fat as a consequence:
They are required to connect to federated servers when they need to do something on that server. This makes the diagram look a lot worse than it really is; clients are typically only going to fetch data from all servers when they first load, after that they will update their internal stores based on events emitted by the event subscriptions. Media connections are only made when the client actually joins a voice/video session.
🔗How to federate
Federation is asymmetric in Refrain. Your server can trust other servers and allow their users to access your server, but that does not immediately grant your users access to that server. Let's take the simple example where your server (ServerA) wants to federate with another (ServerB). As the admin of ServerA, you:
- Talk to ServerB's admin. Determine if federation is something that makes sense for you both. If not, don't federate! Let's say that it does in this case though.
- You configure ServerA to allow users from ServerB to access your server. In doing so, you must specify a default role all users from ServerB will get assigned when they first log in.
After doing so, your server (ServerA) will look like this:
Next, ServerB's admin will trust ServerA and allow ServerA's users to access ServerB, doing exactly what you just did but for ServerA:
Now, at this point, both servers trust each other and will permit each other's users access. However, neither server is advertising this capability to its users yet. To do this, you validate ServerB's access by clicking the button in the UI. Since ServerB is trusting ServerA at this point, it should succeed:
At this point, all of ServerA's users know their client's token is trusted by ServerB and connect to it. ServerB's admin does the same, and all of ServerB's users connect to ServerA. This will be done on every login for each server's clients.
🔗What federation actually does
When you federate with another server, your server tracks the remote server's JSON Web Key config. This contains the public key used to sign JSON Web Tokens produced by the server, and can be used to validate that a JWT was produced by that server.
So, clicking the "Grant federation" button in the UI just triggers the backend to record the JWK the remote server publishes publicly. Any access token from that server contains the issuer (the federated server's domain) and a signature that can be checked against that JWKS config. Clicking the "Validate for discovery" button simply adds the server to the public list of federated servers so clients know about it.
🔗How authentication works
If Refrain were to accept bearer tokens cross-server, the JWK signature is technically sufficient to say that the token was generated by a specific server. One could, conceivably, simply trust that bearer tokens from federated servers are legitimate, and leave it at that. Refrain does not do this because there is a very big gotcha in that system. bearer tokens prove nothing else except that:
- the token was generated by a specific server
- the requester has the token
Bearer tokens do not prove that the client making the request is the one that generated the token. If anyone in the federation gets ahold of a user's bearer token, such as a malicious server admin, they can effectively impersonate that user for the duration of the bearer token. They can't make more bearer tokens (you need a refresh token for that), and the TTL on a bearer token is fairly short (5 minutes), but in that window they could do a lot of things on multiple servers including the user's home server!
The typical mechanism for avoiding this is to do full OAuth flows to each server in the federation to ensure that each bearer token works for only one server. Trust would be configured such that ServerB would delegate logins to ServerA and accept a bearer token with the intended audience of ServerB only. That way, any malicious server admin that gets ahold of the bearer token can only take actions on their own server, which is something a malicious admin could do anyway.
This is, to say the least, heavy. Every client would have to track multiple refresh tokens. When the refresh tokens expire, the user will have to redo the OAuth flow for each server. Not a very clean experience, so Refrain does not do this either.
🔗DPoP
To make cross-server access easy on the clients, Refrain supports a specific kind of OAuth implementation called Demonstrating Proof of Possession. This method of authentication allows the same authorization token issued by the home server to be used on any federated server. At a very high level, DPoP works like this:
- Client generates an ES256 public/private key
- Client gets a signed refresh token from its home server by logging in with user/pass
- Client requests a DPoP token by providing its refresh token and its public key
- Home server signs a new DPoP access token containing the client's public key
- Anytime client makes a request, it includes a DPoP proof signature signed by its private key as well as its DPoP access token
The DPoP signature states what request is being made, which prevents any attacker from impersonating the user on any other server. All that is required is for the target server to respect the signature of the access token and to validate that the DPoP proof both matches the request and is signed by the same public key in the access token.
Note
If any malicious entity gets ahold of the DPoP token, the only thing they can conceivably do is replay the action the client already took since the token is unique to the request.
The scope of this is fairly small, but to avoid it Refrain also tracks the unique jti IDs used for the last few minutes. The jti on each request is expected to be unique, so
each request can easily be checked to see if it has been observed recently.
As long as the cache of jtis is big enough to store requests for ~5 minutes (the TTL of DPoP access tokens), no malicious requests can be replayed before they expire.
Now, all DPoP does is make it possible to prove that the client is who they say they are. What actions they can perform in the federated server is entirely up to the individual server admins to decide on; no permissions from their home servers come along with them. Once two servers become federated, users from ServerA that access ServerB will be allowed in and given a default role assignment that ServerB's admin chooses. This creates a stub User record in ServerB, so for all intents and purposes they are valid users in ServerB with their own permissions and ServerB's admin can grant individuals more access if they desire.
🔗Subscriptions
All federated clients should open subscription connections to each federated server when they are online. This ensures that any changes made relevant to the client are observed, no matter where in the federation they occur. The main reason for this is to ensure the client UI is updated appropriately; if the user is added to a community on the foreign server, there will be no event from their home server to notify them. It is possible to poll federated server to eventually catch up, but that is much more load than a subscription.
Warning
This is the main downside to the federation approach taken here, as it will put the same load on all servers in the federation if all clients connect uniformly. The trade-off is currently considered acceptable as making the clients listen for cross-federation events avoids the data sovereignty issue and subscriptions don't cost a lot on the server.
So far, synthetic testing has proven this to be true for many hundreds of users, but real-world testing may prove this to be wrong. If so, the alternative is to allow federated servers to broadcast events through each other, and have clients only subscribe to events from their home servers. This is effectively what IRC relays do, so it wouldn't be a unique solution. The downside is, of course, some amount of sovereignty loss. If needs be, we could make this an alternative kind of federation rather than a straight-up replacement.
The server automatically selects event filters on authentication from the user's permissions and memberships; clients do not send explicit subscription requests. WebTransport sessions are very lightweight on their own, and unless every user is listening for a large number of specific events, the event filters do not take up a lot of space in memory. This means that all the client really needs to do is decide how to react to each kind of event, regardless of which server the event came from. In the webui, this plus DPoP tokens simplify a lot of logic; an entity from another server is just an entity you access through a different URL. Events are no different.
🔗Alternatives Considered
Federation in Refrain wasn't something decided on easily. There are a lot of federation strategies out there, here are the ones considered and discarded.
🔗ActivityPub
ActivityPub is the federation protocol for most people, since Mastodon and Lemmy both use it. This is a fine protocol for public, Twitter-like interactions, but ActivityPub is really not meant for chat. It's quite heavy and asynchronous, which is fine when posting public threads; eventually everyone in the federation is going to see your post and if it takes a few minutes that's fine. However, in the context of live chat sessions, that will almost immediately fall apart.
Further, data sovereignty is not a thing with ActivityPub. The protocol was meant for federation and the implication of it is that every server in the federation is trusted to some degree. There are extensions to allow for encryption, but messages from a user on one server to a user on another server go through that server. If the server admin is malicious, that data can be collected and stored. For public posts, this is completely fine, but Refrain is intended for private communities and messages.
🔗Matrix-style replication
Under this model, every federated server fetches a copy of the entire chat room's history, updates locally and sends updates to the home server, and resolves deltas asynchronously. In Refrain, that basically means that all servers in the federation need full copies of everything. This is problematic for a lot of reasons, but here are a few:
- Popular chat rooms take up huge amounts of space on the federated server
- The data has to be queried from its original source at load time, which can be very slow
- Edit history is communicated as a big ol' DAG that the client has to interpret, which can be quite slow
On top of this, the opening of events to all federated servers puts sensitive data at risk, no matter how fancily it is encrypted.
🔗XMPP replication
XMPP broadcasts events to federated servers, but keeps the chat rooms (MUCs under its terminology) local to the home server. The federated servers do not need to store anything for this to work (but typically do for some time as the user's client may be offline). When a client joins a new MUC, it requests the history from the home server.
XMPP is a pretty good protocol and is not that different from what Refrain does in shape. Refrain clients fetch room histories on page load from the home server of the room, not the user. The big difference is how events are broadcast; XMPP will send events to the home servers of all participating users. This is an expensive operation, and means federated servers will see the data from the room. XMPP does not specify end-to-end encryption, so the stanzas are typically sent in plaintext which makes the data sovereignty problem much worse.
🔗IRC relay
This is a throwback, but one can picture it as somewhat similar to XMPP but with simultaneously more trust and much more ephemerality. Servers do not store any channel messages, clients are responsible for loading channel history. Clients only communicate through their "home" (more of a gateway than a home, really) server, but since the IRC servers are networked together any messages the client sends get relayed to all other servers.
This runs afoul of most of the same problems as XMPP, but the broadcast problem is even worse as all federated servers would need to get all messages sent to all rooms.