What I Wish Someone Had Told Me About AT Protocol and Bluesky

I've been interested in atproto (the protocol that Bluesky, Blacksky, and many other applications - including pckt, where this blog is hosted! - run on) for about a year now. I'm by no means an expert, but I've read a lot of documentation and discussion, looked at various versions of the architectural diagrams, and spent a lot of time thinking about the implications for scaling, decentralization, and other aspects of design and operation. I run my own PDS. I wrote some clients that consume firehoses from relays and analyze the data. I thought I had a pretty good handle on things.

And then, one Sunday morning, some of the Bluesky services went down for a few hours. You can learn a lot more about a system from its failures than from normal operation, and I learned that a lot of the things I thought I knew were - not wrong, exactly, but also incomplete for understanding how this ecosystem actually functions in practice.

I felt a little foolish that it had taken me so long to learn some of these things, or, at least, to really internalize what they mean. These are not, like, secrets, or anything. This is not airing out the ecosystem's dirty laundry. Plenty of people already knew these things. But I didn't, and maybe you didn't either, and that's why I'm writing them down.

When you learn about the Web, you might learn about HTML, CSS, HTTP(s), and this is enough for you to build and serve web applications. But then something goes wrong, and ... you learn about the DNS. Because it's always DNS. Not because the DNS is bad (okay, maybe a little, in some ways) but because it's so central, and it's one of the places where a large amount of the complexity in the Internet hides. When it works, awesome. When it doesn't work - or it does work like it's supposed to, but not in the way you thought it did - well, now, it's time to learn about the DNS. Sorry.

This is not going to be an overview of atproto, nor is it intended to be a comprehensive view into all of its corners. I'm going to write assuming that you (think you) know the architecture at the level of say, Dan Abramov's "There Are No Instances In atproto" article, which is a great read.

I want to say, again, that I'm not an expert here. I've observed behaviors, read code, looked at specs, but there are other people who know this stuff much better than I do. I welcome corrections, added nuance, etc., and I'll update this post. I also want to make it clear: I am describing how things work today, to the best of my ability. My understanding from talking to some atproto folks is that this is not all necessarily how it's designed to work, and there are things here that are, in some sense, outside of atproto itself, and are Bluesky-specific. But you have the system you build, not the system you design.

So let's see what's built.

The PDS Doesn't Sit Quite Where I Thought It Did

In the standard architectural diagram, you see a repo in a PDS, data flows (maybe through a relay) to an AppView that aggregates data from a bunch of repos, and the application itself talks to the AppView to get that ... uh, view.

That last step is omitting something. In many cases (but not always!) the app actually talks to the PDS, not (directly) to the AppView.

The PDS proxies the app's traffic to the AppView. When an app wants to make an API call to the AppView, it contacts the user's PDS, which mostly (hang on, more in a minute) forwards that along to the AppView and sends back the reply.

There are good reasons to do it this way. A big one is authentication: the user, via the app, authenticates themselves to the PDS, which is what holds the keys that can authenticate the user to the rest of the network. So: request comes in from the app, the PDS knows it's really from the user because it's on an authenticated session. PDS makes a token that can prove to the AppView that the request came from the user, and sends that along with the request. The client gets to ignore a lot of complicated stuff involving keys, scopes, audiences, and more. Great! If you have a complicated application that needs to talk, authenticated, to a bunch of different services? PDS's problem. Making sure that the tokens needed to talk to each service are kept fresh before they expire? PDS's problem.

But, one thing you might not expect from the standard architectural diagram is that if your PDS is down or misbehaving, one failure mode is that you can't talk to the AppView, even if you're already logged in.

There is nothing stopping the app from talking directly to the AppView, and, in fact they often do, for calls that don't need authentication. You can also ask the PDS to give you the necessary tokens, so that you can pass them to the AppView yourself. But some applications, at least the Bluesky app (also called social-app), are built to use the proxy.

The PDS Depends on The Bluesky Appview - Kind Of

I've been saying "the" AppView, and I hope this grated on you, because, no, there is not the AppView. In the specific case of an independent Bluesky-like application (such as Blacksky) there are multiple AppViews running that one can use. In the more general sense, other applications can implement their own AppViews that have very different APIs than the Bluesky AppView, or can operate without an AppView at all.

But.

There are calls one can make to an AppView that looks like the Bluesky AppView, through the PDS, that are really useful. And so, of course, apps use them. Probably even if the app in question is not Bluesky, even if (actually, especially if) the app doesn't have its own AppView.

A good example of this is getProfile - data flowing around the Atmosphere refers to accounts via their did. But you don't want to see the did in the user interface, you want to see stuff like the user's handle, and probably get some other information like their display name. Who can tell you this stuff? Well, one way is to ask an AppView! It has an API endpoint that will tell you!

How does the PDS know which AppView to send this request to? Well...

There's a header, set by the app, in each request that says where it should be forwarded. But what if the app leaves out that header? The reference PDS, provided by Bluesky, has a default AppView configured - it's the Bluesky one, of course, but you can reconfigure it to point to another. Some other PDS implementations seem to have this same defaulting behavior - but not all of them.

Because, having the PDS default to a particular AppView kind of ties that PDS to that AppView, right? In a way that is pretty surprising to most folks that host their own PDS, and is pretty antithetical to the "there are no instances in ATProto" view of the world.

Let's say I set up my own PDS. I leave the defaults alone. Well, a lot of apps - including many that are not Bluesky - are going to end up sending a lot of API calls to the Bluesky AppView if they haven't specifically requested another. If the Bluesky AppView is down, then these apps don't work for me because of a setting in my PDS. Huh.

But let's look at what this failure mode actually looks like. Bluesky AppView is down, so the main Bluesky app is down. But I'm a cool guy, I have my own PDS, and I know that in atproto, I can just log in to any other app with the same account. So I head over to the Blacksky web app and log in there. But some of the calls it makes (details in a second) fall back to my PDS's default AppView! So, stuff is pretty broken! I have to reconfigure my PDS to actually properly use the Blacksky app! People who are on the Blacksky PDS are pretty okay, because their PDS (I assume) points to the Blacksky appview already!

So yeah, I can log into other apps with my one account, but which ones actually work depends on ... where my account is actually hosted and how that's configured.

This is not really the instance-free world we thought we had, is it?

Okay. So, maybe this "default AppView" thing is a bad idea. I know of at least one PDS implementation that doesn't have it - you tell the PDS where the calls are supposed to get proxied to, or you get an error.

But what are you, an app without its own AppView, to do if the PDS you're talking to doesn't have a default AppView? Hey man, I just want to get some display names! But this is ... unclear. Always use the Bluesky one? Have a list and look for one that works? Have the user pick? Suck it up, you have to have your own? Something else? What are the semantics here supposed to be?

Oh, and: I'm writing this all in terms of outages. But different AppViews can return different data, and (in part of the world with a default one), this is determined by your PDS, so ... what you actually see in an app can depend on how your PDS is configured. Probably not what most people would expect.

RAW(r): Some Bluesky Application Semantics Are Implemented In The PDS

I used to think that the PDS was just a place that stored user data and keys. As we've seen above, this isn't the whole story. And we're not done, because, in fact, the PDS implements part of the Bluesky application.

Calls to the PDS don't necessarily just proxy to an AppView or other service. Some of them run substantial logic locally. These include the ones you probably expect, like reading or writing to a repo.

But there are some other ones in there that I found surprising, and many of them focus on Read-After-Write (RAW). Here's why.

You write the world's best shitpost and hit send. Naturally, you want to see the fruits of your labor immediately. But this is not how atproto works. You write it to your repo. It goes out the firehose, through a relay (probably), hits an AppView, gets indexed, and then people see it when they ask the AppView (and/or a feed generator) something that should return it. This is pretty damn fast most of the time! But it's not instantaneous. And sometimes things do get clogged up and it can take a while. I want to see my masterpiece now!

The reference PDS has your back with its RAW optimizations. Sometimes, for some calls, it'll notice if the record you just wrote isn't part of the return value from the API, and it'll sneak it in. Did the mean AppView 404 your oeuvre? The PDS will make up a fake thread, just for you, just for now.

I'm sure this was done for responsiveness. Feels much better when stuff happens immediately, and the protocol itself can't guarantee delivery times.

But let's think about what this means.

It means your experience of the app is, once again, determined by your PDS. Using an alternate implementation that doesn't pull this trick? It'll feel less responsive, even though it's getting your stuff out on the network just as fast. But worse, it builds more application semantics into the PDS. Was this call supposed to return that record? That's application semantics! The PDS has to know which API calls would return that record, and how. And let's say Bluesky changes their AppView to not attach replies that contain the🖕emoji to their parents? It's going to look to you like you successfully gave your opponent the bird. Gottem!

I've been deliberately silly in this section (and ONLY this section, naturally) but the serious point is that actual application behaviors live in the PDS - what you see, when you see it, and whether you see it at all depends on your PDS. If Bluesky ever wants to change any of this, what you see is down to whether your PDS has been updated, or even implements these features. And, if some other application needs or could benefit from these kinds of features? They've gotta convince Bluesky, and everybody else, to add it to their PDS implementations.

I've Seen Bugs You People Wouldn't Believe...

Before we wrap up, let's look at an example of how all this stuff flowing through the PDS, and all the processing it does on it, enables some super pernicious bugs.

In the getFeed call, there are two steps: look up the feed generator, then pass it (and a token addressed to it) to the AppView. The reference PDS from Bluesky has what seems to be a bug, where the first half of this ignores whatever AppView it was told to use via the proxy header, and just talks to whatever the PDS-default AppView is. So, yeah, during a Bluesky outage? Pull up the Blacksky app, great, but if your PDS is still configured to use the Bluesky AppView, you can't load feeds.

Sure hope this is a bug, sure hope that it gets fixed, and that I get to change that paragraph to past-tense. But it's wild that a PDS bug can have this kind of effect.

Oh and while we're here, this two-step feed lookup thing with the extra token is ... yep, Bluesky-specific application semantics. But every PDS better implement it, or people trying to use Bluesky from it might not be able to load feeds. Or at least some of them.

And this is not how I thought it worked.

PDS is Love, PDS is Life

I focused on the PDS here because I really, genuinely thought that the PDS "just" stored data and held my private keys. I thought by running my own, I was getting away from depending on Bluesky infrastructure. I thought that which implementation I picked was mostly a matter of taste, not how applications would operate. I thought I understood the failure models: that Bluesky could go down, I could just switch to another app, and everything would be fine.

The actual story is not that simple.

The most recent downtime was definitely smoother than previous ones. And now I think I kinda know why. Some people were able to, mostly, just keep moving on. But it depended on which PDS they were on, which is not how this is supposed to work. Things are far more entangled than they look on the architecture diagrams.

And the quantity of application semantics built into the PDS? I did not expect that one.

Please, tell me about the other stuff I'm missing. I think I want to know (though maybe I don't).