28th
Scaling Twitter, and the Friend-Follower
On yonder Twitter Developer Blog, al3x points us to his favorite explanation of why scaling Twitter is so difficult. It’s a great read, and it goes into fairly accessible detail about the problems involved in scaling a message-intensive social network. As I understand it, you can either duplicate messages so that each follower has a copy (cheap lookup, massive storage) or you store one copy and aggregate everything per request (cheap storage, expensive lookup). This is a classic comp sci tradeoff.
Unfortunately the author of that post (for his own reasons) and Twitter (for theirs) dismiss the idea that scalability can be addressed through a distributed system. In this approach, everyone (or many people) run their own Twitter-like service that stores their status updates, and makes them available to the world (or to a selected set of followers using OpenID).
The following could be done today using a simple newsreader: if you subscribe to the RSS feeds of your friends, then no matter what service they are using (Twitter, Pownce, LinkedIn, Facebook) you can collect all of their status updates in one place.
Ah, but how do you find out who is following you? One of the great things about a social network is that communication is two-way within the walled garden. I can see who is following my posts, and scan their feeds to see if I want to follow them. If distributing messages is easy, distributing the social graph of who knows who is much harder.
This requires a slight twist on the standard newsreader, and I’m going to call it a “Friend-Follower” application unless someone points me to a better description.
A friend-follower knows who you are, and where to find the feed of your own status updates. It uses this information to aggregate your posts with the posts of those you are following (so that you see them as a conversation). It also uses this information to perform a trackback style notification whenever you decide to folllow or stop following someone else’s updates. Your microblogging software (your own personal Twitter) would also receive these updates and maintain your personal social graph as an Atom feed.
Of course, Twitter and the other big social networks are not going to make this easy, because their business models are likely built around the assumption that they can deliver very large audiences to advertisers. They aren’t going to modify their systems to allow you to have friends on other networks, provided they can solve the scaling problems mentioned above.
Nevertheless, it isn’t hard to imagine that a neutral third-party, like Google, might see the value in being keeper of a social meta-graph that bridges all of these walled gardens. Let a million Twitters bloom!
