What follows is a modest suggestion to all social media sites with a web service API, but is particularly pointed at Twitter. In this note, let me talk directly to issues I have with Twitter's API, but many of the points raised can be applied to other sites to equal beneift.
Twitter's REST API is quite complete in that it provides access to nearly all the data one sees in a user timeline and profile. However, if one wants to write a tool to monitor changes in a user's metadata, one has to poll the API. That is, the monitor write needs to periodically query every Twitter API call that may hold interesting information. This is ineffecient for both the monitor writer and for Twitter, who has to serve a number of calls with information that has not changed since the last call.
What would be better is for the Twitter API to implement an Observer design pattern. Twitter would identify some number of interesting data modification events, such as the following list:
- Add/lose a follower
- Add/lose a friend
- Screen name change
- Location change
- Profile description change
- New direct message
While not exhaustive, this list of events would make the job of writing a monitor tool much easier. A possible workflow might look like this:
- An app registers with a new Twitter API a callback URL and a list of events
- When one of the events of interest occurs, Twitter calls the registered URL with a payload that includes the event that occurred and perhaps some domain-specific metadata (i.e. the ID of the follower who joined/dropped)
Because of the volume of users, the Twitter API is very sensitive to producer/consumer issues. I propose that Twitter queue up events and then send out notifications at some reasonable rate, perhaps imposing a callback rate limit. Even with a rate limit, the callback system will provide much better performance for tool writers than polling the API.
Please discuss below.