chicken beard science

Event-driven callbacks for Twitter, a proposal

What follows is a modest suggestion to all social media sites with a web service API, but is particularly pointed at Twitter. In this note, let me talk directly to issues I have with Twitter's API, but many of the points raised can be applied to other sites to equal beneift.

Twitter's REST API is quite complete in that it provides access to nearly all the data one sees in a user timeline and profile. However, if one wants to write a tool to monitor changes in a user's metadata, one has to poll the API. That is, the monitor write needs to periodically query every Twitter API call that may hold interesting information. This is ineffecient for both the monitor writer and for Twitter, who has to serve a number of calls with information that has not changed since the last call.

What would be better is for the Twitter API to implement an Observer design pattern. Twitter would identify some number of interesting data modification events, such as the following list:

  • Add/lose a follower
  • Add/lose a friend
  • Screen name change
  • Location change
  • Profile description change
  • New direct message

While not exhaustive, this list of events would make the job of writing a monitor tool much easier. A possible workflow might look like this:

  1. An app registers with a new Twitter API a callback URL and a list of events
  2. When one of the events of interest occurs, Twitter calls the registered URL with a payload that includes the event that occurred and perhaps some domain-specific metadata (i.e. the ID of the follower who joined/dropped)

Because of the volume of users, the Twitter API is very sensitive to producer/consumer issues. I propose that Twitter queue up events and then send out notifications at some reasonable rate, perhaps imposing a callback rate limit. Even with a rate limit, the callback system will provide much better performance for tool writers than polling the API.

Please discuss below.

chicken beard science

Beta-testers wanted


Over the past several months, I have been slowly pulling together a project that I hope others will find useful. It is called Nestor.

Nestor is an analytics tool that provides trending information for followers and retweets on Twitter.

The service is free and not spammy. Since authentication is done through Twitter, I don't even collect email addresses.

What I need are users on the system who are not afraid to give me feedback.

I am looking to improve the following areas:

  • User experience
  • Quality of data
  • Improving data visualizations
  • Security
  • Protecting user privacy

If this is something you are interested in, please send mail to for a free registration code to get on to the system. Or message me on twitter: @taskboy3000

UPDATE: The registration part has been removed. Just log in with Twitter.

I want to be clear: this is not an open source project. It could become that at some point, but it is not one now.

All feedback is welcome.

chicken beard science

Freddie got fingered (wrongly)

This is an abbreviated post of rage directed at those who wish to pin the financial panic of 2008 on Freddie and Fannie. Whether F&F made poor quality loans (they did) did not directly cause the implosion that consumed Mighty Lehman Brothers, Merrill Lynch, Washington Mutual and many, many others. That distinction goes to the massive scheme of entangling, unregulated debt insurance known as credit default swaps (CDS).

Credit default swaps are insurance contracts that cover the cost of loan defaults. The trick here is that anyone can buy a CDS on liabilities they do not own. Hence, CDS became the gamble of choice on Wall Street through the 2000s. And baby, the CDS business was a-boomin' until 2008.

Because CDS are private contracts between two parties, we will never know the size of that market, but it has been estimated to be in the trillions.

AIG provided many, many CDS, which is why, when the house of cards that began with Lehman fell, AIG got hit very hard. And at the end of 2008, it looked like the Street would have to pay out those trillion dollars worth of insurance.

Does TARP make sense to you all now?

Break up F&F if you must, but that will in no way prevent another crisis of 2008 from happening. Regulating the invisible CDS market will.

This issue isn't that complex. I'm tired of politicians getting this wrong, even if the error is intentionally malicious.

chicken beard science

On 11/11/2011

Next year, the world will not end, but a sequence will. On 11/11/11, regardless of whether you live in Europe or America, all the bits of the calendar will be turned on. That alone should make it some kind of international peace day. However, this date also triggers some unwanted thoughts in the minds math and computer folk.

For example, this date like a pattern of bits in a byte, which can also be represented by the hexadecimal number 0x3f, or decimal 63 or 2**5-1. And then I think, "I should really refresh my memory on 2s compliment" or "maybe this a netmask to local subnet" or "jeez, there are two whole bits wasted in that pattern."

What I'm saying is that computers have ruined my mind. So having dates like the one above isn't safe for me.

Luckily, this is the last time in my life I will be confronted with such dates. I've lived through the following:

  • 01/01/00
  • 01/10/00
  • 01/11/00
  • 10/01/00
  • 10/10/00
  • 10/11/00
  • 11/01/00
  • 11/10/00
  • 11/11/10
  • 01/01/01
  • 01/10/01
  • 01/11/01
  • 10/01/01
  • 10/10/01
  • 10/11/01
  • 11/01/01
  • 11/10/01
  • 11/11/01
  • 01/01/10
  • 01/10/10
  • 01/11/10
  • 10/01/10
  • 10/10/10
  • 10/11/10
  • 11/01/10
  • 11/10/10
  • 11/11/10 (arriving shortly)

Left on the calendar are the following harrowing days:

  • 01/01/11
  • 01/10/11
  • 01/11/11
  • 10/01/11
  • 10/10/11
  • 10/11/11
  • 11/01/11
  • 11/10/11

So, Nov. 11, 2011 is the last of a pattern that will not repeat this century. Who will ignore the ominous tidings that this date portents?

chicken beard science

After the GOP takes the House in 2010

There can be little doubt that the GOP will become the majority in the House after the midterm elections. The new speaker of the House will be Orange Man Boehner. What will the next few years look like?

Expect no significant stimulus packages (although Pork will still be served hot'n'fresh from the usual suspects). The rich will get tax cuts. Perhaps the payroll taxes (which affect nearly all of us) might increase.

The problem with government austerity right now is that no one else is spending money. If the government doesn't, public sector jobs will get axed (compare to the UK's estimated 500,000 lost jobs thanks to their recent austerity plan). Without those jobs, tax revenues fall causing more government debt and more economic depression.

The only positive thing about austerity is that it may improve the interest rate at which the government can borrow more debt. Super.

The time for government austerity is when the economy is booming. Tax the hell out of the top earners then to pay down debts incurred during times like this.

So the economy will be depressed until 2012. Obama will get tossed out of office. Don't expect Palin to run -- she's got a great gig as a celebrity and can't afford the pay cut. Whoever the Republicans run should take the White House. Then, tax cuts, tax cuts, tax cuts -- none of which will help the middle or lower class.

Remember that tax cuts are a government spending program, especially when tax cuts necessitate more government borrowing (which every state and federal tax cut plan I know of today will do).

Many people benefit from federal entitlement programs and no one wants to give these up. As a nation, we are completely screwed.

(Don't construe this as a pro-Dem rant. There are only bad choices facing the voter in 2010 and 2012. Why worry over the differences?)

Update: One more word on the UK austerity measures. It appears that their government seeks to fight unemployment and rising entitlement costs by creating a massive wave of new unemployment that will require more entitlement spending. Solid thinking there, old bean.

  • Current Mood
    pessimistic pessimistic
  • Tags
chicken beard science

Is the military ready for mere tolerance?

Imagine if a modern senator said:

"At this moment of immense hardship for our armed services, we should not be seeking to overturn the 'Racial Segregation' policy. [It is] imperfect but effective."


"At this point there is no reason to rush to judgment for political expediency until we hear from our military leaders as to whether they think it is a good idea to change this policy [of racial segregation]."

The ugly tone of discrimination in those words is obvious. Yet, these are slightly modified quotes from Senators McCain and Voinovich.

Now replace 'racial segregation' with 'sexual segregation' and the moral reason why the current 'Don't Ask, Don't Tell' policy is incompatible with modern sensibilities is highlighted in bold relief.

The military often likes to present itself as a modern, professional organization. In private business, sexual discrimination is illegal. Those companies foolish enough to try it get sued. Why should the military be any different? Is it really too much to ask our paid, professional soldiers not to be bigots?

Those that do not like homosexuals are unlikely to change their feelings any time soon. There is no need for a transition period: end the policy now. Put the argument in racial terms and see how horrible it sounds: "our fighting men need to get used to working with [black people]."

We dishonor our military by thinking that they cannot make this change or that most would not welcome it. Who are we to turn away those who would die to defend our country?

  • Current Mood
    uncomfortable uncomfortable
  • Tags
chicken beard science

LJ vs twitter

As a stream of information that can be commented by third parties, I still prefer to reaf updates via LJ than twitter. I also have realized that facebook is a social lifeline for new parents.