Why you should aggregate intelligently

I was look at the frontpage of twitter a short while ago and this is what I saw:

This front page aggregates the latest 9 updates from twitter users. These updates can contain links of any kind. What’s going on here is that 4 of the updates are from Media UK TV and 4 are from Media UK Radio, two accounts owned by mediauk.com, a commercial website. Each one of these 8 updates has a link to a page on mediauk.com, so it’s entirely possible that this website is purposely spamming twitter’s frontpage to get traffic. And, with tools that hook directly into twitter’s API, it’s very easy to automatically create twitter updates whenever you update your site, which means that you can definitely stuff your twitter account with updates pointing to your own website and do it fast enough to keep the frontpage of twitter stuffed with incoming links.

All of this could be avoided if Twitter would use a better method for aggregating updates on their frontpage. As much as this “web 2.0″ era encourages trusting your users, you can’t be so careless in a situation like this. You have to prevent potential “front page” spam, and here are 2 ways to do it:

• Ensure that no user gets more than one update displayed on the frontpage at a time by putting the list of updates to display in an array, one by one, and searching that array before adding the next update to prevent duplicates.
• Log each update displayed in a period of time (maybe 3 hours for Twitter, longer for smaller aggregators) to ensure that no user gets more than one update displayed that period of time by checking every new update against the log.

With these methods (especially with the two combined), you can ensure that no user will be able to essentially spam your aggregator by simply “rapid-firing” their updates. Twitter should definitely do this immediately, before their frontpage turns into a billboard for commercial sites.

Updated seconds later: Media UK responds?