I live in a community governed by an HOA (even if I am on the board) I think pretty much ever facet of my life is a walled garden. Makes it easier to accept.

There's no doubt twitter's useful. I use it myself daily. But I'd argue acceptance of the growing world of walled gardens is dangerous to innovation. I was sad reading your article because if this wasn't about twitter you could've written about how you SOLVED the spam problem, instead of begging a large company to smile upon your idea and maybe try it out.

Well, I supposed I could write a Twitter client that checked links and marked tweets as spam before presenting them to the user.

Sorry, it's my nature to try and solve problems working around whatever obstacles are in the way. So when your post made me think, well.... I could... In no way mocking or disagreeing with your opinion which I do comprehend and for a large part agree with.

On the same token I also think a truly open system, while not impossible, is a difficult proposition for financial reasons.

Also from a security stand point. If the system was open enough for me to solve the spam problem, then it would also be open enough to make the spam problem, or even virus/malware delivery much easier.

You can get ~1% of public tweets streamed to you at http://stream.twitter.com/1/statuses/sample.json (see docs at https://dev.twitter.com/docs/streaming-api/methods). You could develop a proof-of-concept using that.

Is that open to anyone now? I thought the process was you developed against user streams then once you scaled beyond those you requested access to the stream api? And, you need to request access to the user streams api too right?

I decided not to do real time streaming for my current project so never followed up on it.

*edit: Actually looks like I'd want access to statuses/links which would solve the problem of filtering for links on my the scanning end.