News for hackers. Stuff that matters.
Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by raganwald on Wednesday June 15, 2011 @10:47AM (#2657310)
This is unacceptable, obviously. It's a massive failure of their security testing protocols. I'm not particularly surprised that an vulnerability like this would get written into the code, it's an easy mistake for an inexperienced developer to make. But I'm not going to pile on the developer. We laud "separation of concerns" in our architecture, and this pattern applies to the organization of software development teams.

I don't expect every developer to be aware of every vulnerability. But I do expect that a financial institution has a specialist somewhere that audits the code before it is sent for testing ("white box"), and then I expect them to have an independent audit team probe for vulnerabilities ("black box").

After the inexperienced developer has had his code rejected for various flaws, he will become quite aware of the obvious ways things like this can go wrong.

Don't get me wrong, I expect that the vast majority of developers wouldn't make this mistake in the first place. But if you aren't a specialist, it is pure hubris to think that you write code that is hardened against all of the attacks out there. And if you have a vulnerability, it really doesn't matter if it's an embarrassingly simple vulnerability or one that requires sophisticated techniques to uncover and exploit. Either way, you're road kill.

High-value targets like banks need two security specialists (the code audit and the penetration test) to accompany the development specialists. That's simple separation of concerns, and it works as well in team organisation as it does in code organisation.

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by tptacek on Wednesday June 15, 2011 @12:05PM (#2657661)
I don't know anyone at Citi, but I do know several firms that do audit & pentest work there, and I'm told it's high-volume --- in other words, it's likely every app gets tested. This is a startling miss.

When you engage a software security firm to check out an app, you're trusting that their report is reasonably complete. As a practitioner, this is basically your nightmare scenario. You know in the back of your head that it's always possible you're going to miss things, but a vulnerability this trivial seems like the kind of thing you find on the first day of testing.

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by joe_the_user on Wednesday June 15, 2011 @12:14PM (#2657709)
This is the kind of vulnerability that should be caught on the whiteboard before a single line of code is written.

I know this isn't what you're talking about, but I hope that an institution that's protecting my money will formally specifying their protocol and not haphazardly putting their app together with "Test Driven Development". But maybe that's how they managed to be this stupid.

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by tptacek on Wednesday June 15, 2011 @12:21PM (#2657737)
Though there are no doubt very, very smart developers scattered across Citi, please bear in mind that the wiring-up-form-fields-using-J2EE jobs in large financials are among the least prestigious in our field. These are the types of apps that are among the first to get outsourced.

Your idea that the input and output paths through a typical banking J2EE app were carefully whiteboarded seems mythologized, based on what I've seen at fisrv companies.

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by joe_the_user on Wednesday June 15, 2011 @12:42PM (#2657837)
That's hair-raising. I have only a small amount of financial firm experience. I'm just used to ordinary software development being white-boarded first.
Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by tptacek on Wednesday June 15, 2011 @12:56PM (#2657896)
People forget that statistically the devs hanging around on HN are in the top brackets. The overwhelming majority of devs are 9-5 paycheck players with exactly the level of competence required to execute J2EE and .NET CRUD apps that pass internal acceptance testing.

If you are an enterprise dev with a knack for making web apps that are't horribly insecure, and you don't have a phantasmagorically awesome comp package already, more advice: start looking for new gigs. You're underemployed.

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by noonespecial on Wednesday June 15, 2011 @04:31PM (#2658905)
The problem is to the PHB's in HR, there doesn't appear to be a difference between you and the guy who just makes it work, security-be-damned except that you likely take longer and cost more.
Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by auzieo on Wednesday June 15, 2011 @01:39PM (#2658169)
What's the lower bound for a phantasmagorically awesome comp package? I am not good at evaluating the fairness of my current salary.
Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by jacquesgt on Wednesday June 15, 2011 @04:56PM (#2658987)
In a market like the New York, Boston, or Seattle areas, with 5+ years of experience, a top tier software engineer should be able to pull down $100k salary plus 401(k), a good health insurance package (ie, something better than a high deductible plan plus HSA), and long term/short term disability insurance. Adjust the salary number for your local cost of living.
Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by Robin_Message on Wednesday June 15, 2011 @12:30PM (#2657781)
Online bank websites are, at least in the UK, universally terrible. They seem to be just a very, very thin layer on top of the bank's back-end systems, giving such delightful results as being two different sorts of recurring payments and two types of non-recurring payments I can do from my online banking (actually, I can't set up or cancel every type from within online banking.)

Do banks make so little from checking/current accounts that it isn't worth spending a dollar per customer to build a decent solution?

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by tptacek on Wednesday June 15, 2011 @12:34PM (#2657796)
Customers pick banks based on how many ATMs they have, not the quality of the bank's web apps.

Though there clearly are people who care deeply about bank UX, those people are not normal.

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by Robin_Message on Wednesday June 15, 2011 @01:04PM (#2657948)
(UK has a shared ATM network, so it's not the same issue it is the States where I fondly remember hunting for damm BofA machines. I agree there are other reasons to choose a bank than website.)

However, it seems like there is relatively little to distinguish between banks, and a web service that truly pleased people might be a good way to get and retain customers, whilst cutting costs (branches and call centres.)

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by gnosis on Wednesday June 15, 2011 @03:52PM (#2658769)
Even for those people who do care about the quality and security of their bank's web services, how many of them are really going to take the time to compare the web services of various banks?

How do you compare them, anyway? It's not like banks give out trial accounts you could use for this purpose.

You'd have to go to the trouble of signing up for a real bank account at each of the banks whose web services you'd like to evaluate. Then spend the time to try them out and compare. So even a cursory inspection of a variety of banks' web services would already be a pretty huge hassle.

Even if you go to all this touble, you're probably not going to know much more than how intuitive a given bank's web interface is, or if they did something glaringly awful in terms of its usability.

You wouldn't know how accessible the interface is under load, or test how well the bank's interface handles the various real-life corner cases you're likely to encounter over time.

And how would you test the interface's security? Is even a reasonably security-aware consumer really going to try breaking in to the bank?

The ironic thing is that most users who use online banking probably do care about the web interface and its security, even if they can't articulate it in so many words. But of them, only a small minority care enough and aware enough of the issues to take the time to comparison shop. And even then they probably won't find out much.

There's an opportunity here for a Consumer Reports style investigation in to the web interfaces of various banks. That would be a valuable service that could save the consumer a lot of time and hassle, and maybe even do things that a lone consumer couldn't: evaluate interface security.

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by mrb on Thursday June 16, 2011 @05:15AM (#2660572)
I did just that when I last needed to open a bank account :) I googled for users' comments about the bank's web interface (was it working fine in non-IE browsers, not using Java applets, etc).
Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by pero on Thursday June 16, 2011 @03:23AM (#2660394)
Out of the major banks in Canada, the one with the most ATMs is the smallest--and incidentally also has the highest Forrester's marks in online banking.
Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by billybob on Wednesday June 15, 2011 @03:35PM (#2658683)
I agree with you about planning, but "test driven development" isn't haphazard; it's not just clicking around to see what happens. It's a way of formally specifying your design that simultaneously confirms that your code does what it should.

For instance, you write a test that says the user should get a "not authorized" if they try to access another user's account. You run the test, and it fails. You change your code to make it pass. And every time you update your app from then on, you re-run the test and make sure you didn't make that test fail.

Having a design on paper isn't as useful as having one that you can PROVE your code conforms to.

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by bguthrie on Wednesday June 15, 2011 @07:48PM (#2659565)
There is no guarantee that anything specified on a whiteboard will actually make it into physical code, particularly if it's handed down from on high. Banking websites are terrible not because they use TDD, but precisely because of the supposition that a good architect and a whiteboard is enough safeguard to skip them.

Tests make it possible to refactor your code towards simplicity and clarity, making security errors more clear. If you're writing a banking application, ignoring them is simply foolhardy.

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by kaiwren on Wednesday June 15, 2011 @08:24PM (#2659635)
So you're taking a ridiculous loophole created by enterprise architects that probably spent six months whiteboarding this shit before letting the devs write a single line of code as an excuse to troll on the subject of TDD? Lame.
Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by ashwinraghav on Thursday June 16, 2011 @12:45AM (#2660138)
Firstly this kind of vulnerability requires no white-boarding. It is fair to assume that a dev with a reasonable understanding of how the web protocols work will not need a white board to figure this out. So firstly the argument of catching this on a WB is a little juvenile.

Associating TDD to being haphazard is pretty random. Being hap hazard and TDDing are really in 2 different math spaces. Care not to mix em?

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by jez_h on Wednesday June 15, 2011 @07:59PM (#2659590)
If you're "haphazardly putting your app together" then you're not doing "test driven development".

TDD and using your brain coexist quite happily. But if you're not using your brain, then TDD won't save you.

So no, that's not how they managed to be this stupid. They managed it through not using their brains.

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by jbrechtel on Wednesday June 15, 2011 @08:06PM (#2659605)
Could you please elaborate on why you perceive a correlation (or causation?) between TDD and haphazard development?
Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by keenerd on Thursday June 16, 2011 @03:22AM (#2660392)
Citi fails on a lot of other basic common sense practices. For example, their password fields are limited to 8 characters.
Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by raganwald on Wednesday June 15, 2011 @10:53AM (#2657340)
Unrelated anecdote:

When working with a substantial financial organisation, there were strict rules prohibiting my team from communicating with the code audit and other security teams. Going for coffee with one of the internal audit people would get you both fired. The external testing was done by a security consulting firm. Their reports were passed on to me with the identifying details stripped out so that I wouldn't even know who they were.

Talk about "low coupling!"

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by biot on Wednesday June 15, 2011 @01:03PM (#2657938)
No doubt the goal there is to prevent collusion between a developer who introduces a subtle but exploitable bug and the auditors/testers who might be interested in failing to find such a bug, for an appropriate percentage of course.
Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by tptacek on Wednesday June 15, 2011 @12:22PM (#2657742)
It shouldn't matter to you who the external firm is. Stripping out their contact info, at least in the archives, is probably smart; it makes it easier for the company to keep multiple security firms working on their projects. Which, when you're a huge bank, is what you want to be doing.
Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by InclinedPlane on Wednesday June 15, 2011 @12:25PM (#2657756)
Sadly the problem is at a higher level than that, and not restricted to security. Today software development might as well be magic. There are no reliable objective methods for determining developer or software quality. The only method that's proven to work well is the subjective judgment of another "magician". The same applies to software security as well.

This creates a boot strapping problem. If you're an organization of magicians then you have a comparatively easy time finding other good magicians. But if you don't have that hsitory and you still have the need you won't be able to know the quality of people you're hiring except on the least granular of scales (lots of failed and low quality projects). This is a big reason behind so much of the problems of software development in "the enterprise". Because people who know how to run a bank or a media company haven't the slightest clue how to run software development, and there generally aren't trusted 3rd parties that people can go to for help.

Worse yet, there are perverse incentives at play. If you can't tell the difference between a good security person and a stunningly mediocre one, you are going to go with the cheapest one who you think is good enough (generally: solid job history, resume fully buzzword compliant, etc.)

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by lifeisstillgood on Wednesday June 15, 2011 @01:07PM (#2657968)
>>> Today software development might as well be magic. There are no reliable objective methods for determining developer or software quality. The only method that's proven to work well is the subjective judgment of another "magician".

Try replacing software development with writing english.

I am going to stop banging on about this eventually, but reading and writing code is something that needs to be understood at all levels of a company before that company gets good at it. If the boss of the company and everyone who reports to him is illiterate, one cannot expect the memos and the policy documents and the amusing posters to be of any decent quality

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by joe_the_user on Wednesday June 15, 2011 @12:09PM (#2657681)
Hmm,

Perhaps the problem is the "security specialists".

I agree that just about any reasonably skilled developer knows to, at the absolute minimum, include a user-specific session key with every transaction.

But I've also seen "security specialists" get a pass for a variety of bad behaviors. And we've heard of the complete incompetence of HBGary. My hunch is that the secrecy/obscurity that the passed for security in pre-Internet culture is exactly what allows total screw-ups like this to happen in these large organizations. The gold-plated consultants who are friends with the VP wind-up being far less competent than your average/decent developer.

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by tptacek on Wednesday June 15, 2011 @12:17PM (#2657723)
There's probably a kernel of truth to your comment, but that last graf makes no sense. The "security specialists" aren't writing the code with vulnerable direct object references; the "average decent developers" are. It's true that there are consultants out there that don't improve the (poor) seaworthiness of typical developer output, but few of them are so bad as to make the situation worse.

And, the notion that "good" developers turn out uniformly secure code is delusive. We're lucky to have gotten to work with some of the best run, most carefully recruited teams in the industry. You find terrifying things on all these engagements.

My advice, again as a practitioner: if your security firm isn't finding terrifying things, at least on the first go-round with an app, ask very tough questions. Ask if they need an extra week or two to catch up and get real findings. Then ask which of the two of you should pay for those weeks.

We are still finding code execution on a good chunk of our web gigs, even at the "good" companies.

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by Locke1689 on Wednesday June 15, 2011 @03:16PM (#2658602)
Just as a corroborating point, although I'm not a professional security developer I've been an interested bystander since I was a stupid, self-taught, grey hat 16-year-old 5 years ago.

Anyway, I once found myself writing some PHP code to demo a slightly complex SQL injection attack for the class I co-lecture at Northwestern (Network Security and Penetration). This code purposefully had a SQL injection vulnerability in it. It wasn't until the third reading of my own code that I noticed that I mistakenly dropped a CSRF vulnerability in alongside it. CSRF was literally the topic I was teaching next Monday and I put one into my own security code accidentally.

Secure code is so difficult to write that I can't believe that even the best developer writes secure code much of the time. Hell, apparently even I can't write secure PHP when I'm looking straight at it.

Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by joe_the_user on Wednesday June 15, 2011 @12:57PM (#2657903)
I worked at a company where we had an individual hired as a "security specialist" who wrote code and got more or less a free on his behavior and approach. I suppose that's different than a full security consultant.
Re:How Hackers Stole 200,000 Citi Accounts Just By Changing ... by tptacek on Wednesday June 15, 2011 @01:05PM (#2657952)
Your pentest firm should not be writing code for you. That conflict of interest is pretty obvious. If I had a consultant doing that for my company, I'd retain a rival firm to review the code, and make sure both the consultant coder and the pentester team knew about each other.

If you told me someone at, say, iSec Partners (a firm I like) wrote something I was pentesting, I'd go f'ing nuts trying to find flaws in it. If you told me iSec was reviewing something I wrote, I'd stay up nights thinking of ways to shore it up.