HTTP proxy - Fairly trivial, annoyingly harder if you want to generate certs to MitM SSL.
Scaling it out - not too bad. fuzzing is highly parallelizable. How your parallel tasks work together is more challenging, (think rotating cookies) but do able, and you need to really do this right to maximize requests/seconds.
The audits themselves - If you actually want pre-built attacks, basic attack+response regexs (ala Nikto) are very easy, though building the DB is time consuming. Multi-request tests (looking for stored XSS, etc) are harder because an audit thread need to share state, or become somewhat sequential.
Dealing with the modern web - Harder. You need a JavaScript interpreter. I used Spidermonkey years ago, PhantomJS and the like more recently. Also means you need to support different packaging formats like JSON, MVC-style path parameters, etc and the associated escaping.
Keeping state - Harder. What does logged out look like? How do you detect this? (and no, don't make the user supply a Regex...) Has you audit screwed up your account so that "logged in" is all jacked up? Was there Stored XSS, but another audit clobbered the payload before your sweeper found the reflected value? (think updating your address on one page, displaying it on another, all while multiple things are auditing that update address form...)
All that said, I have found the main challenge people overlook when writing web pen test tools is the UI/UX. A shitty UI/UX ruins a tool because it ruins your flow. How do I specify what to fuzz with what rules? Can I resuse and reapply settings easily, how are results reported? Can I easily mark something as a false positive? But can I do that in a general way so that all issues matching on this host, or this file, or this URL frag are ignored? etc etc etc.
Please, if you are going to go down this path, think about how people are going to use the tool, and focus on maximizing their throughput.
So, if you asked me what the hard parts of getting Burp right when I took a stab at replacing it in Cocoa, and more recently in ko.js and Riak (more successfully):
* There is just a whole lot of UI involved in making a navigable proxy with the breakdowns you want, and, in particular, with editing and annotating raw requests and having those views do the "right thing" as the user edits the request and flips back and forth between views.
* Storage of requests is hard, because it's not intuitive how they need to be stored. The proxy wants long-term storage with indexes; the repeater wants rapid writes; the fuzzer wants to generate a shitload of requests (multiplying one request by 20k isn't rare), and a request from one part of the system can bounce through all the other parts.
* The proxy wants realtime communication/control with the UI, because some of your requests need to be intercepted and modified before delivery; in Burp, that UI shares most of the controls of the Repeater, which is surprisingly complicated.
Network performance is the easiest part of this to get right. Just don't use Java threads, and use a real timer library. But getting the network performance to cooperate with persistent storage is harder, unless you just let storage and memory explode.
We wanted fine-grained control over all of the tools so that we could automate retesting and regression. Like I said, we had a lot of luck hoisting the components of Burp into a series of web applications and backend systems backed by Riak (and Postgres, for search, index, and UI). We replaced Intruder with a command-line tool written in Golang that has a small specification language for the rules, instead of Burp's clunky interface, but there's still a whole lot of UI involved in parsing a request, automatically finding the places to template-ize, and then allowing the user to alter those selections.
When I took my stab at doing this in Cocoa a few years ago, I made a beeline for NSWebView; having a real webkit render is an enormous win. We do something similar to you with headless webkit now. This is a place Burp is genuinely constrained, due to Java.
Unfortunately, replacing Burp is a bitch.
It just happens that this is a conversation that happens semiweekly at Matasano. :)
These are all valid points. Writing security tools is technically very challenging even for the most basic things. Even a spider, regardless how basic this looks like in principle, is really hard once you get down to implementation. "In theory there is no difference between theory and reality but in reality there is."
This is why I am a big believer at simplifying the workflows in order to be more manageable. One of these workflows is recording requests and responses generated to/from the application. This is what Proxy.app does well. This feature can be extended further by other tools which we have already developed.
Proxy.app is not meant to compete with Burp. Burp is a different beast and not necessarily, in our humble opinion, the best approach to penetration testing web applications. We know how difficult it is to write security tools because this is what we specialise doing. So you comment resonates with us very well.
Although we could have bundled-in all our security testing frameworks, it was decided to take a different approach for the time being. Later on we may decide to add security testing capabilities or these features may be provided by some of the other tools that we are currently working on.
This turns out to be a pretty difficult thing to build well. Every pentester who can code that owns a Mac wants to rewrite Burp; I myself have died on that hill several times. But rewriting Burp is a bit like rewriting Microsoft Word; so much of the value is in the details, and there are so. many. details.