1) Don't mess with the database. Back it up, remotely.

2) It's good to have a semanticly sane code base with all the common pages elements factored out into templates, but it's not a requirement. If it ain't broke.

3) Involve the community. Have a poll asking your users what to fix first, and start from there.

4) With a copy of the database, you can start developing the site using better technologies and processes; invite a few users to test the redesigned site, preferably on another machine. Report back to the community, publish a newsletter, show screenshots, etc.

5) Don't fuck with that database. Don't lose the server logs.

Good luck.

Never ask your users what they think before you ask your logs what they do.

Users lie. Users misremember. Users are inexpert at describing their own use of your software.

Logs don't lie. Logs don't misremember. Logs are experts at describing your users' use of your software.

If for some reason you didn't build your web app with an eye to getting useful information from your customers, start adding Google Analytics (or one of the numerous copy/paste Javascript competitors) and give it a few weeks to start collecting data.

Thanks for the reply! I have infact been talking to users as well as watching the logs and analyzing google analytics data. I do have some insights on whats missing but as I said, the real challenge is in designing interfaces that can solve these problems. For example how do I display primary and secondary information on a profile page. What should be emphasized and de-emphasized. Since these are common problems, I was wondering if there is a book or something that I can readup to get some answers/pointers. Profile page is one example. Pages like account setting or stats page are other common pages that should have some standard design patterns.

In regards to books, I would recommend at least checking out Steve Krug's Don't Make Me Think, possibly some other design orientated books and then read up upon usability in general.

For example, take a look at: http://www.usabilitypost.com/ or http://www.uxbooth.com/ and http://useit.com/

You may wish to find a usability consultant too, or look for someone who hasn't used your app before with experience in this area to give you a few pointers.

I've posted a reply to you on my blog (cause it's too long for here): http://bjclark.me/2009/07/03/how-do-you-redesign-something/

Excerpt: The biggest thing to manage in a redesign is risk. Risk you alienate your user base or do something that negatively impacts revenue, or whatever you're most afraid of.

The easiest way to manage it is to shorten your feedback cycle (not user feedback, any type of feedback) to the smallest possible interval. When talking about design, this means you need to spend the least amount of time on the design you can before you put it in front of someone that exemplifies your average user (persona).

Looked at your reply! Very good for the usability part. Whether it's possible to start with the skin first depends on how much the changes impact upon the underlying processing.

Don't think of it as starting with the "usability" part. This is starting with the "user" part. What do your users want? What do they need? Choosing the users that you want to support and what features they need will drive what you need the backend to do and thus the underlying data-structures needed.

Really depends on what needs redesigning. You are basically talking about three interlinked projects. As far as possible it is best to chunk down into lots of small tasks, keeping them as separate as possible. That's one of the huge benefits of webapp world, you can maintain clean separation between functions. I'll assume that you can't just build a complete new site, migrate the data and axe the old one.

If the database needs redesign, then you'll need a way to have transactions from the current database propagate through to the new one. You don't want to break anything from the users' perspective while the redesign takes place.

If business rules need to be improved, then you'll have to make a decision whether to hit the current database or the new one.

Finally, the skin/usability part, keep it modular so that you can incrementally 'improve' your site. You would call the business rule codebase from either the old or new part as appropriate.

Just look at Amazon, eBay, Google, etc. They do continuous improvement without breaking much at any given time. The key is to keep each change small, test and be able to revert if something goes wrong. The reversion needs to be designed into every database change to minimise pain and lost transactions. Of course, good project management is key, yes! even with just the two of you working on it. "Failing to plan is planning to fail." :-)