I've often wondered about the efficacy of having a simple catch-all script that just tracks everything a user does on the site. I would think it would pound the database relentlessly under even modest load.

One trick is to not use a database but a flat text file. You can optimize for querying nightly.

AWS comes into its own for these kinds of use cases - make a lightweight autoscaling events API on Elastic Beanstalk pumping events into DynamoDB coupled with a bunch of workers running at the EC2 spot price sorting and processing these into a traditional RDBMS = massively horizontally scalable events handling for pennies an hour.

You could read all the responses into a Queue-type structure and then populate the database in the background.

Clickish can do this with GA is you wanted. It's also intelligent with how it routes request back to GA. http://www.clickish.org

Then you have a lot of form data including passwords in your poor-manDB

That's why he said "Set up a separate DB with one big table" ;)

I think it can actually work quite well, provided you can setup a separate service just for analytics. It's not too unreasonable to assume there are sites already doing this since the concept of a central repository that you alone control (and isn't provided by a third party) can be appealing.

Also, you have to worry less about privacy and just concentrate on keeping your own systems and services secure.

Or a table with two columns, Key and Value. Index by key. Specify keys in code with constants. No need to alter table when adding new metrics.

Please don't do this! Entity-Attribute-Value tables are a nightmare waiting to happen about 98% of the time.

If there's something wrong with your DBMS where the alter table to add a row is an expensive or dangerous operation, just add a new table for every attribute, having lots of tables in a database is no worse than having lots of variables in a program.

If your DBA gets angry at you for adding new tables all the time, get a new DBA.

This!

I have a system exactly like this at my company, with a prettier UI on top of the SQL querying. I've been considering open sourcing it for a while; I wonder if there really is a demand for this, or if it's one of those things that everyone prefers to roll themselves.

Nothing wrong with putting it out there and no uptake. At least you won't have pull requests to deal with.

But, seriously, everyone rolls this themselves and does it wrong. If you've done it right, and can release it, you'd be doing good for the world.

Very interested.

yeah or I could just pay them $25 / month and let them pay for bandwidth on their own server time

you still have the issue of another party domain and all the blockers