We, unfortunately, edit the live db all the time.

Hope everything doesn't blow up.

It's not ideal.

All dev and testing is dont against the live database. YOu just have to be very, very, very careful.

It's something we are trying to implement now, honestly. It's on the TODO list.

The problem I'm having is where do you start? How do you simulate the data flowing through live in the test / dev enviornment.

It doesn't take that long to put some safeguards.

Here is the thing: WHENEVER you say "you have to be VERY careful" or blame someone for making a mistake, that's a place you need to put more checks. The computer should check routine things, not people.

"The problem I'm having is where do you start? How do you simulate the data flowing through live in the test / dev enviornment."

I am guessing you use MySQL so simply set up MYSQL REPLICATION and test all your stuff on the replica database first. Actually you should periodically dump the replica and test on a snapshot, so you can mess up the snapshot.

Main db -> Replica -> Snapshot.

Got any good articles on how to setup test environments. I would then need all the dev code to point to the test one, not the live db and any Web APIs to be able to switch to point to the test db (which must mean running 2 web apis, a live and a test one...)

Seems like it gets very complicated, but probably worth figuring out.

We don't even have CI yet.

If all you want to do is functional testing of the application by yourself, then it is OK to deploy a copy of the software on your desktop along with a test database. I do this for the application I work on, and when I need to test against MSSQL, the Express edition is good enough.

If you need to do load testing or integration testing or other more complicated forms of testing, you usually try to duplicated the expected production environment as well as you can. This may not be possible, so you may have to make decisions about what's most relevant and simply accept those tradeoffs.

EDIT: Don't worry about CI or anything else that fancy and complicated for now. You may not even need it; that depends on your team and how often your application changes. Just for now take the simplest steps you can until you get this basic stuff figured out first.

> How do you simulate the data flowing through live in the test / dev enviornment.

Most don't, rolling with the schema and 'minimum viable' data flow instead.

Make darn sure your backups work by testing restores regularly. Restoring a production backup elsewhere may be an option if it's possible to avoid sensitive data (scrubbing afterward introduces risk of missing something, and should be scripted/repeatable).

Exactly matching production data flow is not as important as avoiding accidentally hosing the prod db.

1. Set up replication, with your test db the replication slave of your production database, such that changes in production are mirrored in testing nearly instantly.

2. Take your production database backups (you do have those, right?) and restore them to the test database. This is less fresh than a replication setup but if you're just doing functional testing it should be enough to say "Well, okay, this little change isn't going to blow everything up."

Depending on what you're working on, chances are there even are people who use _security_ as an excuse to basically prevent any halfway useful emergency strategy one could think of when handling data impossible to pull off (oh the irony).

This isn't really an issue at all but can upset those with OCD-ish tendencies.

There are arguments for and against any technique, but this one seems really ridiculous to me, especially if you consider what any "countermeasure" would do to the code (which is usually much more important than the data in the long run!).

It's less about direct "pwn3ge" than about business intelligence. The number of users could be sensitive information for some businesses, but it's not just users. Any item where your volume is valuable proprietary information, you better make sure you aren't exposing an auto-incrementing id. Number of accounts, number of daily orders, number of distribution centers, number of product models, these might be data points that a growing business would prefer not to expose to their competitors via, for example, URL query parameters. You should make competitors work harder for their intelligence.

Imagine that the social network does a rewrite, and rather than converting all the old accounts/data it just uses an additional flag to indicate which database to use. In that case a user id would not be globally unique for the website, but it would be unique for the specific database.

This is actually more likely than it sounds. The company can hire a fresh new team to make an experimental new site, and if it's a hit, they can make some ugly hack to do calls to the old website using web APIs to get any old users from that system. This way the new team doesn't need to learn and figure out how the old system was built, which likely lacks any documentation and where the original developers has left the company.

Without a full understanding of the layers of indirection it would be impossible to determine with certainty the number of users a social network has.

https://www.troyhunt.com/kids-pass-just-reminded-us-how-hard...

Also, this is only the case if your raw user IDs are visible. You only need a very basic encryption scheme to hide them. It can be as simple as taking two large prime numbers and sending (uid + x) % y to the user.

Mark Zuckerberg is famously 4 and you can find a lot of famous silicon valley types in the first million user ids :)