Distant relative of 5 whys. We need NoSql document store -> so we can store json blobs -> so we can do databasing at app level -> because DBAs with their insistence on schemas are slow. Oh, so we can solve the problems by hiring one DBA and maybe training two devs instead of hiring full dev team and refactoring stuff for a year?
Funny thing is we are now migrating stuff out of datastore (and new stuff is not in datastore to begin with) into an RDBMS, but we are doing it microservice style with each microservice having its own separate database. So relationships are now cross-services concerns...
Not that having EVERYTHING in a single DB is the best approach always, but IMO we should default to keep everything in one single DB.
Put up 2 or 3 read replicas, split your queries so writes happen to main and reads come from replicas (supported out of the box by many modern ORMs), and you can scale to millions in daily active users for most startup workloads.
Really the hard part of BI is that folks who need the info don’t wanna learn SQL. The ones who can do SQL, will struggle to keep up with your changing schema.
Data analysts are fine with SQL though. Every "get into data analysis as a career" course will teach you SQL (about 70% of what the querynomicon teaches [1]).
Yes! I haven’t seen startups hiring these though. Somehow I always end up doing this as a side-gig on my engineering job.
Add analytics reporting views in your app database as the 'API' is the way.
Ugh. That sounds good on paper, but in practice it can become a problem. You're making your _database_ schema a part of the public API. It's an example Hyrum's Law, people will, sooner rather than later, start depending on internal details of the data representation.
And your development velocity will crater, as you'll now need to update all the reports (that are not necessarily even tracked in version control!).
Investing some time early to add code to pull out the data relevant for analytics can be worthwhile.
There's also a question of the personal information.
Realistically this guide should be bifurcated in terms of scale.
Yes this is good advice, until you get really large scale you don't need anything more fancy than some SQL in a read replica.
Wait, was that the reason people were doing NoSQL? JSON support? I thought it was about sharding, write scalability, etc.
My favorite example is that Twitter used mysql for all tweets, writing ~5k/s 24/7/365, until about 2016ish. Well into being a public company with billions in revenue and 300mm+ MAUs.
3/4 companies in the Bay Area senior software engineer interviews require a System Design interview where they will tell you "what if you had 10m users" and expect a distributed write-heavy sharding answer
When I do get a chance to implement compliant processes at the beginning, it’s one of those amazing IT things where we prevent WW3 but never get the credit for it.
This is in many ways the spirit of SOC2, no? There are a lot of startup founders, far more than I'd like, who would purposefully eschew such "simple best practices" unless they had an axe like a SOC2 audit swinging over them.
I think you're both right, for what it's worth, and my take is that you are more aligned with TFA than you perceive.
(We have other ways of tracking prod changes, but our auditors don't know anything about them.)
SOC2 attestations being easy to get also runs counter to what I have heard from every single other person on this topic. Generally what I hear is that it is extremely hard and time consuming. What am I missing? I would love to be wrong here and for this to be easy.
It's time consuming, but not all consuming. I think I spend <2 hours a week on compliance now that we're set up.
The "fun" part was engineering ways to implement things like PHI scanning and WAF protection as cheaply as possible. There's almost always a nearly-free cron job/python script/slackbot alternative to every "mandatory" 5-6 figure SaaS subscription in the space.
do you have a policy for XYZ?
or confirm you have a process for "thing"
So what ends up happeneing is if you feel stressed about an audit, just getting a list of the audit, you will realize how much you can just say "yes" to and feel less daunted by the audit.
So, its a good self-check even if youre just crossing out the things you should have already have a framework for.
The whole intro reads like a puffy resume and lots of gilding. Even a section of gushing testimonials.
And he puts his name on the title so you don't gotta read the author byline. Total cheese.
Downside is there is a lot of startup founders that will need help getting the basics in place.
I worked in place where 2 business guys hired 4-5 freelancers and as freelancers took high salaries not even one of them had any clue about setting up infra or SDLC let alone secure SDLC. They would write the code and not give a damn about anything besides that.
Business guys thought they have great technical guys because they were expensive.
But maybe not go full attestation mode right away - but also tricky to find one.
Further: while checklisting tools may only cost a couple thousand dollars, the actual process of getting a SOC2 attestation isn't the real expense. I could get OWASP WebGoat a SOC2 attestation if I wanted to (a ham sandwich would be even easier). The actual expense in SOC2 is the engineering work you do in support of it. Those checklist tools are fine if you know exactly what you're doing and don't let them add any engineering work, but what I've seen happen repeatedly is a SOC2 checklist from a tool leading a team into building a pasteurized process cheese food security practice, with IDS and WAF and server agents and code scanners and Nessus scans, at great expense.
What made it easy was talking to a startup that wanted soc2 and had it themselves who recommended an auditor who helped us untangle what was actually required.
It took a couple of months to get type 1 from start to finish with very part time attention.
Don't try to get a compliance certificate at the last minute. Preparing for and conducting an audit such as for PCI DSS or SOC 2 from start to finish is a lengthy process, ranging from six to twelve months for most startups. Starting early and maintaining compliance is cheaper than starting late and doing rework.
This is basically the opposite of the advice I would give a startup. SOC2 attestations in particular are easy to get, and are a waste of money to obtain preemptively before there are purchase orders on the line for them.
There are things you should start doing early that lay the groundwork for attestations, but you should be doing them anyways, even if you never plan to get a SOC2 (and if a big-ticket customer never demands it, you shouldn't SOC2). That's stuff like setting up single sign-on and having protected git branches; simple best practices.
Anyone else want to spot check other parts of this document? I wouldn't feel qualified to challenge most of it.