So perhaps the role of the DBA isn’t necessarily dead, it’s just moved
to its new home at the datastore-as-a-service provider. The successful
DBA will understand that this new world means handling petabytes of
data and billions of operations on thousands of logical databases.
They will cope with less mature database technologies in increasingly
difficult workload environments. They will automate or die.
Long live the DBA.
The whole post was making the exact same point as you in the end, the author actually runs a datastore-as-a-service business.> The successful DBA will understand that this new world means handling petabytes of data and billions of operations on thousands of logical databases.
is untrue. Most of our customers have DBs that are in the GB size. A few have TB size DBs, and none are on that scale.
datastore-as-a-service doesn't replace DBAs - we make good money being remote DBAs for people who are using datastore-as-a-service providers, because they still run into the same problems as everyone else.
I wish our DBAs were like that. I am a developer with Sublime Text. I have to make the queries fast. I have to design good indexes.
They can complain if a query is slow, but they never actually help to fix it. They only have to make and restore backups when disks die.
Sorry to hear it, either way.
> You can't get a DBA from a college, from a technical school, or from any other form of formal education. Most DBAs these days are grown internally from developers or system administrators who decide to (or are forced to) specialize while on the job.
I can't say how much truth is in this statement. This stuff if learned organically by doing it on real world projects. It's scary at times to think that you just can't teach this stuff.
Yup. And with the supply being so low, it's hard to get a DBA (you'll probably have to steal one from another business), so people are flocking more and more to DBaaS, and DBaaS providers are more than happy to propagate the fiction that "you don't need a DBA, you have us!".
It honestly doesn't bother me much that they make these statements; it's marketing.
On the other hand, believing those statements harm our customers; they spend time and money to migrate to these providers and find out the hard way that they still need someone who can handle their DBs for them. That does bother me.
Databases are becoming pretty good at managing themselves and the marginal performance gains from tuning usually are easily offset by throwing bigger kit at the problem or throwing more cash at the plan you are on.
More memory? Increase the buffer pool size.
Faster HDD? Tweak the settings that determine how many disk operations are attempted every second.
Bigger CPU? Figure out the point of diminishing returns on the number of CPU cores for your DB, and start sharding onto multiple DBs to make sure you can use all of the cores.
SAN? But I thought you wanted performance. ;)
Plus, what gives you the best DB kit for the buck? I could probably tell you that (I am a DBA, and get paid to answer those questions), but do you know? Do you know where to find out?
As are most databases (which are 10-20 years old).
Compared to a webserver, a DB is significantly more complicated, and significantly more important to your average business. Nginx takes a nosedive or performs poorly, and nobody really cares. Your DB takes seconds to respond to basic queries, and your entire business suffers.
I'm guessing you've yet to be bitten by DB performance issues that couldn't be resolve by adding indexes or basic query profiling. I'm glad to hear it, because it's not fun to have to set aside your day job and dive beneath that.
I hope you're paying them well, because if they're all crack DBAs on top of being crack developers, they're the proverbial unicorn, and are probably worth $2-3 hundred thousand apiece.
Or your dataset and query volume will not grow to the point where you actually need a performant database, but that's a business problem, not a DB solution.
Development teams that go after speed will soon realize that re-serializing the data they need for a web page push and sending it over the network results in a factor 5-50x drop in performance over having that data in a local data structure. And God forbid you have even minor packet (like 0.0000001%) loss in the network between your webserver and database server. Your 99% tail latency will sprint for the 10s mark.
And as the DBA's here said : most databases are in the GB range, with a few in the tens or hundreds. SQLite will beat the crap out of any other solution at those sizes.
People don't realize how much tail latency affects maximum QPS. Once you calculate how one affects the other you see that tail latency is the enemy of performance in webservers. A webserver that can generate 100% of responses in 1ms can serve 1 million qps (ie. can saturate a 10Gbps link). At 99% at 1ms and 1% at 10ms (the very minimum to execute a single query against a database that isn't local to your machine), you're left with 900k. More typical database figures would be 20ms average and 600ms for the 1%, which will leave you with 30k QPS. In this example using a database cost you 97% of your original performance.
That 97% figure is perfectly normal. So if you want decent performance, using a database server (in the serving path) is just not in the cards at all.
But Oracle needs the finetuning, and I can't see that changing anytime soon, with Oracle some queries basically require that you use a IOT, other some are better with partitioned storage, and so on.
So your premise is extremely dependent on what DB you use.
Is ETL considered a DBA task?
Do you think the overlapping speaks for the decrement of either profession specialization?
In other words, efficiency and performance both depend heavily on the machine your DB is running on.
As such, a good DBA needs to be able to do sysadmin tasks. The business won't care that it was the sysadmin's fault for not realizing that a battery had gone dead on the raid controller, and a DBA shouldn't care either. Their purvue is the database, and everything that it entails.
I was in a hurry when I wrote my earlier reply - system monitoring is not the only monitoring we perform. We watch the database itself for health and performance. We watch for abnormally long queries, abnormal locking, excessive deadlock resolution, replication health, and exploits.
Backups themselves require a bit of specialization - there are at least four different methods I'm aware of off the top of my head, and the proper choice for which to use is going to depend on the business (we typically set up 3 of the four - logical dumps, rolling binlogs, and binary dumps).
HA plans, particularly live failover, requires a deep knowledge of the database, its replication solution, and your business needs. Safely failing over a live MySQL database in a way that results in no data loss is hard to do.
Quite a bit of specialized knowledge is required to work with databases, even for normal sysadmin tasks.
Having your data on a different server as your website introduces, effectively, a much higher minimum latency. It adds 5-6 ms to every single page load. If I were writing the site that would be a factor 5 drop at least. This is extra time that your webserver has to keep state for a request, and so effectively can mean a drop of a factor 5 or more.
The DBA said then that this would undermine consistency. And, of course, he's right. However for most things websites don't actually need consistency (plus the web request response model makes it impossible, as the data client-side is not included in transactions). The prices at checkout, sure, there you want consistency. Everything else ... do you really care if it takes 1-2 minutes for your webfarm to be entirely in sync ? (and if it's important enough you can implement "clear this cache item now" requests too, which I did for user records after a complaint)
Databases are so ridiculously slow it's not funny. If your data fits in 64-bit address space, an mmap'ed serialized data structure will blow it's socks off, and for everything that doesn't absolutely require synced data it's fast. If your critical data is small enough, keeping it in memory with a background thread getting signaled to dump it to disk after every update may also work.
I wrote a network-as-a-service infrastructure for an isp once (something to do with data on cell phones), where each machine would happily do 1Gbit of web traffic, directed by it's 5Gb database (meaning it would need to look up user account balance, potentially update it, for every request), consistently and quickly (99.9% of requests was < 1 ms). Mmapped protos FTW !
Databases are a language + a B+tree data structure for those who can't, or won't implement it themselves. Generalized databases have zero hope of competing with specialized ones.
I agree with you on above statement. Though my conclusion is different from yours. I think in the future the line among DBA/SysAdmin/Developers will become even more blurred , developers will be trained/required to take over more and more work from DBA and sysadmin (DevOps anyone?); consequently, the demand for dedicated roles such as DBA and sysadmins will diminish. Hope I am wrong though.
Given that background I still think that the topic of databases is just too deep for a generalist. I know a lot about the MySQL database (and a little about PostgreSQL), enough to write failover software, automate deployments, write guardian crons which slap down problematic queries & pre-emptively, automate backups, do vip failover and haproxy configuration... and I still have to go to my boss for most of the hard questions.
His knowledge encapsulates 12 years of working with and around MySQL, and it's proven invaluable to our customers. Knowing when to force certain optimizations, how to make subqueries run O(1) vs O(n), how to rebuild a complete database from binary logs, how to configure MySQL to work with SSD caches... these problems don't come up often, but when they do, not having a DBA available to you means contracting out to one at exorbitant rates.
It's the difference between a few minutes of downtime when the proverbial dung hits the fan, versus a few hours or days.
Start with the book High Performance MySQL. [1]
Follow up with the whitepaper "Causes of Downtime". [2]
Then find a copy of the IMDB dataset, put that in a database, and write an app against it. Make that app perform well, then simulate load against the app (pretend it hit the top of Reddit and Hacker News simultaneously), and keep it performing well.
After that, it's a matter of practical practice.
[1] http://www.amazon.com/High-Performance-MySQL-Optimization-Re...
[2] http://www.percona.com/redir/files/white-papers/causes-of-do...
> These days, the technology decision maker is the dude with Sublime Text open and a cloud control panel up in Chrome.
And when he is successful and gets clients, and a few thousand rows in his database, he realizes that he needs someone to keep that database alive. He needs someone to figure out how to make the cartesian product queries he's written into efficient queries.
At first, he hires a consultant for a few one-off gigs. However, then he's paying someone $200[1] an hour, typically with 8-16 hour engagements. After getting sick of that cost, and still lacking any kind of long term caring about his product, he comes to our team, and hires us to be his DBA, albeit remotely.
Business as a DBA is booming. Nobody thinks they need a DBA, but the reality is that you really can't afford to not have a DBA. We have customers coming on board with no backups, no high availability plans, no disaster recovery plans, queries that are performing cartesian products (and thus taking minutes against very small datasets), and no monitoring. (And yes, a good portion of users come to us while using the "solution" proposed by the OP (like AWS RDS), for many the same problems.)
We set them up with comprehensive backups, automated failover solutions, and 24x7 monitoring. Suddenly, their DB is no longer the primary source of downtime. They're no longer loosing customer engagement because their frontend takes seconds to render. They're no longer in the position of loosing their entire company because some junior developer accidentally dropped their users table in production.
In short, DBAs are a required part of your business, if you're using a database. You just haven't been burned bad enough by a poor database setup to realize it.
[1] Actual hourly rates for a planned engagement. Emergency rates are closer to $450 an hour. Why so much? You can't get a DBA from a college, from a technical school, or from any other form of formal education. Most DBAs these days are grown internally from developers or system administrators who decide to (or are forced to) specialize while on the job. There are single-digit thousands of us world wide, and we're in high demand.