News for hackers. Stuff that matters.
Re:Why is modern web development so complicated? by nostrademons on Wednesday August 07, 2019 @07:21PM (#20640261)
> "sure, that material might not be the most durable on the market, but unobtainium would have added $35 million to the project cost and ongoing maintenance costs would have gone up by another $5 million / year."

> That right there is a comparison that developers can never, ever offer anyone that's writing the checks for a project.

Sure they can. At Google there was an internal cheat sheet of "numbers every engineer should know" which included the cost in dollar terms of trading off engineer time vs. memory vs. CPU vs. network vs. user-perceived latency etc. (Actually there were two of them - the more famous one is the Jeff Dean version that traded off CPU cycles vs. L1-L3 cache misses vs. RAM latency vs. sending data over the network vs. compression vs. disk latency etc, measured in nanoseconds) If you had any moderately complex new system, you'd prototype a few different approaches on a small set of sampled data, do some multiplication to figure out how much it'll actually cost in production, then compare that against the engineer time needed for various optimizations. If you were in a revenue-generating area like Search, somebody (usually at the executive level) would also be calculating how much the project was likely to make or lose for the company and ensuring that's acceptable. Not all that unlike any traditional engineering field.

The thing is that the overhead for this process really only makes sense at Google scale. For the vast majority of new software projects, revenues on success >>> developer time >>> hardware cost. So you just assume that if you can get the job done in a way that satisfies the customer, it'll be wildly profitable, and the primary factor to optimize for is developer time. And if the developer can build it, the hardware costs needed to run it will be negligible, so just buy more hardware.

This is an artifact of the software industry essentially being a gold rush, where it's rapidly eating the rest of the world and replacing non-computerized processes that are millions of times slower. When you have 6 orders of magnitude to play with, you can play fast and loose with estimates, because any acceptable solution at all will make you a lot of money. When software has eaten the rest of the world and new systems are competing with existing computerized systems, then it's worth it to start paying attention to things like efficiency, security, TCO. There's some evidence we're approaching that point, but we're not there yet, and it's still more advisable for an entrepreneur to tackle an old-school non-computerized industry than to try and improve on existing software solutions.

Re:Why is modern web development so complicated? by TeMPOraL on Thursday August 08, 2019 @03:20AM (#20642494)
> At Google there was an internal cheat sheet of "numbers every engineer should know" which included the cost in dollar terms of trading off engineer time vs. memory vs. CPU vs. network vs. user-perceived latency etc.

Could someone alert Google's web developer teams (in particular the GMail frontend team) about that cheat sheet? If released products are considered as evidence, it doesn't seem like this cheat sheet is used.

Re:Why is modern web development so complicated? by nostrademons on Thursday August 08, 2019 @12:55PM (#20646206)
I left 5 years ago, but a friend of mine who stayed said that Google's basically given up on trying to hold the line on latency, and recent projects don't care. Which is kinda a damn shame - when we were Nooglers we fought for bytes - but I get the sense that the institutional knowledge about how much latency affects revenue left when Marissa, Amit, Craig, et al departed and Larry & Sergey moved upstairs to Alphabet. Sundar's background is Google Toolbar, Chrome, and Android, all of which are product areas where latency doesn't matter because it's installed desktop software.
Re:Why is modern web development so complicated? by aswanson on Wednesday August 07, 2019 @08:57PM (#20640856)
Ugh. You've basically made the case for me doing my next project in c++ 17 instead of using it as an excuse to learn Rust. Im morally opposed to c++....
Re:Why is modern web development so complicated? by iamwil on Wednesday August 07, 2019 @10:11PM (#20641205)
isn't protection against de-referencing null pointers a good reason?
Re:Why is modern web development so complicated? by MaxBarraclough on Monday August 12, 2019 @05:31PM (#20680248)
In C++'s defence, the language itself doesn't mandate any protections here (the famous 'undefined behaviour'), but there's no reason your compiler can't add automatic runtime checking against illegal use of null. GCC supports it with "-fsanitize=null" [0]

[0] https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.h...

Re:Why is modern web development so complicated? by aswanson on Thursday August 08, 2019 @07:17AM (#20643441)
Yeah, but I've been around the block enough times to avoid those types of bugs.
Re:Why is modern web development so complicated? by apta on Thursday August 08, 2019 @01:21PM (#20646460)
It's much more than just null pointer dereferencing. Time and time again, it's been proven that Rust protects even very experienced C++ programmers from bugs they didn't realize they're even making.
Re:Why is modern web development so complicated? by MaxBarraclough on Monday August 12, 2019 @05:40PM (#20680329)
This is a good point. Personal assurances count for little. No-one is above writing buggy code. Even C/C++ codebases written by very expert programmers, often have pointer-based bugs, especially if large and complex. Operating systems and drivers, for instance.

As well as having free-form pointers, C/C++ also have many, many dark corners.

What proportion of C++ programmers know that comparison = (c == c++); causes undefined behaviour?

Re:Why is modern web development so complicated? by thaumaturgy on Wednesday August 07, 2019 @08:29PM (#20640697)
It's awesome that Google did that. This is the first I've heard of it. But:

> The thing is that the overhead for this process really only makes sense at Google scale.

The software industry is far larger than Google. If we want to claim that software is a discipline of engineering, we all need to have access to that kind of reference data, rather than trying to feel our way around in the dark in comment threads about blog posts about somebody's personal experiences.

I won't go so far as to say there's no such thing as software engineering -- a number of people working for companies with large budgets have well earned that title, I'm sure -- but the extent to which the term is traded like currency in our industry is disrespectful to the discipline.

> So you just assume that if you can get the job done in a way that satisfies the customer, it'll be wildly profitable, and the primary factor to optimize for is developer time.

The fetishization of developer time is one of the problems slowing the maturation of our industry. It ignores end-user time, it ignores management time; it often focuses solely on the initial development time of a project and ignores entirely the ongoing maintenance time for a project.

How many codebases are there out there that are gigantic balls of mud? How much developer time has been wasted on trying to maintain or refactor those codebases? How many really dumb security issues have negatively impacted companies or end users? How many comments has tptacek left on one little forum in one musty corner of the internet over the years trying to explain cryptographic and security-related best practices to an infinite legion of under-trained developers?

How many people have lost their lives to software defects? Not even the really tricky CS stuff, I mean the kind of defects that any other engineering discipline uses as case studies in their introductory courses.

Which case study would you choose for our Galloping Gertie?

Certainly software has done an incalculable amount of good in the world, and its impact is far and away net positive.

But while software has been busy "eating the world", it has rarely paused to also consider the cost and damage it has wrought and whether there are practices that could be standardized to reduce those negative effects. So I disagree wholeheartedly that we're not yet at the point that these things are important. Depending on who you ask, computer programming is between 60 and 180 years old. That's well long enough for us to begin asking questions like, "what data do we have to support this approach or that approach?"

There is still plenty of room for experimentation and research. There is still plenty of room for hobbyists and tinkerers. This isn't an exclusive-or thing.

As much as I really do appreciate blog posts like this one, and the discussion they tend to generate, and especially that there's a new one to read every single day (he said, wryly), I can't help but see every one of them as another indicator of the immaturity of our profession.

Re:Why is modern web development so complicated? by james-mcelwain on Wednesday August 07, 2019 @09:49PM (#20641100)
I’d argue that the multiplicity of different programming languages and programming language paradigms suggests there likely isn't a teleological trajectory towards some single standard for programming.

Part of this is because many systems are evolving, because business needs have human inputs and are not always "logical." At the edges where this human interaction happens, we expand the domain of programming to that of human psychology. There definitely are empirical claims we can make human-computer interfaces, but it begins to get a bit messy.

To be clear, I am personally interested in becoming more rigorous about empirically justifying claims I make about the code I write. But I’m also interested in code because many of the problems it attempts to solve map the very complexity of the real world. In many ways, abstraction is a properly philosophical exercise, and I think it’s impossible to separate a pure engineering discipline from this broader context.

Programming is multi-disciplinary, and I think that’s a strength.

Re:Why is modern web development so complicated? by unimployed on Thursday August 08, 2019 @12:18AM (#20641815)
Physical constraints are more restrictive than information constraints. Physical constraints keep physical engineering from fragmenting into thousands of different but similar frameworks because the physical constraints rarely allow more than a few solutions to be competitive as best solutions. In information, the bounds/constraints are something more of a constructed popularity contest who the largest players are (Google, Amazon, Microsoft, Facebook, Apple, etc.) which carries as much or more weight than rigorously empirical comparisons.
Re:Why is modern web development so complicated? by kd0amg on Thursday August 08, 2019 @11:39AM (#20645519)
I’d argue that the multiplicity of different programming languages and programming language paradigms suggests there likely isn't a teleological trajectory towards some single standard for programming.

We also have a multiplicity of building materials, tools, fabrication processes, etc. What's the standard way to make a steel object? Do you cast, forge, stamp, mill, sinter, roll, weld? I would expect engineering as a discipline to have more solid arguments for choosing which (combination) of those processes to use for a particular task than what is typically thrown around in discussions about choosing a programming language or paradigm.

Re:Why is modern web development so complicated? by pingyong on Thursday August 08, 2019 @10:21AM (#20644836)
>The fetishization of developer time is one of the problems slowing the maturation of our industry. It ignores end-user time, it ignores management time; it often focuses solely on the initial development time of a project and ignores entirely the ongoing maintenance time for a project.

The problem is that in many projects, especially small web-dev or simple automation projects, you simply don't know if they're going to be profitable. The margins are huge, since the upfront development cost is almost constant, you might lose $50,000, or you might make $500,000, or anything in-between, but you just don't really know beforehand. So obviously in this situation you want to become at least somewhat profitable as fast as possible, and then think about "doing things right" later. Is this horrible practice? Sure. But it's also the most logical thing to do here, because "doing things right" at the start is simply a huge risk that only pays off if you're successful anyway. And people would rather minimize losses than maximize gains.

On the other hand more traditional engineering tasks tend to require big up-front investments anyway, and "failing is not an option". I mean it really kind of isn't, if you need more money and/or more time you'll get that much more often here because the initial investment was so big already. So doing things right from the start makes complete sense.

Re:Why is modern web development so complicated? by dhruvkar on Wednesday August 07, 2019 @07:42PM (#20640398)
>> internal cheat sheet of "numbers every engineer should know" which included the cost in dollar terms

Would love to see that. For companies determining what's worth what, this would be great to see, even just for the format.

Re:Why is modern web development so complicated? by nostrademons on Thursday August 08, 2019 @01:04PM (#20646288)
The format is IIRC something like:

  1 engineer-hour =
     W ms/search of latency
     X GB-year of RAM
     Y CPU-years of processing time
     Z TB-year of hard disk space
     etc.
The actual numbers are very confidential, I've forgotten them, and they're probably different from when I was there. Basically, everything is computed in dollar terms and normalized to engineer time. You assume your code will live for a certain lifetime (a year seemed reasonable when I was there), and you know how many queries your service currently gets (that's another number-you-should-know: usage data for the product you're working on). Multiply out the CPU usage increase by number of queries to get total CPU increase (you need this for capacity planning anyway), and you can compare that to the number of engineer-hours spent working on it to get cost effectiveness.
Re:Why is modern web development so complicated? by SquishyPanda23 on Wednesday August 07, 2019 @10:07PM (#20641183)
The Jeff Dean version is public, but does not include cost in dollars

https://youtu.be/modXC5IWTJI?t=3548