News for hackers. Stuff that matters.
Re:Everyone is still terrible at creating software at scale by abiro on Saturday April 10, 2021 @05:35AM (#26760213)
Torvalds once said [1] that the only scalable software development methodology is open source, and I tend to agree for two reasons:

1. The project structure, tooling and documentation that lets new contributors jump in quickly makes software development easy to scale. In Coasian terms, the transaction costs around development are minimized.

2. It enforces hard interfaces and it clearly separates development from operations. Lack of discipline around these issues is a source of much accidental complexity.

[1] I can’t seem to find the quote, I read it in an interview a few years ago and stuck with me since.

Re:Everyone is still terrible at creating software at scale by rakoo on Saturday April 10, 2021 @08:16AM (#26760935)
To go further I think it's not exactly open source but remote-first that made software development scalable.

If you grossly simplify it to its innermost core, making development scalable means that if you have 10 times more people you can have 10 times more features/bugfixes/speed improvements/... The only way to do that is to make sure that an additional developer doesn't depend on other developers to work, and that can only happen if everything is properly documented, the build instructions are up-to-date, the processes are clear, basically anyone can start from scratch and get up to speed without the help of anyone else.

That kind of organization traditionally doesn't happen in on-site companies, where newcomers are followed by senior people, they have to follow some introduction course to familiarize themselves with the processes, they have to ask many questions every day, they need to be synchronized with other persons which brings some inefficiency beacuse everyone works at their own pace, etc... This all disappears when everything is properly documentend and every contributor can work in the middle of the night if they wish. I think the Gitlab Handbook goes over this quite well and describes a framework to implement that kind of organization but the rules are retrospectively obvious for people already used to open-source (https://about.gitlab.com/company/culture/all-remote/guide/):

- write down everything

- discussion should happen asynchronously. Any synchronous discussion (by text or call) should be only very small points. Whatever the type, write down the conclusions of that discussions

- Everything is public (to the organization), including decisions taken, issues, processes

Re:Everyone is still terrible at creating software at scale by marcosdumay on Saturday April 10, 2021 @10:39AM (#26761799)
Yeah, remote-first is important, but it's not only it.

Another very relevant factor is that people can just clone stuff, create new projects, and everything moves independently. So the open source development model has a pile of solutions for dependency management that team based development doesn't adopt.

But the one thing I don't get is why team based development doesn't adopt those solution. They are not expensive. Yet, even when I was able to dictate into everybody's requirements, I wasn't able to force teams into adopting them. Instead they insisted on synchronizing themselves by much more expensive and less reliable means. My guess is that most developers never dug any deep into open source and have no idea how it's done.

Re:Everyone is still terrible at creating software at scale by dragonwriter on Saturday April 10, 2021 @05:47AM (#26760268)
> the only scalable software development methodology is open source

“open source” isn’t a development methodology, or even a distinct set of methodologies.

> The project structure, tooling and documentation that lets new contributors jump in quickly makes software development easy to scale.

Plenty of open source projects don’t have that, nor is there anything restricting those things to open source projects.

It is true that some open source projects, because they see the value in new developers jumping in quickly, prioritize having structure, documentation, and tooling that supports that. It’s also true that some proprietary software projects do, too, because the project owner sees value in that.

Re:Everyone is still terrible at creating software at scale by abiro on Saturday April 10, 2021 @05:58AM (#26760317)
There are books written about open source development. Don’t confuse somebody dumping source files on Github with open source development.
Re:Everyone is still terrible at creating software at scale by dragonwriter on Saturday April 10, 2021 @10:43AM (#26761831)
> There are books written about open source development

There are books written about different authors idealizations of how to do open source development. The fact that many people have written about different methodologies for approaching a particular challenge doesn’t make the challenge a methodology.

And, yes, I am aware that the very approach known as “the bazaar” (from the essay “The Cathedral and the Bazaar”) is sometimes referred to erroneously as “open source development”, which is a particularly glaring error since all of the examples of both “cathedral” and “bazaar” development were open source projects.

Re:Everyone is still terrible at creating software at scale by watwut on Saturday April 10, 2021 @08:49AM (#26761096)
Yeah, and that does not mean those books describe how open source projects function in reality. But they should, how things are done changes.
Re:Everyone is still terrible at creating software at scale by oblio on Saturday April 10, 2021 @07:06AM (#26760597)
There are a gazillion ways to develop open source. It's more like an ethos than a methodology.
Re:Everyone is still terrible at creating software at scale by marcosdumay on Saturday April 10, 2021 @10:40AM (#26761807)
> There are a gazillion ways to develop open source.

Yet every large project behaves in a similar way, with only a few larger variations.

Re:Everyone is still terrible at creating software at scale by oblio on Saturday April 10, 2021 @11:11AM (#26762011)
That's why one of the seminal books about Open Source is the Cathedral and the Bazaar? Because "there is only one way to do things"? :-)
Re:Everyone is still terrible at creating software at scale by marcosdumay on Saturday April 10, 2021 @12:31PM (#26762694)
Hum... The Bazaar on that book is a very specific way to do things. So specific that I don't think anybody really follows it.
Re:Everyone is still terrible at creating software at scale by kasperni on Saturday April 10, 2021 @05:46AM (#26760261)
> the only scalable software development methodology is open source

Some companies such as Microsoft, Google, Amazon would disagree.

Re:Everyone is still terrible at creating software at scale by abiro on Saturday April 10, 2021 @05:54AM (#26760298)
The Linux kernel had 20k contributors since its inception, you’d be hard pressed to find those numbers for a single project at any company.
Re:Everyone is still terrible at creating software at scale by rcxdude on Saturday April 10, 2021 @07:16AM (#26760642)
Consider the bit at the end of this blog post:

https://devblogs.microsoft.com/oldnewthing/20180326-00/?p=98...

Bonus chatter: In a discussion of Windows source control, one person argued that git works great on large projects and gave an example of a large git repo: “For example, the linux kernel repository at roughly eight years old is 800–900MB in size, has about 45,000 files, and is considered to have heavy churn, with 400,000 commits.”

I found that adorable. You have 45,000 files. Yeah, call me when your repo starts to get big. The Windows repo has over three million files.

Four hundred thousand commits over eight years averages to around [130] commits per day. This is heavy churn? That’s so cute.

You know what we call a day with [130] commits? “Catastrophic network outage.”

Re:Everyone is still terrible at creating software at scale by jka on Saturday April 10, 2021 @07:35AM (#26760733)
To get closer to an apples-to-apples comparison, it'd be necessary to know whether the commit counts in each case include all development branches for all development groups. By design, git-based development can be highly distributed.

Also, even if we normalized both cases for 'code files only' and/or 'kernel code only', there could still be architectural, code style, and development process differences that lead to different metrics for each project.

Re:Everyone is still terrible at creating software at scale by AlexCoventry on Monday April 12, 2021 @11:09PM (#26787737)
He's really not selling Microsoft as a fun place to work, is he?
Re:Everyone is still terrible at creating software at scale by kasperni on Saturday April 10, 2021 @06:04AM (#26760342)
Yes, the linux kernel is a big codebase but so are a lot of others [1]. And you will find lots of private companies not mentioned that have long-standing projects with >10 mil LOC.

If you took the 100 biggest code bases in the world. I would be surprised if more than 10% of them were open source.

[1] https://www.informationisbeautiful.net/visualizations/millio...

Re:Everyone is still terrible at creating software at scale by oblio on Saturday April 10, 2021 @07:12AM (#26760619)
I know a medium sized French-American multinational. You've probably not even heard of it. A decade ago they had multiple products that had several million lines of code. Their entire platforms probably had 100 million. I can't even imagine how many contributors they had, the oldest project was started in the 80's and it probably had 1000 contributors over time, at least.

And again, that's for a middle size company you haven't even heard of. FAANG, Microsoft, Oracle, IBM for sure will have stuff dwarfing that.

Re:Everyone is still terrible at creating software at scale by abhijat on Saturday April 10, 2021 @08:00AM (#26760860)
Someone from oracle posted here a while ago, where they described Oracle 12.2 has 25 million lines of C https://news.ycombinator.com/item?id=18442941
Re:Everyone is still terrible at creating software at scale by marcosdumay on Saturday April 10, 2021 @10:43AM (#26761836)
And yet it's a less appealing product than Postgres.

An explosion on the number of lines of code is one of the way development teams fail.

Re:Everyone is still terrible at creating software at scale by _dps on Saturday April 10, 2021 @11:51AM (#26762361)
Is it actually less appealing? My understanding is that DBAs consider Oracle very good, just very (very) expensive. This also lines up with my experience tuning queries against Oracle vs Postgres backends. The folk wisdom seems, to me, to be Postgres is 99% as good in the common cases and 90-95% as good in the difficult cases.
Re:Everyone is still terrible at creating software at scale by marcosdumay on Saturday April 10, 2021 @12:28PM (#26762670)
Oracle DBAs consider Oracle clearly superior. SQL Server DBAs that know all the ways to optimize SQL Servers consider SQL Server clearly superior. I don't know of any PostgreSQL DBA, just generalists that work with Postgres, so I don't know about them.

The truth is that Oracle is ridden with coherence bugs, and have a much worse performance picture out of the box than Postgres. But while improving the Postgres performance requires deep digging into the DBMS itself, Oracle has a lot you can optimize just on the SQL interface. But there are plenty of cases where Oracle just can not become as fast as out of the box Postgres, and many where it could in theory, but a bug prevents you from doing things right.

Overall, three is no definitive ordering on the speed of the "good 3" DBMS. It always depends on what you are doing.

Re:Everyone is still terrible at creating software at scale by qznc on Sunday April 11, 2021 @08:37AM (#26769924)
SAP told me their framework is a billion LoC of Java and 30k tables in the database. That was years ago so it has probably grown further. It is only the framework, so no useful application yet.
Re:Everyone is still terrible at creating software at scale by kgeist on Saturday April 10, 2021 @06:14AM (#26760382)
How many of those 20k contributors worked on drivers, and how many - on the actual core (~150kloc)? Every driver is like a separate subproject, having 20k people work on hundreds/thousands of drivers (unrelated to each other) wouldn't sound as impressive.
Re:Everyone is still terrible at creating software at scale by thrower123 on Saturday April 10, 2021 @06:56AM (#26760549)
That's the scary thing. How many real, core contributors does even something like the Linux kernel have? People who have actually written more than a couple patches that landed and stayed in the kernel. I'd be astounded if it's much more than a hundred.

Most open-source projects have one to maybe five real contributors, outside the drive-by pull requests to fix some bug.

Re:Everyone is still terrible at creating software at scale by misterS on Saturday April 10, 2021 @07:04AM (#26760589)
I guess (and I have no way to verify this, it's really just a guess) that the number of developers that worked on the Windows code base in its 35 year history is in the same order, if not higher.
Re:Everyone is still terrible at creating software at scale by qznc on Saturday April 10, 2021 @05:57AM (#26760311)
Since Open Source software drops liability by definition, a huge complexity factor disappears.