It takes years for a single person to get a project to the point where it's a good learning ground for scaling and maintenance.
Gluing a few libraries together is real software engineering but unless you're really invested in the outcome it's not that engaging and it's not that educational.
That's only true if complex systems don't interest you.
Personally, I have always found the experience of "putting the pieces together" and orchestrating highly diverse systems into a coherent whole to be much more educational than learning about algorithmic details. I also generally find making things work well more interesting than making things work.
Gluing a few libraries together will certainly produce a complex system.
Deciding when to optimize isn't too hard when nobody is ever going to use it (only optimize run time if something really takes too long, otherwise don't do anything tremendously stupid, but optimize for less development time because this is taking forever)
Readable code would be nice, because sometimes it takes a long time to get back to it, but I've also designed in a real reason to come back and retouch everything (x86 -> amd64; aarch64 as a stretch) and maybe I'll make things reasonable then... Although it's not my strongest skill.
Even most simple OSes become complex. Maybe the dependency chain isn't too long though.
Seems to tick all your boxes, IMHO.
CS I learned in school, engineering I learned while working.
I'm still not totally convinced that "software engineer" is even a thing, frankly.
It's certainly not people who are lost in details like string matching or how ACID is implemented in distributed databases.
Of course you need all these things in order for the search engine to work – but if you write them yourself, even if you think about them excessively, you will never finish.
Software engineering is knowing that you don't need to know everything, you only need to know where to find everything. "Programming" is an important part of it, but far from the only one.
A really good programmer who is free from the tyranny of software “engineering”.
It’s not all progress.
Literally by stitching together people you’ve listed here and making them work together:
> It's certainly not people who are lost in details like string matching or how ACID is implemented in distributed databases.
> Of course you need all these things in order for the search engine to work – but if you write them yourself, even if you think about them excessively, you will never finish.
Being able to write something from scratch is not the same as always writing from scratch. Let me turn this around. Who would you trust with writing search engine, a software engineer who worked on and wrote search engine from scratch or someone who only theoretically knows about it?
> Software engineering is knowing that you don't need to know everything, you only need to know where to find everything. "Programming" is an important part of it, but far from the only one.
I hear this excuse more and more when it comes to threads like this. You don’t need to defend your life position, it’s okay to not know everything.
i've met a good fraction of the first 128 google employees, and this could not be farther from the truth:
> It's certainly not people who are lost in details like string matching or how ACID is implemented in distributed databases.
the people who thought those things were unimportant details were the ones who tried to compete against google and failed, like lycos, inktomi, and pets.com
like, check out the wikipedia article on udi manber, who is best known for spending 15 years 'lost' in string matching:
> In 2002, he joined Amazon.com, where he became "chief algorithms officer" and a vice president. (...) In 2004, Google promoted sponsored listings for its own recruiting whenever someone searched for his name on Google's search engine. In 2006, he was hired by Google as one of their vice presidents of engineering. (...) In October 2010, he was responsible for all the search products at Google.
the fact that in https://news.ycombinator.com/item?id=38769506 you claimed that orchestra conductors don't need to play any instrument further calls your credibility into question
And sure, use of the word "engineer" will cause angst among some - because it used to have a precise meaning which has been diluted (and in the case of software, ignored.)
Indeed the list of projects runs the gamut of where a programmer might go in life. They might do user UI, or games, or work on an OS or compiler. More likely they'll work at some random company doing databases and reports. Or they'll be Web focused doing lots of JavaScript. Or they'll be deeply involved in the field of Advertising (Advertising Engineer anyone?)
Of course wherever you end up, you can still have some fun. And if you're just starting out then it can help you understand your desires, and limitations.
So don't worry too much about the "engineer" word. It's pretty irrelevant. Even the title of the thread is "programmer" not "engineer".
But these points raised by GP... are basically self-evident.
> - Deciding what to write yourself and what to take from a library
> - Identifying high-quality libraries and frameworks that meet your project needs
> - Deciding where optimization is worth the effort and where it is not
> - Writing code that will still be readable to you (and others) a few years from now
> - Thinking about the project as a large-scale, complex system with software and non-software dependencies
... But since they don't have immediate effect (along the tune of "look I made this shiny thing myself in a weekend!"), it's hard to brag about it and perhaps even hard to establish causality that your skills in the areas above contributed to the software/business' success. But you can always brag about how you implemented a Fibonacci heap or something and made your heap operations 0.2ms slower.
-
This will be a wonderful feature for copilots, to allow for the human to provide a verbal narrative of the logic/process or even the environment and of others in the room commenting on why a such-and-such is being done, and the copilot elegantly documenting the narrative.
Wait until we have "Thick Code" (where an app comes with an AI generated Mini-Documentary-Series on the creation of the piece (as an optional function of your enterprise Copilot Agent (With the [Security Assistant] [Compliance Assistant] [Legal Assistant] [Media Assistant] and whatever Alignment Agents are required and inform the narrative of the project.
EDIT: @Tao3300 - I do that all the time. You just need to be a better MeatPT and infer "like, my, opinion, man"
i've been at it for years and my current goal is figuring out how to cut paychecks without investing myself in that "cRaFt"
not hard to grift given that it's complete nonsense though. It's embarrassing who can be a CTO of a billion+ market cap company nowadays.
Almost all "programming" problems have been solved. Today's software is laughably inefficient not because we need better implementations of data structures, but because putting all those existing pieces together properly is a neglected discipline.
The best place to see this is in Machine Learning. The vast majority of ML/AI software that is published (which often implements revolutionary techniques!) is of incredibly poor quality from a software engineering standpoint. As a result, practical AI applications are orders of magnitude slower than they could be.
That's quite the claim. Why should anyone take your word for it?
If the "engineering" part of ML had kept up with the "science" part, I have no doubt that performance for typical use cases would be 100x-1000x higher than we are seeing today.
Even more of a claim than previous one.
YES!
But You learn that writing low level code, not high level.
Writing Text editor, or even a compiler, you don't spend much time on algorithm optimization. You spend it on designing good abstractions. You are given basic tools, and have to come up with good architecture on your own. And because you are forced to use more levels then with pre-made libraries (that create the structure for you), you will see your mistakes much faster.
But GP is completely correct that all of the fun, magical projects in the OP article are antithetical to the profession.
I guess I was emphasizing GP's point. The profession is the problem. Not the subject matter. I love computers.
Not all hackers are overly obsessed with the most efficient Big O and various low level (sorry pun intended) details.
A NAND gate is low level. Big O is just theory that anyone should know if he's worth something as a programmer.
The good news is that, while still valuable, you won't have to lose time creating a new broken wheel. The bad news is that this part of CompSci is now mostly fundamental research.
No it isn’t.
It’s a great thing to learn and understand, and essential for designing and maintaining data-intensive systems, but your statement simply isn’t true.
To me, it's essential.
It’s easy to make things that are accidentally quadratic but are fine until your big customer has 10 times the data and they suddenly aren’t fine anymore.
Doesn’t mean you need to optimise everything, but it doesn’t mean you shouldn’t think about this stuff.
This attitude is precisely why we have computers thousands of times more powerful than we had in the 90s, and yet they perform the same tasks slower.
>The idea here is to implement a well-defined game from start to finish without getting bogged down on the other fun stuff (e.g., game design and art). Also, it is best if you use a barebones 2D graphics library (e.g., SDL, SFML, PyGame), not a big game engine that'll hide all of the interesting bits from you.
This kind of game won't go on sale nor even be a portfolio piece for a non-junior engineer, but it's a good enough idea for someone like me who say, wanted to really wanted to solidify their Rust skills and show what my and the language's weaknesses are.
After I can get that done, I may feel confident enough to move on to work on a toy renderer based on what I figured out in the game. And then later contributing to a proper engine that already has a lot of groundwork figured out. I'd just be getting in the way if I jumped straight into trying to throw anything but the simplest PRs into Bevy.
Then again, big-O is useless in many cases because real computers have too many arbitrary performance thresholds.
I suspect software engineering is impossible, or at least, nobody has made the model required to do it.
Asymptotic complexity is about how things scale up. You should care about it when you're working with things that scale. Doesn't mean it's useless if your data is static, just means you need to understand when to go for the high scaling solution and when not to.
① you're writing a real-time control program such as a motor controller and you care about not just the asymptotic complexity of your algorithms but their worst-case execution time, in microseconds; or
② you should just do the computation by hand with pencil and paper instead of writing and debugging a program to do it for you
at the point that the input data becomes too big for ② to be an appealing option, n is at least 100, which means you probably care a lot about whether your program is O(n) or O(n⁴), at least if you wrote it in something slow like python
maybe the time to start worrying about it is after you start the program debugged and running for the first time
Speaking for myself, software engineering is something I do because I have bills to pay. Programming is something I started doing because it’s fun. The projects in this article are the types of projects I would do for fun if I had the time.
Same for me. While I strive to do stuff fast, in a way that is easy to maintain, modify and extend for work, see the big picture, take the right time of shortcuts, understand business and customers, there is not were the fun is.
I enjoy the time when I am alone with the PC, far from user stories and scrums.
I love to program, to convince the PC to do the trick I tell it to do. What I enjoy also is CS, algorithms, data structures, abstract thinking.
Sudden clarity.
I feel like printing it and putting it next to my workstation, so that I remember this both when doing something fun and when having to work so that my bills get paid.
Other points to internalize:
- in business, software is made to either earn money or pay less money. All other effects are at best secondary
- code is a liability as much (or more) as it is an asset
- programming is the easy 0-20% of the job of the software engineer, the rest is making sure the right thing gets programmed
- no matter what they tell you it’s a people problem
- software is always broken but it keeps running as long as practitioners are keeping it within the operational envelope
To me learning the engineering part is easiest. And also not terribly valuable if you don't know CS fundamentals.
It's not possible to be a "productive" musician on a traditional instrument (i.e. excluding iPads) without knowing how to play scales, chords, etc.
That’s demands for faster delivery of software and ridiculous demands on user experience.
Does looking down on higher level devs make you feel superior or something like that?
Absolutely not. It's about bloat. Many times people "just want to code" or eschew optimizing repeating the mantra that "premature optimization is the root of all evil" that you often end up with bloated software.
As someone who has played guitar on and off for years as a hobby, I'm shocked by how much harder it is than programming.
I can pick up almost any random codebase on GitHub, and as long as they use libraries and don't touch low level stuff, I can probably start working with it in 10 minutes to a day at most.
Then, I could leave that project for a year and be almost as productive as when I left. So much of the action is on the screen, not in the head, and there's no muscle memory required.
If any instrument was as easy as coding... I think a lot of coders might have music careers instead....
They do make idiot managers happy though, because look at all those features and scrum points they “finished!” That’s why armies of them will always be there in the software ecosystem, blithely and ignorantly a net drain on whatever unlucky company is currently employing them, busily making a mess for more diligent programmers to clean up for the rest of eternity. It’s called “job security.”
No, it's like recognizing that an orchestra conductor doesn't need to play every instrument (or even any instrument!) in order to make music.
Or alternatively, that a violin player doesn't need to understand the physics of acoustic dispersion in order to be the best in the world at playing the violin.
Also, I don’t think you’ll find a single conductor on this planet who is not skilled in at least one instrument. Terrible analogy.
We learn the high level practical behaviour of devices: resistors, capacitors, BJT and MOSFET transistors, opamps, logic gates, etc in 2nd year. What we learn there is enough to be an effective practical circuit designer.
We also dive deep into how that stuff works under the hood. One course I remember fondly started the first lecture with “This course is called semiconductor physics. To understand semiconductor physics you need to understand solid state physics. To understand solid state physics, you need to understand quantum mechanics. We have a lot of material to cover so let’s get started.” A similar course was Electric and Magnetic Fields, where we spent a ton of time applying vector calculus to situations of varying complexity to get a better understanding of how the things we learned in 2nd year actually work and some of the limitations that you wouldn’t pick up on from the practical (generally first-order) models.
You don’t get to graduate with an EE degree without learning this stuff. And in my mind, that stuff is like a mixture of data structures, runtime complexity, assembler, and cache coherency. Fundamental understanding that while you won’t use directly in your day-to-day work but underpins everything you do.
That’s all kind of missing the point though. This whole thread has been a discussion about whether Software Engineers need to get down into the nitty gritty low-level stuff. My take is absolutely yes. You might not use it directly every day, but you’ll make consistently better design decisions if you really feel it in your bones. Over on the electrical engineering side of the world you don’t get to graduate without that background understanding. In the software engineering world some people seem to take pride in their ignorance of the foundations they’re building in; in my 20-ish years of experience, those people often end up building things that work poorly and they don’t have the background understanding needed to fix it.
Ok, this is clearly a side-topic AND at the risk of being pedantic: Is this actually true?
Like, I can see how theoretically one could learn to sight-read music well enough to be able to direct an orchestra of individual musicians, then do enough ear-training to identify enough notes to keep tabs on everyone (especially if you're conducting a high school or (god help you :) ) middle school orchestra), etc, etc.
But does anyone actually do that? Has anyone ever done that?
"I'm going to learn how to conduct an orchestra without learning any instruments" kinda feels like "I'm going to become a software engineer using an LLM instead of learning any foundations)"
(To be clear - the LLM path might be viable in the future, possibly the near future, but at least today it's not quite there. My apologies in advance if my analogy doesn't work in the future :) )
A conductor should have a deep understanding of music, theory, and rehearsal pedagogy. At least in current western schools of music I don't see how you would explore these topics outside of the study of an instrument. Maybe there is some esoteric path to this; I just can't imagine how you go to Berklee and begin to explore the nuances of a composition without ever engaging with it as an instrumentalist.
On top of that, the conductor isn't just engaged at performances, they're also responsible for leading rehearsals. If they've never deliberate practiced the learning of music from an instrumentalist perspective I think they would be very hard-pressed to structure strategies for getting the larger group to a high level.
There is at least one (so I'm speculating he's not the only one, but it is likely to be rare) proficient conductor (Leopold Stokowski) who had no real proficiency with any instrument but he did have some very rudimentary piano training... and then pretty much taught himself conducting. Whether that rudimentary piano ability counts as "play an instrument"
"He studied at the Royal College of Music, where he first enrolled in 1896 at the age of thirteen, making him one of the youngest students to do so. In his later life in the United States, Stokowski would perform six of the nine symphonies composed by his fellow organ student Ralph Vaughan Williams. Stokowski sang in the choir of the St Marylebone Parish Church, and later he became the assistant organist to Sir Walford Davies at The Temple Church. By age 16, Stokowski was elected to membership of the Royal College of Organists. In 1900, he formed the choir of St. Mary's Church, Charing Cross Road, where he trained the choirboys and played the organ. In 1902, he was appointed the organist and choir director of St. James's Church, Piccadilly. He also attended The Queen's College, Oxford, where he earned a Bachelor of Music degree in 1903" [1]
That sounds like a bit more than rudimentary piano training to me :)
[1] https://en.wikipedia.org/wiki/Leopold_Stokowski#Early_life
i heard this before and it seemed dubious to me
on investigating, it turned out that there were literally zero famous orchestra conductors who don't know how to play any instruments. most of them know how to play numerous instruments and do so in their spare time, though there were one or two who had stopped. from my experience with musicians i think it's likely that they could play any conventional orchestra instrument after a short exploration period, even if they don't have documented histories of playing it
i suspect that this is true of non-famous orchestra conductors as well, but i could only find public information about the famous ones
of course that doesn't prove anything about programming, which is not the same skill as playing a musical instrument or conducting an orchestra, but it does show that you're constructing your arguments without much concern for veracity
Yea, no thanks.
If I'm going to work on something in my spare time, I am going to work on something that I am interested in and find fun. I generally loathe the dirty work often involved in software engineering and debugging all of these dependencies all the time. It's exhausting. Doing things from scratch puts me in charge.
And I mean, creating a search engine is the ultimate "not invented here" project. I'd just pay for and use Google, Bing, or whatever else API to do things for me. So it doesn't even make sense in the context of your objections.
Except that none of the established search engines actually does full-text search (anymore). There is no way to prevent those engines from breaking apart and rewriting your query as they see fit. No matter how many quotation marks you put around it, and even if you enable "Verbatim" mode or whatever.
So if you just implement the option to do that (which is actually the default if you leverage a standard FTS library), you already have something that AFAIK you can't get anywhere else. Then you can work on kicking the blogspam out of your index, and before long you will have a search engine of real value for yourself (and possibly others).
Or, you can build a crappy clone of a text editor from the 70s.
How much do you think hosting would cost?
Literally all of the listed projects, text editors, compilers, operating systems, and ray tracers, can exercise the exact same activities.
I'm more inclined to think that your comment is really more revealing about what kind of knowledge is in your own personal wheelhouse. Having worked on a ray tracer that was subsequently used on many feature films, I can assure you that all of your bullet points apply to that project, too.
It's much more a matter of whether you want to do something small scale and fun, or whether you want to suck all the joy out of it by applying the same soul crushing constraints we already get paid to do in our day jobs. Bleh.
In the linked article, these projects are all explicitly described as opportunities to learn about low-level stuff like how to efficiently store editable text. The difference with a web search engine is that nobody today can build such a thing completely from scratch, therefore it forces you to give up the toxic NIH mentality, which in my experience is usually driven by elitism and ego (on full display in this comment thread), not by some lofty desire to learn something new.
I'm sorry, but can you substantiate this claim? I've seen no indication that a search engine is not buildable from scratch at all.
The point is that it's too much work for a single person to build all the parts themselves. It's only feasible if you rely on pre-existing HTTPS libraries and database and text search technologies.
That's not a textbook algorithm.
The problem space is embarassingly parallel, so sharding is no problem. Although, realistically, you probably only need 1 server to cope with the load and storage needs. This isn't 2004. Servers are big and fast now, as long as you don't try to use cloud compute.
And of course all of this is based on "standard" algorithms. But it doesn't change the fact that implementing them all to build a functional performant web crawler+database+search is a project that is larger than any one person could build from scratch.
It would only be feasible to build on top of existing libraries, which is the whole point here.
I don't think you've made a single point that substantiates that claim and neither has anyone else, really. The person you're arguing with has built (as per their own estimation) more from scratch when it comes to the actual search engine core than anything else in their own project (https://marginalia.nu/).
Honestly, it seems more like the people arguing a search engine is somehow "off limits" from scratch are doing so because they imagine it's less feasible, probably because they simply don't know or are in general pretty bad at implementing things from scratch, period.
To a large degree there is also a wide difference in actual project scope. Remember that we are comparing this to implementing Space Invaders, one of the simplest games you can make, and one that is doable in an evening even if you write your own renderer.
To talk about "webscale" (a term that is so deliciously dissonant with most of the web world which runs at a scale several 100 times slower and less efficient than things should be) is like suddenly talking about implementing deferred shading for the Space Invaders game. Even if this was something you'd want to explore later, it's just not something you're doing right now because this is something you're using to learn as you go.
Well let's do the math!
1 billion bytes is 1 GB. The average text-size of a document is ~5 KB, that is the HTML, uncompressed. 1 billion documents is thus about 5 TB raw HTML. In practice the index would be smaller, say 1 TB, because you're not indexing the raw HTML, but a fixed width representation of the words therein, typically with some form of compression as well. (In general, the index is significantly smaller than the raw data unless you're doing something strange)
You could buy a thumbdrive that would a search index for 1 billion documents. A real server uses enterprise SSDs for this stuff, but even so, you don't even have to dip into storage server specs to outfit a single server with hundreds of terabytes of SSD storage.
> And of course all of this is based on "standard" algorithms. But it doesn't change the fact that implementing them all to build a functional performant web crawler+database+search is a project that is larger than any one person could build from scratch.
I'm still not seeing why this is impossible.
> It would only be feasible to build on top of existing libraries, which is the whole point here.
I don't think there are existing libraries for most of this stuff. Most off-the-shelf stuff in the search space is for indexing smaller corpuses, they generally aren't built to scale up to corpus sizes in the billions (e.g. using 32 bit keys etc.)
Add docids to sorted update lists. Zipper merge and arithmetic coding once lists get large, is the textbook baseline from back then.
Tack on a skip list, and you have fast intersections.
This is well trodden ground.
I do pull a lot of libraries, but an enormous amount of what the search engine does is very much built from scratch. The libraries generally deal with parsing common formats, compression, serialization, and various service glue like dependency injection. I think the number of explicit dependencies is a bit inflated by the choice to not use a framework like springboot, which pulls many of the same (or equivalent ones) implicitly.
What makes the search engine a search engine, the indexing software (all the way down to database primitives like btrees etc.), a large chunk of the language processing, and so forth; that's all bespoke. I think it needs to be. A lot of existing code just doesn't scale, or has too many customizations that would add unnecessary glue and complexity to my own code.
I'm going to echo SerenityOS Andreas and suggest that it's a skill like any other. If you shy away from building custom solutions to hard problems, you will never be good at it; and it will become a self-fulfilling prophecy that these NIH solutions are too hard to build.
At the same time, there's a time and a place and you should indeed be judicious as to when to roll your own solutions, but maybe that time and place is exactly in a hobby project like the ones suggested in this thread (and is how my search engine started out; a place to dick around with difficult problems).
I'd also add that being able to tackle problems yourself, rather than needing a library to do all the heavy lifting at all times, is a great enabler. Sometimes there is no adequate library, but that doesn't mean the conclusion has to be "welp, I guess we can't do that yet..."
You can still find search-engine-from-scratch courses on Udemy, complete with all the necessary algorithms [1].
[1]: https://www.udemy.com/course/build-a-search-engine-with-pyth...
"introduction online cs course that uses Python the build a search engine from scratch"
is udemy CS 101
Even through my smartphone autocorrect mistakes of 'introductory -> introduction' and 'the -> to'
You know those car guys who will rebuild their engine, just because? Or those retro computer guys that will recap an ancient board rather than buying a modern pc?
For you it is a job, for me it is a hobby. I have no interest in making something 'professional' I want to take it apart to understand how it works. Want to understand how a text editor works? Write one.
What component of that is ego?
It seems to me you're the elitist, you're not far off saying mere users shouldn't be allowed to modify their own software, shouldn't be allowed to install software that hasn't been okayed by the people who know what they're doing.
Your lack of constraints on personal exploration in software is interpreted as hubris by people who came to software seeking high paying regimented recipe-following.
Similar to compilers or tiny OSes, one can go as hardcore as necessary, or just stick to basic stuff.
Of all the typical personal challenge style projects, databases are probable the hardest to build, and even that is not impossible.
Does not have domain-specific dependencies.
So a code editor based on ACE or CodeMirror would not be from scratch, obviously, but one that involves writing all of the domain-specific logic would be. Using generic libraries doesn't stop something being from scratch. (In my case Tree-sitter is arguably domain-specific, but an early version did use a hand-coded JavaScript tokeniser in its place.)
Case in point, I've spent a year excising bloated frameworks from my stack at work and replacing the few corners we needed from those frameworks with, e.g. 50 lines of curl calls. The C compiles instantly and is tailored for our tiny use case, produced much quicker delivery on our one related feature we wanted, and removed chains of dependencies.
Being reliant on far-too-abstract libraries and frameworks to do simple jobs is also a curse. But nobody at work has the experience to know that curl was sitting right there on our image available for our use. And nobody has that experience because nobody took the time to build something from low level libraries. Now we know how and can make an intelligent decision without defaulting in either direction because we were afraid to try.
It's just faster, cleaner, and easier in a few cases to do precisely what you need right now, rather than anticipate a million things you might need and refactor your code to adopt a given "solution".
Having said that, there are libraries there are high value/cost. So I keep those. But the dirty reality is that most libraries are pretty bad.
Really? I think there are a couple out there. The main issue today is scale, but you could constrain that by limiting your crawler to a fixed set of sites.
Because when you write things from scratch, you actually have the space to innovate. But when building product with preexisting puzzles, there is much less space to actually make any usefull changes that would make your product actually standout from existing alternatives (which nih is all about)
That's my read as well.
Amen. And further, what better prepares a programmer to assess the relative costs of implementing a thing vs using a library providing that thing than having attempted an implementation?
Learning by doing is a valid approach, and this can even be called fun.
1) Much more often than not, the ready-made building blocks are crap. Your software will reflect that crappiness, and your life will consist not of writing software, but of maintaining and massaging crap.
2) Even if your ready-made building blocks are high-quality, they limit what kind of software you can write. Ideally, you imagine what you want your software to do, and then you write software to do it. You create the building blocks in that process. If you are only using ready-made building blocks, there is a lot of great software you just cannot write. And I think we all see that, because there isn't much great software around.
These problems might not be a concern for you. If you write software that is pretty much just more of the same, 2) does not apply to you. And while 1) is still problematic, it will be just manageable because you use these building blocks mostly on their happy path.
But I have bad news for you. This kind of software will soon be written not by you, but by an AI.
I claim that the opposite is true: The reason there are so few software projects that are really great is because high-level skills, not low-level ones, are in short supply. Every CS graduate can implement Paxos. The number of people with the experience needed to take an implementation of Paxos, and many other components, and put them together into something that works really well is much, much smaller.
> But I have bad news for you. This kind of software will soon be written not by you, but by an AI.
AI is actually much better at writing low-level code than high-level code. If you tell it to implement a piece table or a Fibonacci heap, it will do so without problems, in any language. If you tell it to create a simple CRUD application using existing libraries, it will output something that doesn't actually work.
I am actually currently writing a novel text editor, and I can guarantee you, my requirements mean that an AI cannot help me with that directly, because the underlying mechanisms I need don't exist yet. The AI is still helpful though when researching existing mechanisms, and to pinpoint why they don't work for me.
> AI is actually much better at writing low-level code than high-level code.
> If you tell it to create a simple CRUD application using existing libraries, it will output something that doesn't actually work.
If your low-level code is just a copy or adaptation of something existing, then yes, AI is really good at that. Something novel though, AI cannot help you much here.
On the other hand, simple CRUD applications will all be done by AI very soon. Probably not using existing libraries though, because these are crap.
But there is certainly a role for humans currently in high-level design, I agree with you here. But this high-level design crucially relies on your ability for low-level design, with the ability to imagine and then implement (possibly with the help of AI) the building blocks you need, not just reuse the ones that are there.
It seems to show up more in hobby stuff, or in cutting edge projects.
I'm always excited whenever I see any kind of algorithm work, because it's such a novelty. Most everything I do is just a UI around a few libraries.
If we can’t create special libraries that work well for common tasks then as a species we are too dumb.
Is all of this satire?
I have lived evidence that not nearly every CS graduate even knows about the topic that Paxos addresses.
I've got masters degree and I don't think anyone even mentioned word "paxos" or "raft" during my 5 years there.
Fortunely internet is a thing and I could read something about this topic.
I know graduates from various public schools and people who heard about paxos (let alone can implement it) are tiny %.
I guess if you’re always using libraries though, you may mistakenly be thinking that these libraries are just doing trivial work. Once you dive in and try to implement the low level stuff, then you realize how big of a disconnect there is between a fuzzy idea in your head and the lines of code that constitute that idea in reality.
Press X to doubt
But something you learn from writing a larger piece of software yourself is that you get to appreciate complexity and "bulk" (for lack of a better term) that you yourself created. You get to ask yourself questions like:
Is all this code needed? (Why is it 400 lines to do X?!) Why did this get so involved/ complicated?! How do I progress (eat the elefant)? Can I (teach myself to) make small iterative units of progress every day, for instance?
These kinds of katas are very very useful for a 5-7 year beginner (which I use as a label of respect - the longer we can remain beginners, the longer we learn with an open attitude!)
1. To be faif, these kinds of "hacking" projects aren't technically meant for learning software engineering in the context of a job, but they are really interesting and a treasure trove of new knowledge for anyone willing to put in the effort. I don't think it's a bad thing. A person can learn to be a good software developer and simultaneously have deep knowledge of the systems they are working with.
2. The goal of this project is not to research and implement some novel algorithm to replace existing works. The goal is to learn how things work. You are still going to use libraries, procedures and algorithms designed by other people and compile them into a cohesive system, it is merely the depth and complexity of the problem that makes it an interesting learning challenge. It's not about some absurd NIH philosophy, it's about curiosity.
3. Software development is not as straightforward as you make it seem. It's not just about best practices, management and finding an optimal way to combine existing libraries into your project. Sometimes you run into unforeseen issues, strange niches that your libraries may not account for. Sometimes you need to dive into the specifics of these libraries, understand at a lower level why they do not function for your use case. You might need to "hack" these libraries. Fork and customise them to truly fit in your project and produce the desired results. And the only thing that will help you then is deep knowledge and the ability to work with low level software.
Hmm, while I don't fully agree with his comment,
then your comment reminded me how I hate to discuss programming languages and programming ecosystems with C people.
It feels like C/low lvl people often refer to some "standard" that they treat as a bible of programming languages even when nobody mentions C and they treat GCC/CLANG as a perfect reference for compilers.
So, while high level devs dont understand low lvl stuff, then I believe low lvl programmers aren't creative at very abstract level, very abstract system modeling cuz they're too used to modeling everything with integers in C.
And they're too used to being hurt by C ecosystem that they believe it is "normal".
High level people (web developers) spend a lot of time arguing about their buzzwords like microservices, ddd, oop vs fp, etc.
But hey ultimately I just think people should satisfy their curiosities. My work is often always in Python, JS or Java, but at some point I got tired of just doing regular software development stuff and lately I've happened to gain interest in low level development. (Like any definitely sane person I chose to start with rust and decided my first ever rust project should be an emulator, so maybe I'm just dead inside and I don't know it yet)
If it is not not invented here, on the limit is not invented anywhere because nobody knows how to build these things, which we have been on that path more or less.
Perhaps you were visualizing that guy on your job who creates his own libraries and obscure systems within the system before telling anyone and that is generally a bad peer.
But the Universe is much bigger and diverse than that, people who programmed their own things but do a Google search first and use other software because they know it’s more feature complete, more known, trusted and better future supported are the best to work with.
We are all using the products of not invented here syndrome right now, don’t forget that.
My rule of thumb is that you should always understand at least one level of abstraction down.
This is a really important skill that I find difficult and frustrating. Does anyone have good advice or resources?
Think about it as a hobby. Most people have time for hobbies. The rest spend way too much time on social media or watching YouTube. Manage your time better.
Thank you for the advice which is no doubt well-intentioned.
But don't kid yourself. If you use your spare time for research and development to solve problems at work, then it is not really a hobby, it is just you working overtime for free. If that makes you happy, good for you, but it doesn't change the fact that time and resources are limited. The time is just cheaper in your case.
And yes it might also happen to benefit the job I do currently. But that is not the #1 priority.
I eventually learned to like solving lower level problems and I'm much better off for it. And these days I can tell people who haven't paid their dues because the only solutions they ever come up with are slow quadratic ones no matter the occasion.
No, building your own text editor, compiler, OS, ray tracer is not going to make you a worse engineer nor keep you from learning how to google and evaluate libraries or think critically, nor are those things reserved for an engineer who isn't building those things.
I get your point against NIH, but I don't see how doing such a project on your own in your freetime could possibly make you a worse programmer. If anything, you would likely learn exactly what it takes to go down the NIH path so strictly and be able to make better choices
This is the distinction between solving "local" problems and solving problems at scale.
But I think there is a transitional phase where requirements (gathering, documenting), unit testing, UI design (human or programmatical) are essential. Learning these as nuts-and-bolts in any language is essential to becoming skilled in the craft.
>might make you a better programmer, it won't make you a better software engineer
I think it is a short-sighted definition of SE but in any case, there are not any mention anywhere in the article claiming that this was aimed to software engineers.
Also, although I appreciate the concerns of "Not Invented Here" doctrine, we also have the other side of the spectrum, where we don't know how to do a left-pad anymore [1], and introduce dependencies everywhere.
The third point is that I don't think that having more knowledge on how things works contributes with the "Not Invented Here" mindset. I would argue is the opposite. When engineers are hungry on learning, they find excuses to make address that appetite at work, but once they know the tradeoffs and amount of effort to make it work, they will think twice before starting anything from scratch.
[1] https://qz.com/646467/how-one-programmer-broke-the-internet-...
You mostly learn by doing, true, but you learn by copying what the masters did. Software engineering education feels more like taking classes and writing code for exercises, whereas it probably should have more projects of the form "study this complicated-implementation-of-X and use those techniques in your implementation of Y", where x = google search, postgresql, whatevers and y is a complicated project [for one person or a small team] like your own single concurrent user search engine
I've held this idea for a really long time, in fact, I've not actually looked at comp sci education recently, so maybe this isn't true anymore, or maybe it never has been
If we are talking making more money at a job, I'd say that for most people I worked with and me included there are more important things to concentrate on. Like, soft skills, communicating clearly, understanding politics at work.
Myself, I'm okay with learning things you listed from more senior colleagues, on the job, and just having fun in my spare time.
Hackers like to obsess about Big-O, data structures, HoTT, and other high-theory stuff, yet the following skills, essential for software engineering, are almost never discussed and even more rarely practiced:
- Deciding what to write yourself and what to take from a library
- Identifying high-quality libraries and frameworks that meet your project needs
- Deciding where optimization is worth the effort and where it is not
- Writing code that will still be readable to you (and others) a few years from now
- Thinking about the project as a large-scale, complex system with software and non-software dependencies
In that spirit, I offer the following alternative challenge: Create a web search engine. Don't bother with string matching algorithms etc., others have already done that for you. "Just" make a search engine (and crawler) that can actually work, even if it only supports a subset of the web and a single concurrent user at the beginning.