komon

Can't you just run it under Rosetta?

Effective system design comes from understanding the constraints of the problem, understanding the constraints of technologies you could use to solve the problem, and understanding what parts of your solution needs to change as those constraints change.

For instance, a web crawler. Establish your boundaries right away. You can't crawl the entire web right out the gate, so you need to decide where to stop. 1M pages? 1B? Does your crawler run continuously, or in batches, on a schedule? Are you concerned with avoiding crawling duplicate links? How often do you want to revisit links you've seen before? Are you going to limit page sizes you'll keep? Are you keeping images and other files, or only HTML content? Do you need to be able to execute JavaScript? How do you want to access the results: search, machine learning, analytics, etc? How many requests for those results do you expect in a given time frame? What's the maximum time you'd like to wait for a response?

All of these questions will affect your system design.

Take those answers and do some napkin math to figure out the scale you're working with.

If you are crawling only 1M pages and keeping only the HTML of pages 50kB in size you can easily use any database off the shelf with little tuning, for pretty much any use case (modulo response time requirements). For allowing larger files and images you'll probably want to reach for commercial object storage (unless you're a whizzkid file server guy), and if you're crawling a billion links you'll likely need to think carefully about horizontal scale and partitioning, and if you need to provide search or analytical processing you'll likely want to bring in specialized data stores, which means setting up some pipes to connect them through replication or batch processing...

It goes on and on like that. You need a broad knowledge of the capabilities and limitations of different softwares across various categories to understand what your options are and when the complication of adding them would be warranted. I find it helps to understand the historical context under which they were first developed, so that you can understand the problem they were trying to solve.

In general, I've had the misfortune of not being the dumbest guy in the room at any given time, so I've not received much in the way of mentorship. I've had to mostly learn by doing, which gives me a particular bias towards building over buying and avoiding bringing in new software dependencies where I can help it.

I encourage you to learn by doing as well, even if you stop around the first or second roadblock, or about the time you start needing to spend your own money to go further. Maybe try implementing things locally in a way that doesn't let you do more from your personal machine per se, but instructs you in the complication of setting up e.g. kafka or elastic search, and their benefits and tradeoffs.

Think hard about how you would monitor the system to make sure it's working well, think about the dashboard you'd want to see to assure you of the health of the system. Think about where the bottlenecks to performance would be. Think about what you'd change to ensure it stays up 99.999% of the time, even when you're deploying changes to it or scaling up the database. Think about what you'd need to do to ensure that you never lose data.

And then read other people's experiences, but remember to take a grain of salt with you. People like to talk a big game, you'll hear people say that tool XYZ is the ONLY choice for a serious project and... that may or may not be true. Depends on the scale.

But usually in system design interviews people want you to whip out all the keywords and show you know why you're using them and what they're for. So when prepping for those, don't be afraid to say "streaming CDC events with debezium and kafka" instead of "batch job loading new records from postgres" or w/e, as long as it's applicable and you can justify it.

System design interviews also tend to revolve around a few categories of problems (web crawling/search, Google docs, building social media feeds, etc etc) so reading other people's answers for those will start giving you the flavor.

I give the system design interviews at my company (well, now I mostly train other people how to do them) so usually I look for people to ask this constraint questions, show breadth of knowledge, reveal where they're strongest by going deep on a few topics, and the more senior they are the more I want them to lead the conversation and ask me questions rather than the other way around. I don't really expect people to come in and deliver a rote-memorized google-scale answer off the top of their head, and tbh I'd be really suspicious if they did.

I think you need FDA clearance to be able to market a test as actually being effective for treatment/prevention etc. For exactly the same reason woo-woo supplements must disclaim that they aren't intended for that purpose

These are mutually exclusive states, actually.

The "chance at making FU money" and "100% more work/stress" and "losing funding or... goes bust and can't make payroll", that's stuff that happens at startups that are paying _below_ the VC-backed market rate.

Once a company gets money to be paying employees in the third hump of the trimodal distribution of SWE pay, those "exciting" parts are long gone. You're no longer in the running to receive a hilarious amount of money should everything 10-100x, and you're no longer worried about going bust and not making payroll.

Even a pre-series A startup I worked at saw the writing on the wall with 5 months of runway left and switched from trying to get funded to finding an M&A deal. At the very least you can get the team acquired by someone in the expansion phase and get the rare chance to interview as a group and compare notes and have an exec try and negotiate your salary _up_ for once.

By the time a startup is paying VC-backed market rate, the level of stress you have just comes from the usual suspects: management, culture toxicity, etc. some of that you can absorb, some might prompt you to find a new job, just like normal.

There's still a level of "hoo-rah, we're a startup" present of course, but I just consider this a bad attempt to keep the magic of when the company and products were small alive. I let it affect me none.

$JOB-2 got sued over website accessibility. The way these cases go is someone uses the WAVE browser extension to identify accessibility issues, and anything yellow or red becomes part of the cause of action. IIRC we settled out of court.

We were pretty large, big-name online merchandise retailer. Definitely put our heels to the fire on giving the frontend a design refresh.

A reference to Postel's Law: be conservative in what you produce and liberal in what you accept.

The law references that you should strive to follow all standards in your own output, but you should make a best effort to accept content that may break a standard.

This is useful in the context of open standards and evolving ecosystems since it allows peers speaking different versions of a protocol to continue to communicate.

The assertion being made here is that the world has become too fraught with exploiting this attitude for it to continue being a useful rule

> Is there some reason a cryptographic algorithm developer must track the latest release of a compiler?

Tracking the latest release is important because:

1. Distributions build (most? all?) libraries from source, using compilers and flags the algorithm authors can't control

2. Today's latest release is the base of tomorrow's LTS.

If the people who know most about these algorithms aren't tracking the latest compiler releases, then who else would be qualified to detect these issues before a compiler version bearing a problematic optimization is used for the next release of Debian or RHEL?

> Logically, therefore, must we not also expect CPU designers to also forego changes that could alter timing behavior?

Maybe? [1]

> freezing all compiler development

There are many, many interesting areas of compiler development beyond incremental application of increasingly niche optimizations.

For instance, greater ability to demarcate code that is intended to be constant time. Or test suites that can detect when optimizations pose a threat to certain algorithms or implementations. Or optimizing the performance of the compiler itself.

Overall I agree with you somewhat. All engineers must constantly rail against entropy, and we are doomed to fail. But DJB is probably correct that a well-reasoned rant aimed at the community that both most desires and most produces the problematic optimizations has a better chance at changing the tide of opinion and shifting the rate at which all must diminish than yelling at chipmakers or the laws of thermodynamics.

[1]https://en.m.wikipedia.org/wiki/Spectre_(security_vulnerabil...

I enjoyed Schild's Ladder and Permutation City immensely

The output from an LLVM frontend compiler is LLVM IR, which is sort of a high-level architecture-agnostic assembly.

The LLVM backend compiles this IR to the target architecture's machine code.

Some optimizations can (and do?) happen in the backend compiler, but for the most part it is up to the frontend to emit IR that is well optimized within the semantics of the frontend language.

The history of Rust, for example, is filled with examples of optimizations made not to the standard library, but to how the frontend emits IR.

Beyond that, different languages and ecosystems are going to have differing levels of optimization in their stdlib and runtime that will have an outsize effect on day-to-day performance.

I wonder what your actual cache hit rate is? Sounds like it could use some tuning since 100-150ms sounds consistent with most requests hitting the origin?

Well, due process is a right co-equal to free speech, so which rights override which others in which circumstances will come down to legal precedent.

My understanding is that the FBI or other non-judicial body cannot unilaterally issue a gag order. Subpoenas and gag orders related to them are granted by judges.

(Which isn't to say that the relationship between the judicial branch and law enforcement bodies is always pure and equal)

KV storage is SQLite, and SQLite replication and syncing using an out-of-process daemon is becoming a solved problem.

For that application at least no changes to the runtime would be necessary

Here's two models:

"Open Core", where the core product is open source and you provide closed source extensions, generally high-value integrations and enteprise-y features like SSO, active directory support, etc.

"Paid Services" where you have offerings on top of the open product. You might offer trainings, consultations, support contracts, custom development work (e.g. aforementioned high-value integrations). Also very common is to offer a fully-managed SaaS experience, but I wouldn't expect that to be your key to success; many open-core businesses fail expecting that full hosting to be more attractive than it is.

It's becoming more popular to offer "on-prem"/"self-hosted" support. Cloud costs are going to get cut after 3rd party products among your customers, so getting your product counted in column B gives you a little edge. Even if they need to cut their support contract down they're still using your product and are more likely to come back when the storm is over.

What you're observing is not the rise and fall of best practice and good engineering. All you're seeing is the hype cycle.

Best practice and good engineering in web dev is the same as it is in any other software field: the best stack is the one you know cold.

Unfortunately, there are many popular stacks, and all of them have something to say about all the others. And 80% of learning materials for any given stack aren't from official sources, but are instead blog posts and videos and tweets from tech influencers. When they parrot those same criticisms, it lends the air of a grassroots shifting of the whole community.

The fact of the matter is that despite the huge levels of hype flux, large, popular, stable apps are still being made with technology "boring" tech.

Despite SPAs, rails and Django apps are still built rendering server side templates and sprinkling jQuery where necessary. Despite GraphQL, REST APIs are still the state of the art for most APIs. Despite kubernetes and docker, many apps are still deployed to heroku or VPS providers.

It can be intimidating for newcomers because it can be difficult to separate hype from fact. My advice is to pick something, the more boring the better, and know it cold and ignore all claims of its death.

To address concerns about projects dying, that is pretty legitimate. Unfortunately webdevs deploy trusted code into a highly untrustworthy and antagonistic environment. New technologies do come to play and requires adapters and plugins to be written. I wouldn't worry about overall size of community as long as security issues are getting addressed.

DynamoDB is an interesting beast. It's hard for me to recommend it for most use cases anyway.

It was designed to power experiences that look like shopping carts: relatively short-lived, mostly append-only, needing a flattish performance curve as the number of items and users go up, with a really natural partition key.

Using it as the primary storage of your app means you play the part of query planner a lot of the time.

If your data is especially graph-structured you might end up doing so-called "single table design" where all objects end up in the same table that has its keys constructed carefully (compound keys formed by string concatting), And has secondary sparse indices to power other access patterns.

And definitely plan on overfetching as an _optimization_ to prevent many small queries to dynamo, especially in a GraphQL context where we have no idea what fields are being requested at fetch time.

FWIW, DynamoDB (at least, an API-co patible, sqlite-backed stand in) can be installed to developer machines.

But your point about on-prem stands

The Rifters Trilogy, also by Peter Watts is a great read as well. It focuses a bit more on the transhumanism aspects but non-conscious intelligence is a secondary theme of the last two books.

Pretty much any book by Greg Egan covers these ideas since nearly all of his stories take place somewhere in a shared universe at different points of humanity approaching the singularity. Check out his short stories in Axiomatic or the full length novels Permutation City and Schild's Ladder.

If a, b, c don't compel you, you may do fine with keeping an engineering journal or a personal wiki if all you need is a place to collect your knowledge.

However, to advocate for blogging I've got 2 points:

1. Refining your opinions -- related to becoming a better communicator, I think that forming a cogent opinion post requires you to really think through your thoughts, opinion, and experience on a topic. As a project it can be a great learning exercise, I find that I gain a much more solid understanding once I feel the need to back up my opinions with experience and references.

Which brings me to 2. Learning in public.

Taking the time to write up a good post and share it around exposes you to people who will agree with you, challenge your ideas, and importantly learn from your experience. It's a way to find a community and give back to it, and gain more in return.

Does 1 help you with c? Yes. Is 2 also networking, branding, maybe building a following? Sure, if you try at those parts.

Most the blogs that I want to read care a lot more about concentrating knowledge and experience so they can learn from and help grow others e.g. Julia Evans or Alexis King

As long as the service container's members are typed you should

"no notes" is an idiom meaning "I have no suggestions for improvement"

The author was being sincere, rather than sarcastic as your initial parse suggests

Honestly a cheap boom mic is going to be the optimum of ease-of-use and noise input isolation.

There are plenty of fairly cheap earbuds with built-in boom mics, of varying quality.

If folks really have an aversion to over-ear headphones I'd suggest that direction.

Only do this if you a) love chasing down bug reports of lambdas firing at 6 minutes rather than 5 or b) make it really, really clear in your marketing that the only guarantee is firing _after_ the given interval

Well, the best way to protect that information would be to not store it at all.

Whenever they select a candidate to vote for, you would place them into match groups bucketed by score (based on their first/second/third choice), maximum group size, and some other criteria like signup date.

You end up with buckets of IDs that are known to match to some degree, and you would know enough to be able to say, "you matched with Bill because he wants to vote for your third choice candidate" without needing any idea about who that might be.

By limiting the sizes of the buckets, you limit the impact of knowing any particular member of the buckets' affiliations should a malicious actor get their hands on the data.

There's plenty of other precautions to take, storing the bucket data encrypted at rest, secure backups, separating bucket data from any PII.

In defense of the first comment, when first approaching a body of code like this, it's not always going to be clear what returning None is going to mean in any given method.

Also, one thing I think doesn't get mentioned enough is the use of comments as anchors for text search. Having "disable limiting" expressed in English that way makes it easier for newcomers to explore the codebase.

An example of an output constituting a covered work might be that if you use a GPLv3-licensed library to output a timezone database shipped with the library in a JSON format, because the content of the JSON output is itself a derivative of a GPLv3 work, not only an output, it would be covered by the GPLv3.

In order for NN weights to be covered, the weights would need to be in themselves a derivative of a GPLv3-licensed work.

Training a NN on GPLv3 licensed training sets would (IMHO) make it a derivative work, that's what a lot of the Copilot uproar is about.

Programs compiled with GCC are not GPL by extension. Arguably the net weights are a kind of program themselves.

If you have a non-GPL license to use and redistribute the data you use to train the neural net, I would imagine not.

Depends on what you get out of programming, but here's some suggestions:

* Re-create one of your favorite websites or apps

* Create games, these can be as simple or as complicated as you like, and might be a re-creation of a classic (Tetris, Solitaire) or something from a relatively well-trodden genre (visual novels with Ren'py, idle games à la cookie clicker, text adventure games) or something all-new

* Corollary for the above, participate in game jams

* Contribute to open source projects

* Create interesting toys or demos (fractal rendering/zooming, "single-div" CSS demos, fun-looking pixel blotting with SDL or JS canvas, generative art)

* Try your hand at programming microcontrollers for automation, robotics, building keyboard or other peripherals

* Write scripts to automate away boring work, either at home or for your job. You might be surprised how much time/energy you can save by stringing together some python or building some IFTT/Zapier workflows. Automate the Boring Stuff with Python could be a good starting point here.

There are other ideas, but the most important thing is to pick something you find enjoyable, and don't be ashamed to drop a project when it ceases to be interesting.

Some of the most long-lived hobby projects are sustained by just one or two hour nightly sessions, or by weekend hacking.

That's only going to become a solution if you can convince people that the process preserves continuity of experience.

How do I know that I actually wake up on the other side, instead of dying and having a clone with my memories wake up instead?

You gotta sell everyone involved in the benefits.

Convince the devs that you're not just giving them make-work to do, get the execs to buy in on giving you the space to do what must be done.

Keep the pitch focused, you're not selling them on a rewrite after all, just on better practices.

A code review process means fewer defects and better ramp up on changes made across the code base.

A clean codebase is going to make future changes easier, faster, and less buggy. It's also going to help with engineer retention.

Execs don't really care about fiddly perfectionism or process or even technical debt. So speak their language.

Engineering can be done on short notice, but running up the pressure is only going to result in a broken product. If you can't get the execs to stop pressuring developers directly, it's a lost cause.

For the devs, they need automation and clear benefit. If you just tell them to start doing code review it writing tests, the starting friction is going to be too high.

Instead, give them working PR templates that reduce how much typing they'd have to do. Give them build scripts that test the code cleanly in one step. Plan some mob programming sessions and write some tests together.

Most importantly, you have to get everyome on the team in a (virtual or otherwise) room and have them choose the standards themselves. You can't mandate things like test coverage percentage targets or code formatting standards without the devs feeling like it was something that they all agreed on.

Then, make it your personal business to follow up with automation to back it up. Get some plug and play static analysis tools in there to enforce whatever test coverage minimums were agreed on. Build formatting and linting into a pre-commit hook so they never have to think about it. Take as much friction as you can it of it so that even the most pressured dev can see the value in writing some tests for the new feature.

You have to get them from agreeing to trying it these new processes to getting value out of those processes as quickly as possible.