menthe

First of all, what are you optimizing for? Expiry latency shouldn't matter, only keeping pace with insert throughput. And while I appreciate that you sidestepped the immediate issue through the solutions presented in the article, dropping the foreign key and adding an anti-join doesn't feel like a long-term solution, since you trade referential integrity for orphan risk.

There's a point in PostgreSQL scalability where the need to vacuum (following updates/deletes) becomes a problem in itself, and that point is a lot closer than most anticipate. For instance, vacuum can’t be unable to reclaim versions unexpectedly (e.g stuck transaction holding the TID horizon), leaving you with millions of dead tuples & massive stale index linked-lists, which can lead to major service degradations.

An alternative could be to tombstone rows with a HOT UPDATE on a `deleted_at` column. If that update is HOT (heap-only), the new row version lives on the same page and no new index entries are created. The old version does become a dead tuple, but it's pruned in-page and ad-hoc by ordinary reads and writes, so VACUUM is never needed to reclaim them, and dead tuples never accumulate, least of all in the indexes. On PG16+ this even works with a BRIN index on `deleted_at`, given some fillfactor headroom.

That also addresses the 20-minute scan, which seems to have been caused by walking millions of dead index entries at the oldest end of created_at, each needing a heap visit to confirm it was dead. With HOT tombstoning, those dead entries never get created in the first place. A persisted keyset cursor (the last processed `created_at`) would avoid re-walking that region either way.

The manual VACUUM also couples your business logic to the storage layer, which is a bit of a slippery slope that’ll cause issues when you scale to the next order of magnitude.

I understand why long-running workflows and per-class retention complicate partitions. But if most workflows are short-lived, partitioning plus a tombstone sweep for the long-lived stragglers could avoid most DELETEs, and would also help index sanity, caching.. and let you profit from the many other benefits of partitions.

Seems everyone on Hacker News has already forgotten about Anthropic’s amazing new watermarking algorithm. You don’t need to look any further to understand why it writes so awfully - it literally swaps words programmatically all over the text from the result of a hash function…

Unfortunately - the widespread propaganda seen around both nuclear power plants and datacenters is ultimately going to cost America everything - the American way of life our child’s will inherit is going to mean something very different than what it meant for us.

China controls 90-99% of the rare earths - which are necessary for everything from cars to defense, chips, energy and even construction steel;

China controls 70%+ of critical minerals (99% of galium/magnesium, 85% tungsten, etc, etc).

China owns the world’s manufacturing - with the associated skilled tradesmen (hard to produce anything fully locally - the experts are retired / passed already).

China has abundance of power and energy.

China has got both power, energy, and they’ve got the will.

China has got the best intelligence models today (as of this month) - which means scientific superiority, increased productivity, and military superiority.

What does America have, and what does that mean for our economy in the coming 20-40 years?

The reality is that we are in a new race to intelligence and re-industrialization, but we seem to be anti-everything at this point. Nobody wants to hear it, I know I am going to get downvoted to hell with this, but this is ultimately the new reality, and our eternal debating / 4-years cycles vs. life-long strategies in China, is going to cost America everything.

False.

As of PG16, HOT updates are tolerated against summarizing indexes, such as BRIN.

https://www.postgresql.org/docs/16/storage-hot.html

Besides, you probably don't want "done" jobs in the same table as pending or retriable jobs - as you scale up, you likely want to archive them as it provides various operational advantages, at no cost.

Nothing a well-trained model won't be able to instantly solve. It's literally just grunt work, not rocket science.

F DRMs though. Good news is those AAA games are rarely worth anyone's time anyways. Better spin up indies or classic games - a good SNES game is worth a hundred of those garbo AAA license rehashes.

Can we just flag this kind of misleading and propagandist content? It’s literally content+karma farming, and brainwashing hot garbage.

Total rage bait that doesn’t have its place on platforms catering for higher educated and science oriented people.

Nuclear plants evaporates water, water rains down a few kilometers away, gets collected in sewer, recycled/filtered and re-injected in the system.

You know what doesn’t get recycled? Coal & gas burning pollution & CO2. And you know what doesn't get re-injected? Lost lives to pollution and global warming.

Take this to daddy Elon - this has absolutely nothing to do with Rivian.

Meh, typical overblown “take” ihmo.

> All this stuff is just done automatically in Rust. If you have to write out the code by yourself, that's more work, and a huge chance for bugs. The compiler is doing some very complex tracking to figure out what to drop when and where. You'd have to do that all in your head correctly without its help.

What prevents anyone from dedicating a Zig memory allocator to the job (and all of its subtasks), and simply freeing the entire allocator at the end of the job? No baby-sitting needed.

Or if the mindset is really to be assisted, because “very complex” and too “much work”, may as well use a garbage collected language.

> It's knowing the compiler is on your side and taking care of all this that makes it magical.

Until you got used to it, and trusted it so much, and it suddenly misses something - either after a compiler update, or after some unsupported code introduced, and that shit takes down prod on a Friday. I’m not going to take the chance, thank you, I can call free() and valgrind.

I am sorry, are you non-sarcastically arguing that being able to pass through airport security, potentially accessing cockpits and planting bombs onboard airplanes, with a high-school level SQL injection on a federal website used by dozens of airlines & airlines employees, is actually, "fine"?

Besides, I am not sure what sort of "security through obscurity" you are talking about? Ian and Sam found it, and frankly - with a public page, page title + first h1 tag clearly stating that this relates to a Cockpit Access system, this has got to show up in a shit ton of security research search engines instantly.

Not 100% sure why it’s often idolized on HN.

We’ve maintained a financial exchange w/ margining for 8 years with it, and I guarantee you that everyone was more than relieved - customers and employees alike, once we were able to lift and shift the whole thing to Java.

The readability and scalability is abysmal as soon as you move on from a quant desk scenario (which everyone agrees, it is more than amazing at.. panda and dask frames all feel like kindergarten toys compared), the disaster recovery options are basically bound to having distributed storage - which are by the way “too slow” for any real KDB application given the whole KDB concept marries storage and compute in a single thread.. and use-cases of data historical data, such as mentioned in the article, become very quickly awful: one kdb process handles one request at once, so you end up having to deploy & maintain hundreds of RDB keeping the last hour in memory, HDBs with the actual historical data, pausing for hourly write downs of the data, mirroring trees replicating the data using IPC over TCP from the matching engine down to the RDBs/HDBs, recon jobs to verify that the data across all the hosts.. Not to mention that such a TCP-IPC distribution tree with single threaded applications means that any single replica stuck down the line (e.g. big query, or too slow to restart) will typically lead to a complete lockup - all the way to the matching engine - so then you need to start writing logic for circuit breakers to trip both the distribution & the querying (nothing out of the box). And then at some point you need to start implementing custom sharding mechanisms for both distribution & querying (nothing out of the box once again..!) across the hundreds of processes and dozens of servers (which has implications with the circuit breakers) because replicating the whole KDB dataset across dozens of servers (to scale the requests/sec you can factually serve in a reasonable timeframe) get absolutely batshit crazy expensive.

And this is the architecture as designed and recommended by the KX consultants that you end up having to hire to “scale” to service nothing but a few billions dollars in daily leveraged trades.

Everything we have is now in Java - all financial/mathematical logic ported over 1:1 with no changes in data schema (neither in house neither for customers), uses disruptors, convenient chronicle/aeron queues that we can replay anytime (recovery, certifying, troubleshooting, rollback, benchmarks, etc), and infinitely scalable and sharded s3/trino/scylladb for historical.. Performance is orders of magnitude up (despite the thousands of hours micro-optimizing the KDB stack + the millions in KX consultants - and without any Java optimizations really), incidents became essentially non-existent overnight, and the payroll + infra bills got also divided by a very meaningful factor :]

"without ever having a job"

First Google Result for "nofx retire":

> The punk group have all got normal jobs, which they will continue to pursue, but are pulling out all the stops for one final tour for their army of fans.

Seems in direct contradiction with this anarchist click bait title.

I don’t get it. If you do not want to participate in the preservation and distribution of the archive, why don’t you just move on instead of complaining?

Besides, gluetun+chihaya+qbit containers do the job without breaking a sweat, and without ever having to remember that you run a VPN - as it’d only be tunneling the containers of your choice. gluetun is the best image ever made!

Everyone’s talking about how you MUST hit the database for revocations / invalidations, and how it may defeat the purpose.

How is no one thinking of a mere pub-sub topic? Set the TTL on the topic to whatever your max JWT TTL is, make your applications subscribe to the beginning of the topic upon startup, problem solved.

You need to load up the certificates from configuration to verify the signatures anyways, it doesn’t cost any more to load up a Kafka consumer writing to a tiny map.

Thank you very much for having taken the time.. Your comments and function are both very helpful!

Frankly, who can read this!? I am not sure what's worse, the multi-line comments spanning multiple lines of code, having multiple instructions on a single line, or the apparent disconnect between the pseudo-code of the article.

The day SMB supports server-side thumbnail generation/caching, kindly let me know :]

It absolutely can, and it does not duplicate nor modify the medias. I mount my several TBs large library with the read-only flag in Docker.

https://immich.app/docs/features/libraries#external-librarie...

Absolutely was using the AIO image, with thumbnail generation enabled for every formats of my library (another thing you need to manually edit in Nextcloud’s configuration as by default the format list is limited).

And it’s only “pre-optimized” if you are cool with PHP memory limit crashes, PHP operation timeouts, PHP request size limits, and the works.

Another joy associated with using Nextcloud sync is that uploads don’t even seem to support multi-part resumable uploads. So not only is it crazy slow, if there’s any error during the auto-upload of a 2G video clip, or the app is temporally backgrounded by iOS, it’ll go into an exponential back off (which you can force start), and eventually just start the upload for that/those file(s) over from scratch - good ways to waste days burning in your screen while in a trip and trying to ensure your medias are backed up in case you lose your phone on a trip. Try uploading raw images & 4k clips shot on iPhone to Nextcloud using the Nextcloud app + the AIO image from abroad.

I’m telling you, I’ve tried to use them for quite some time, and I’m far from DevOps-illiterate - I’ve been using k8s since it’s infancy, we wrote the original Operators at CoreOS way back.

For having used Memories on Nextcloud, and having spent hours trying to micro optimize the Nginx & PHP configuration, I can safely say that, while it is better than the Nextcloud’s native Photos app, this is absolutely nowhere near to Immich, Filerun, or surprisingly even a dumb SMB share (which doesn’t have thumbnail caching…!). I’ve really tried hard, as Immich’s support for external libraries was still in a PR at that time, and didn’t want to have two separate tools to grab files and grab photos.

A big part of the problem, it seems, is that, when you have a large library, and you jump/scroll to a specific year or so, it won’t cancel the previous page(s) worth of thumbnails loading. So as soon as you’re scrolling to search for something, it quickly accumulates hundreds of useless requests that quickly overload the PHP workers, and make everything crawl to a standstill.

I personally had to give up. When trying to grab photos from abroad for my shortly upcoming proposal, I’ve literally deleted Nextcloud/Memories, plopped Immich in docker compose, let it index/transcode/generate thumbnails from scratch against my “external library” (so Immich doesn’t duplicate the medias), and that ended up savings me days of buffering, and was able to find the nice pictures for the occasion!

(R740xd with 48 cores and 96TB SSD-backed ZFS pool)

This is moot.

- Immich supports external libraries

- Use docker compose and never worry about versions breaking

You are making a major conflation here. While they do millions of users, they were last reported to only have ~60k tenants.

Decently sized EKS nodes can easily hold nearly 800 pods each (as documented), that'd make it 75 nodes. Each EKS cluster supports up to 13,500 nodes. Spread in a couple of regions to improve your customer experience, you're looking at 20 EKS nodes per cluster. This is a nothingburger.

Besides, it's far from being rocket science to co-locate tenant schemas on medium-sized pg instances, monitor tenant growth, and re-balance schemas as necessary. Tenants' contracts does not evolve overnight, and certainly does not grow orders of magnitude on week over week basis - a company using Figma either has 10 seats, 100 seats, 1000, or 10,000 seats. It's easy to plan ahead for.

Hell, OVH does tenant-based deployments of Ceph clusters, with collocated/coscheduled SSD/HDD hardware and does hot-spot resolution.

This, seriously. The long-term maintenance, tribal knowledge & risks associated with this giant hack will be greater than anything they'd ever have expected. Inb4 global outage post-mortem & key-man dependency salaries.

There's no virtually no excuse not spinning up a pg pod (or two) for each tenant - heck even a namespace with the whole stack.

Embed your 4-phases migrations directly in your releases / deployments, slap a py script to manage progressive rollouts, and you're done.

Discovery is automated, blast / loss radius is reduced to the smallest denominator, you can now monitor / pin / adjust the stack for each customer individually as necessary, sort the release ordering / schedule based on client criticality / sensitivity, you can now easily geolocate the deployment to the tenant's location, charge by resource usage, and much more.

And you can still query & roll-up all of your databases at once for analytics with Trino/DBT with nothing more but a yaml inventory.

No magic, no proprietary garbage.

In WolframAlpha (no Kago subscription, but I assume it works just the same), to achieve your desired result, you merely have to use:

> 14:50 - 16:28

Or

> 14:50 to 16:28

It makes sense when you think of it.

I’ve been using this for years, and it works wonders for datetimes too.

Kinda surprised anyone would still using WeaveNet as a CNI.. it was a bit of a dumpster fire: https://blog.quentin-machu.fr/2020/08/07/our-breakup-with-we...

More technical article about the new deployment: https://science.nrao.edu/facilities/vla/observing/cosmic-set...

So glad to see SETI alive. Next step, building a proof-of-work token to bring back SETI@Home and leverage thousands more GPUs :']

I hope you realize that this won’t even compile.

The err in your if is a newly allocated variable by :=, different than the err allocated by the x prototype, and you’d also return a nil error even if impl returned an error.

The Go compiler is smart enough to prevent you from doing this beginner mistake:

result parameter err not in scope at return. inner declaration of var err error.

https://go.dev/play/p/VdWVOL51wZf

You would have to first declare thing prior to the if statement, then replace := by = .. the beauty of adding verbosity and longer-lived variables for the sake of trying to write return instead of return err in Go.

if you are mixing work and personal networks, I am sorry to say but you are a liability and D&R + HR should be on your ass.

This includes heat, navigation, drive modes, etc. Rivian estimates fix may require physical intervention.

Great comment, and same. Running it on Proxmox also enables running the router in a high-availability fashion quite easily too.. which is pretty neat.

I was running OpenWRT on my Xiaomi AX9000 (based on master but also IPQ807 specific forks), but everything OpenWRT is software-based so 1/ the AX9000's acceleration chips are idle & performance is quite limited (unless using one of the crazy experimental NSS builds where mostly everything is broken) 2/ the integration with the onboard switch itself is far from adequate, e.g. VLANS+batman-adv are quite broken - very few configurations yield a partially working network (e.g. randomly dropping ARPs).. and don't count of LACP to work even though it can be configured. That's forgiving the fact that we also have to manually patch the BDF files on OpenWRT for the AX9000 if you want the radios to be connectable, as OpenWRT do not want to ship with the correct BDFs for some vague legal liability reasons.

Used VyOS in Proxmox for a while too, but at the end of the day, the firewall boilerplate is quite huge (and can't easily be edited on the go in the house as necessary), tailscale is not natively supported, etc. Relatively sad to let go of IaaC and committing the configuration though, but somewhat manageable with snapshots.