News for hackers. Stuff that matters.
Re:Ask HN: What is new in Algorithms / Data Structures these... by yes_man on Wednesday May 10, 2023 @05:03PM (#35893472)
Dumb example but still example from practical world. Your body (assuming you are a human) has trillions of cells. Each cell way more complicated than what a 1000-dimensional vector can represent, but maybe it could be a loose estimate of some properties in each cell. Now the algorithm could be about finding the most similar cell. Could be useful for finding e.g. other cancer cells based on properties in one cancer cell.

Not that this is a practical example because we technically cannot index all cells in each body. But even if such an algorithm being studied today might be useful one day when we do capability to collect such data

Re:Ask HN: What is new in Algorithms / Data Structures these... by marginalia_nu on Wednesday May 10, 2023 @05:06PM (#35893519)
Where are you going to store this data? It's dozens of petabytes.
Re:Ask HN: What is new in Algorithms / Data Structures these... by yes_man on Wednesday May 10, 2023 @05:15PM (#35893630)
Should theoretical research of data structures and algorithms have been capped at 1GB in 1980 because that was the biggest single hard drive available back then and you couldn’t store for example a 2GB dataset on a disk?
Re:Ask HN: What is new in Algorithms / Data Structures these... by marginalia_nu on Wednesday May 10, 2023 @05:38PM (#35893876)
Not at all, I'll still call out fantastic claims when I see them though.
Re:Ask HN: What is new in Algorithms / Data Structures these... by karpierz on Wednesday May 10, 2023 @06:07PM (#35894137)
Google has definitely indexed over a trillion pages.
Re:Ask HN: What is new in Algorithms / Data Structures these... by marginalia_nu on Wednesday May 10, 2023 @06:17PM (#35894244)
Do you have any sources for this claim?

As far as I am aware Google doesn't publish any statistics about the size of its index, which no doubt varies.

Re:Ask HN: What is new in Algorithms / Data Structures these... by groby_b on Wednesday May 10, 2023 @07:34PM (#35895067)
Re:Ask HN: What is new in Algorithms / Data Structures these... by marginalia_nu on Thursday May 11, 2023 @02:46AM (#35898611)
Well what do you know, they contradict the claim made above.
Re:Ask HN: What is new in Algorithms / Data Structures these... by karpierz on Thursday May 11, 2023 @11:12AM (#35903338)
Sorry, they've crawled trillions of pages, and narrowed it down to an index of 100s of billions. Conveniently, the link answers your question of "can you have PB sized indices?" to which we can clearly say, yes.
Re:Ask HN: What is new in Algorithms / Data Structures these... by sophacles on Wednesday May 10, 2023 @06:45PM (#35894489)
It's only a few racks worth of disk servers.

If I was building it from my 5 minutes of googling, using 15TB nvme u2 drives, and easily available server chasis, I can get 24 drives per 2u of a rack. That's 360 TB + a couple server nodes. So ~6u per PB. A full height rack is 42u, so 6-7PB per rack once you take up some of the space with networking, etc. So dozens is doable in a short datacenter row.

Realistically you could fit a lot more storage per U, depending on how much compute you need per unit of data. The example above assumes all the disks are at the front of the server only, if you mount them internally also, you can fit a lot more. (see Backblaze's storage pods for how they did it with spinning disks).

Dozens of PB is not that much data in 2023.

Re:Ask HN: What is new in Algorithms / Data Structures these... by SoftTalker on Wednesday May 10, 2023 @10:45PM (#35896692)
> Dozens of PB is not that much data in 2023.

Yes it is. Just transferring it at data center speeds will take days if not weeks.

Re:Ask HN: What is new in Algorithms / Data Structures these... by rsrsrs86 on Thursday May 11, 2023 @01:02PM (#35904981)
A truck has more bandwidth than network adapters technically
Re:Ask HN: What is new in Algorithms / Data Structures these... by marginalia_nu on Thursday May 11, 2023 @02:49AM (#35898632)
This is still like tens of thousands of dollars of equipment to store information about a single person's biology.
Re:Ask HN: What is new in Algorithms / Data Structures these... by sophacles on Thursday May 11, 2023 @03:20PM (#35907067)
Probably an order of magnitude or two more. Still something that is feasable in a research context - early MRI and genome sequencing had similar "too much data" problems like this, but the researchers still built it out to learn stuff. Tech marched forward and these days no one really blinks about it. I presume that if such a "all the cells scanner" was invented today, it would only be used for research for a long time - and that by the time it became widespread data storage will have caught up.
Re:Ask HN: What is new in Algorithms / Data Structures these... by tourist2d on Wednesday May 10, 2023 @06:57PM (#35894606)
Where do you think computers store data?