News for hackers. Stuff that matters.
Re:The Epstein Network by pickpuck on Monday November 17, 2025 @04:16PM (#45958412)
What if we extended this idea beyond one dataset to all discrete news events and entities: people, organizations, places.

Just like here you could get a timeline of key events, a graph of connected entities, links to original documents.

Newsrooms might already do this internally idk.

This code might work as a foundation. I love that it's RDF.

Re:The Epstein Network by j-pb on Monday November 17, 2025 @04:25PM (#45958506)
If it's RDF it won't work as the foundation.
Re:The Epstein Network by axus on Monday November 17, 2025 @04:38PM (#45958629)
One wonders what the US government agencies use.
Re:The Epstein Network by PaulHoule on Monday November 17, 2025 @04:44PM (#45958701)
Isn’t that what Palantir’s product is?
Re:The Epstein Network by sswaner on Monday November 17, 2025 @10:16PM (#45961008)
Pretty much, at least at the semantic layer. https://publish.obsidian.md/followtheidea/Content/AI/Ontolog...
Re:The Epstein Network by cjohnson318 on Monday November 17, 2025 @04:46PM (#45958720)
They probably use Excel, maybe Microsoft Access.
Re:The Epstein Network by ToucanLoucan on Monday November 17, 2025 @05:36PM (#45959203)
Microsoft Access form that connects via IIS to an Excel spreadsheet acting as a database. Also the server it's running on is sitting on a wooden table.
Re:The Epstein Network by cjohnson318 on Tuesday November 18, 2025 @11:04PM (#45975729)
Bro you can't just leak operational secrets on the world wide information highway like this.
Re:The Epstein Network by abnercoimbre on Monday November 17, 2025 @04:42PM (#45958683)
I think you meant one shudders. And yeah, Snowden made it clear there's orders of magnitude more data than this graph explorer for them to sift through.
Re:The Epstein Network by fancy_pantser on Monday November 17, 2025 @04:52PM (#45958779)
Software like i2 Analyst's Notebook.
Re:The Epstein Network by dboreham on Monday November 17, 2025 @05:34PM (#45959182)
Internet search engines have their origins in government projects fwiw. They had search engines before Alta Vista, used for searching data sets that pre-date the internet, and some of the people involved in those went to work on the original commercial search engines.
Re:The Epstein Network by arthurcolle on Monday November 17, 2025 @08:52PM (#45960509)
Probably not particularly useful but GCHQ & NSA both have neat graph related repos

UK: https://github.com/gchq/Gaffer

US: https://github.com/NationalSecurityAgency/lemongraph

Re:The Epstein Network by VikingCoder on Monday November 17, 2025 @05:31PM (#45959158)
Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale

Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus

Re:The Epstein Network by throwaway290 on Tuesday November 18, 2025 @06:37PM (#45973755)
...and of course it's in RDF!
Re:The Epstein Network by darth_aardvark on Wednesday November 19, 2025 @06:35PM (#45986818)
Palantir, arguably the closest thing to Torment Nexus Inc. IRL, literally builds a product that does this.
Re:The Epstein Network by FanaHOVA on Monday November 17, 2025 @05:44PM (#45959273)
One co trying: https://www.system.com
Re:The Epstein Network by jandrewrogers on Monday November 17, 2025 @05:50PM (#45959323)
This has been attempted many times. They all fail the same way.

These general data models start to become useful and interesting at around a trillion edges, give or take an order of magnitude. A mature graph model would be at least a few orders of magnitude larger, even if you aggressively curated what went into it. This is a simple consequence of the cardinality of the different kinds of entities that are included in most useful models.

No system described in open source can get anywhere close to even the base case of a trillion edges. They will suffer serious scaling and performance issues long before they get to that point. It is a famously non-trivial computer science problem and much of the serious R&D was not done in public historically.

This is why you only see toy or narrowly focused graph data models instead of a giant graph of All The Things. It would be cool to have something like this but that entails some hardcore deep tech R&D.

Re:The Epstein Network by stevage on Monday November 17, 2025 @08:05PM (#45960262)
>These general data models start to become useful and interesting at around a trillion edges

That is a wild claim. Perhaps for some very specific definition of "useful and interesting"? This dataset is already interesting (hard to say whether it's useful) at a much tinier scale.

Re:The Epstein Network by zozbot234 on Monday November 17, 2025 @08:12PM (#45960301)
This is not a "general purpose data model", though. A better example would be Wikidata which at about 100M nodes and 1B edges (so orders of magnitude less than that 1T claim) is already enabling plenty of useful queries about all sorts of publicly-available data and entities.
Re:The Epstein Network by jandrewrogers on Monday November 17, 2025 @08:51PM (#45960504)
It was a widely observed heuristic going back to the days when the Semantic Web was trendy. The underlying reason is also obvious once stated.

Almost every non-trivial graph data model about the world is a graph of human relationships in the population. If not directly then by proxy. Population scale human relationship graphs commonly pencil out at roughly 1T edges, a function of the population size. It is also typically the highest cardinality entity. Even the purpose isn’t a human relationship graph, they all tend to have one tacitly embedded with the scale implied.

If you restrict the set of human entities, you either end up with big holes in the graph or it is a graph that is not generally interesting (like one limited to company employees).

The OP was talking about generalizing this to a graph of people, places, events, and organizations, which always has this property.

It is similar to the phenomenon that a vast number of seemingly unrelated statistics are almost perfectly correlated with GDP.

Re:The Epstein Network by michelpp on Monday November 17, 2025 @07:19PM (#45960002)
There are open source projects moving toward this scale, the GraphBLAS for example uses an algebraic formulation over compressed sparse matrix representations for graphs that is designed to be portable across many architectures, including cuda. It would be nice if companies like nivida could get more behind our efforts, as our main bottleneck is development hardware access.

To plug my project, I've wrapped the SuiteSparse GraphBLAS library in a postgres extension [1] that fluidly blends algebraic graph theory with the relational model, the main flow is to use sql to structure complex queries for starting points, and then use the graphblas to flow through the graph to the endpoints, then joining back to tables to get the relevant metadata. On cheap hetzner hardware (amd epyc 64 core) we've achieved 7 billion edges per second BFS over the largest graphs in the suitesparse collection (~10B edges). With our cuda support we hope to push that kind of performance into graphs with trillions of edges.

[1] https://github.com/OneSparse/OneSparse

Re:The Epstein Network by babelfish on Monday November 17, 2025 @05:56PM (#45959382)
I don't have any experience on graph modeling, but it seems like Neo4j should be able to support 1 trillion edges, based on this (admittedly marketing) post of theirs? https://neo4j.com/press-releases/neo4j-scales-trillion-plus-...
Re:The Epstein Network by jandrewrogers on Monday November 17, 2025 @06:56PM (#45959854)
The graph database market has a deserved reputation for carefully crafting scaling claims that are so narrowly qualified as to be inapplicable to anything real. If you aren't deep into the tech you'll likely miss it in the press releases. It is an industry-wide problem, I'm not trying to single out Neo4j here.

Using this press release as an example, if you pay attention to the details you'll notice that this graph has an anomalously low degree. That is, the graph is very weakly connected, lots of nodes and barely any edges. Typical graph data models have much higher connectivity than this. For example, the classic Graph500 benchmark uses an average degree of 16 to measure scale-out performance.

So why did they nerf the graph connectivity? One of the most fundamental challenges in scaling graphs is optimally cutting them into shards. Unlike most data models, no matter how you cut up the graph some edges will always span multiple shards, which becomes a nasty consistency problem in scale-out systems. Scaling this becomes exponentially harder the more highly connected the graph. So basically, they defined away the problem that makes graphs difficult to scale. They used a graph so weakly connected that they could kinda sorta make it work on a thousand(!) machines even though it is not representative of most real-world graph data models.

Re:The Epstein Network by babelfish on Tuesday November 18, 2025 @04:34PM (#45972450)
Thanks for taking the time to respond! Inspired me to go read the Facebook TAO paper.
Re:The Epstein Network by theteapot on Monday November 17, 2025 @08:24PM (#45960362)
> It would be cool to have something like this ..

Aren't LLMs something like this?

Re:The Epstein Network by djtango on Monday November 17, 2025 @08:43PM (#45960459)
An LLM probabilistically produces tokens over its model which is why it can hallucinate whilst an actual graph model would not have that issue
Re:The Epstein Network by mmooss on Monday November 17, 2025 @10:18PM (#45961019)
> It is a famously non-trivial computer science problem and much of the serious R&D was not done in public historically.

Could you point us to any public research on this issue? Or the history of the proprietary research? Just the names might help - maybe there are news articles, it's a section in someone's book, etc.

Re:The Epstein Network by johongo on Monday November 17, 2025 @05:56PM (#45959385)
Emil Eifrem (founder of Neo4j) has a talk about them doing this with the Panama papers
Re:The Epstein Network by afavour on Monday November 17, 2025 @07:22PM (#45960015)
The New York Times has an API that lets you query “tags” or “topics” and the articles associated with them:

https://developer.nytimes.com/docs/semantic-api-product/1/ov...

The Guardian has similar:

https://open-platform.theguardian.com/documentation/tag

Either or both could be an interesting starting point for something like that. I tried to find something for the BBC and was surprised they didn’t have anything. I would have figured public media would have been a great resource for this.

Re:The Epstein Network by ggm on Monday November 17, 2025 @07:43PM (#45960134)
Given 6 degrees is rooted in reality, this means we can draw causal graphs from anyone (bad) to anyone (we don't like) and then invent specious reasons why it means "it's all connected, man"

That said, some networks of shorter paths than 6 are interesting. Right now, there's a 1:1 direct path from these documents to a bunch of people with an interest in confounding what evidentiary value they have in justice processes. That's more interesting to me, than what the documents say right now.

Re:The Epstein Network by Centigonal on Monday November 17, 2025 @08:23PM (#45960357)
Re:The Epstein Network by pbronez on Tuesday November 18, 2025 @01:28AM (#45961995)
Yup, this is a fantastic project and probably the most mature attempt at a global knowledge graph for contemporary news.
Re:The Epstein Network by scotty79 on Tuesday November 18, 2025 @01:29AM (#45961999)
300 categories, 60 attributes ... Doesn't sound very high res.
Re:The Epstein Network by pjc50 on Tuesday November 18, 2025 @06:35AM (#45963779)
Someone did one for (a small subset of) UK media. People were furious. https://brokenbottleboy.substack.com/p/mapped-out