marknadal

Note: I doubt anyone will see this comment as HackerNews has shadow banned my account because my Open Source project competes with some of YC's investments (it undermines & destroys the need/market for some of their crypto-coin scams).

I interviewed with YC a few years ago and their entire process was terribly unprofessional - I've gone through 3 accelerator programs now, and interviewed at many more. YC itself seems pretty awesome, but beware of the politics - if a single person (HN mod in my case) feels like you are a threat to their status, that person has enough marketing power to hurt you. Find the GOOD people in YC and work with them, don't leave it to chance or you'll fall out of good graces.

In contrast, look what Mozilla is doing! They're helping move the internet (the world, the community) forward, and are a non-profit foundation. I highly recommend applying to their program. They are GOOD people, doing good work. Please please do everything you can to help them make this a success - the internet depends on it.

Meanwhile I can barely get Chrome/NodeJS to parse 20MB in less than 100ms :(.

How useful (or useless) would Simdjson as a Native Addon to V8 be? I assume transferring the object into JS land would kill all the speed gains?

I wrote my own JSON parser just last week, to see if I could improve the NodeJS situation. Discovered some really interesting factoids:

(A) JSON parse is CPU-blocking, so if you get a large object, your server cannot handle any other web request until it finishes parsing, this sucks.

(B) At first I fixed this by using setImmediate/shim, but discovered to annoying issues:

(1) Scheduling too many setImmediates will cause the event loop to block at the "check" cycle, you actually have to load balance across turns in the event loop like so (https://twitter.com/marknadal/status/1242476619752591360)

(2) Doing the above will cause your code to be way slow, so a trick instead, is to actually skip setImmediate and invoke your code 3333 (some divider of NodeJS's ~11K stack depth limit) times or for 1ms before doing a real setImmediate.

(C) Now that we can parse without blocking, our parser's while loop (https://github.com/amark/gun/blob/master/lib/yson.js) marches X byte increments at a time (I found 32KB to be a sweet spot, not sure why).

(D) I'm seeing this pure JS parser be ~2.5X slower than native for big complex JSON objects (20MB).

(E) Interestingly enough, I'm seeing 10X~20X faster than native, for parsing JSON records that have large values (ex, embedded image, etc.).

(F) Why? This happened when I switched my parser to skip per-byte checks when encountering `"` to next indexOf. So it would seem V8's built in JSON parser is still checking every character for a token which slows it down?

(G) I hate switch statements, but woah, I got a minor but noticeable speed boost going from if/else token checks to a switch statement.

Happy to answer any other Qs!

But compared to OP's 2.5GB/s parsing?! Ha, mine is a joke.

But for every 1 hater there are 100s of hackers that have found, starred, used, or told me (in our chat channel) they were thankful they discovered GUN via HackerNews.

& HN guidelines encourage on-topic submissions & comments (https://news.ycombinator.com/newsguidelines.html) "Anything that good hackers would find interesting ... anything that gratifies one's intellectual curiosity" even you have to admit that GUN discussions have sparked a lot of intellectual & algorithm chats.

Finally, you state I'm not spamming then say I am spamming. Spam is indiscriminate posting of something, yet your very own comment says "you always post GUN in discussions about CRDTs & decentralization." That is indeed on-topic according to HN policy.

Just to be clear, Martin (site owner) added GUN to the list, not me. Proof:

https://twitter.com/martinkl/status/1244672269395406849

Edit: (reply is now showing)

https://github.com/topics/crdt

Mind adding it to the site?

Not that SF isn't already a ghost town tho.

Or just travel originating from Europe and going inbound to USA?

If you ever have time, it'd be great to snag whatever code or data you had, to create a replicable test case for us to make a fix against. Sorry about the wasted time you experienced, I hate that.

Could you expand further what didn't work? We have some pretty heavy load in-production systems running. Definitely some rough edges still, so I'd like to hear what problems you had, so I can focus on fixing them in the future.

Thanks again.

https://era.eco/

http://axe.eco/

I have the same question.

In contrast, GUN, has 20M+ downloads/month from known sources: Internet Archive, HackerNoon, etc. (https://github.com/amark/gun see jsdelivr download stats).

I gave a presentation at Berkeley a few years back, right before Bram Cohen (BitTorrent) presented his Chia proof, Proofs of Space and Time (before IPFS ripped the name from Bram).

https://www.facebook.com/BlockchainatBerkeley/videos/2006069...

I'm handling Millions for $0 :/ with my https://github.com/amark/gun .

I'm curious to go back to the "old days of graphics" with limited RAM, loading from slow CDs, etc. & program like that myself, where can I start?

Just hit 19M+ monthly users.

In production with Internet Archive, HackerNoon, etc.

( https://github.com/amark/gun )

Scaling systems that have to run out of the browser by default is tough, but we're doing this on $0 in server costs, in javascript, with millions of users.

This means all other nodes in the network (routers, relays, security, storage, etc.) can safely error or not recurse or not retry, without any loss in redundancy guarantees.

Either the operation did succeed silently, and it is OK for originator to try again (idempotency keeps this safe), or things never finished processing, and it is OK to try again despite being out of order (CRDTs keep this safe).

It needs a fundamentally different economic model, that measures the gain in value added to the network, not the transaction (rebalance, distribution, etc.) of it: http://free.eco .

What are other projects in this space?

I know Martti Malmi (Satoshi's 1st Bitcoin contributor) is doing some stuff (no tokens) in this space.

Anybody else/know/have links/talks/articles on other (no blockchain) decentralized search?