News for hackers. Stuff that matters.
Re:Why Searching Through 500M Pwned Passwords Is So Quick by zaroth on Tuesday February 27, 2018 @11:28AM (#16474960)
Ummm... because it’s an O(1) array lookup not a search at all? Infuriating.

It’s read-only static data. Spending even 60ms on the response is ridiculous. Reading from files in blob storage... WTF?

Ctrl-F Redis - was disappointed.

Actually, even forget Redis. Pre-generate each of the 1 million possible HTTP responses and store in a string array. The 5 character hex is the index into the array. Write < 100 lines of Go to load the data structure and serve it. What am I missing?

This is like “Hello World” in those HTTP Framework Benchmarks that used to make the rounds every few months.

Re:Why Searching Through 500M Pwned Passwords Is So Quick by djhworld on Tuesday February 27, 2018 @02:16PM (#16476859)
The 250mb redis instance on Azure costs $0.055 an hour[1], so nearly $500 p/year

Going down the small hosting a 100 line Go program route, the cheapest "B1" instance type with 1GiB of RAM costs $105.12 a year, and then if you want the service to be HA you need probably another instance in a different zone, maybe a load balancer as well.

I suspect Azure Functions + Blob Storage costs a lot less. Although he didn't mention how much CloudFlare was costing him.

[1] https://azure.microsoft.com/en-us/pricing/details/cache/ [2] https://azure.microsoft.com/en-us/pricing/details/virtual-ma...

Re:Why Searching Through 500M Pwned Passwords Is So Quick by dchuk on Tuesday February 27, 2018 @03:48PM (#16477733)
You could host this whole thing on the cheapest Digitalocean droplet if it were written as the person you're replying to described.
Re:Why Searching Through 500M Pwned Passwords Is So Quick by manigandham on Tuesday February 27, 2018 @05:36PM (#16478696)
And then it's a single server that has none of the scaling, ease of maintenance, or high availability of just using functions and table storage. This is the perfect scenario for serverless, why would you want to run a VM instead?

The current system works great, most of the hits will be cached, and optimizing the origin to be that milliseconds faster isn't worth it at all.

Re:Why Searching Through 500M Pwned Passwords Is So Quick by dchuk on Tuesday February 27, 2018 @10:25PM (#16480456)
It's actually NOT a perfect system for serverless, because it's a read-only (no writes), simple search "web app". The one thing that you don't get from a VM in your list is high availability I'll give you that, but otherwise this solution is an overengineered monstrosity for a simple problem if done in a more traditional way, as many many other people are calling out here.

If the only reason your web app is fast is when its data is cached, you need to take a long hard look at how your web app is built. It's not that hard to build a performant query system like this even with that amount of data.

Re:Why Searching Through 500M Pwned Passwords Is So Quick by manigandham on Tuesday February 27, 2018 @11:41PM (#16480727)
Read vs write has nothing to do with serverless, it's about well-encapsulated logic that doesn't need any server management and has granular pay-per-use billing.

It doesn't get much simpler than a function that does a key/value lookup or returns a text file; what exactly is over-engineered about that? The app is fast, it's < 100ms which is perfectly fine for non-cached access. This isn't mission critical real-time, it's a completely free service to lookup a password hash.

Everyone here is caught up in the usual "we can do better" perspective without actually thinking about why it was done this way. In reality the current method is cheaper, more reliable, more scalable and has 0 maintenance compared to anything else suggested so far.

Re:Why Searching Through 500M Pwned Passwords Is So Quick by emodendroket on Tuesday February 27, 2018 @04:53PM (#16478338)
I'm guessing he didn't do that because he's a prominent .NET/Azure guy and I don't think that stuff is well-supported in that environment.
Re:Why Searching Through 500M Pwned Passwords Is So Quick by jmiserez on Tuesday February 27, 2018 @02:50PM (#16477186)
He mentioned that Cloudflare is free for him.
Re:Why Searching Through 500M Pwned Passwords Is So Quick by th3byrdm4n on Tuesday February 27, 2018 @11:52AM (#16475255)
Precisely my first thought. Static readonly data? Why isn't this just sitting in memory?
Re:Why Searching Through 500M Pwned Passwords Is So Quick by vortico on Tuesday February 27, 2018 @12:12PM (#16475478)
Don't filesystem caches basically do this for you?
Re:Why Searching Through 500M Pwned Passwords Is So Quick by zaroth on Tuesday February 27, 2018 @12:23PM (#16475618)
Yes but with a heck of a lot more overhead than;

  response.body = result[(int)hexCode];
Re:Why Searching Through 500M Pwned Passwords Is So Quick by tgsovlerkhgsel on Tuesday February 27, 2018 @08:00PM (#16479774)
No, with a lot less overhead because you don't need to write custom web server code.

Edit: and if he points the requests directly at blob storage as he's suggesting, we're very close to to flat files on a filesystem again.

Re:Why Searching Through 500M Pwned Passwords Is So Quick by dullgiulio on Tuesday February 27, 2018 @01:45PM (#16476525)
You can do that after mmap(2) and you never have to reach out for read(2). Might need to use unsafe language constructs though.
Re:Why Searching Through 500M Pwned Passwords Is So Quick by mzzter on Tuesday February 27, 2018 @12:11PM (#16475467)
Sounds reasonable, 500mil * 160 bits of SHA-1 would be 500 mb of space. Even with some metadata to manage it, the smallest instances should be able to handle that.
Re:Why Searching Through 500M Pwned Passwords Is So Quick by lipnitsk on Tuesday February 27, 2018 @12:22PM (#16475603)
500,000,000*160bits/(8bits/byte)=10,000,000,000 bytes=9,536MB=9.313GB
Re:Why Searching Through 500M Pwned Passwords Is So Quick by quadrature on Tuesday February 27, 2018 @12:36PM (#16475747)
I think this solution is banking on storage + azure function cost being cheaper than the per instance cost to service the same load with an acceptable latency.
Re:Why Searching Through 500M Pwned Passwords Is So Quick by nodesocket on Tuesday February 27, 2018 @05:15PM (#16478523)
Couldn't he run a simple KV Consul cluster in a few zones and regions in either AWS or Google Cloud and be done?
Re:Why Searching Through 500M Pwned Passwords Is So Quick by manigandham on Tuesday February 27, 2018 @05:37PM (#16478712)
That's ridiculously complex and more expensive than just writing two functions that call table storage and blog storage APIs. Given the popularity of serverless, I'm surprised by all these answers saying he should run actual VMs when this the perfect use-case for functions.
Re:Why Searching Through 500M Pwned Passwords Is So Quick by dpweb on Tuesday February 27, 2018 @12:21PM (#16475588)
For an extreme diet, Id html app cache the pretty pictures. I know its deprecated but is still massive support in all browsers and is instantaneous fast. Removing all origin hits except api req for sure.

On the lookups CF workers has a global state that is shared amongst nodes. Been playing with that. Wondering if you hold string array in that and can get the origin hits below 10% you can run this svc with many million reqs day practically for free. Vps tho, not amzn.