News for hackers. Stuff that matters.

A comment on Scaling Clearbit to 2M API requests per day.

Re:Scaling Clearbit to 2M API requests per day by manigandham on Monday September 07, 2015 @07:11AM (#10180987)
Maybe I'm missing something but as the other comments have pointed out, this is very low scale. I'm not sure it can even be called at-scale in any sense these days.

Also the architecture seems incredibly fragile and complicated for what it's doing. I get that they've built some integrations and backend processes to assemble this data but the API serving could all run on a single app and database server since it's all just reads.

[ Parent | on HN ]
Re:Scaling Clearbit to 2M API requests per day by harlow on Monday September 07, 2015 @01:43PM (#10182287)
Hi manigandham, Harlow here (author of the post). You are absolutely correct; serving 2M API requests from a local data store wouldn't be considered "at-scale" these days.

In hindsight I should have added more information about where the actual work is being done -- I definitely missed the mark on parts of this post.

The lions-share of our work is done in an async fashion. 2M API requests (lookups) turns into 40M+ background jobs. These jobs fetch, aggregate, and scrub data from a number of downstream providers.

[ Parent | on HN ]