"There is only one exception if the page should be rendered with a full browser (not headless). In this case, 5 credits get charged."
About the product.
1. It logs all requests which had issues with the more descriptive cause. Always go through all of them and fix the issues. The more people use it the more stuff breakes and the product can be improved. So I guess will get way better in the next days ;-) 2. Will also check and fix. 3. Will definitely look into that!
If you run into more issues or have more comments would love to hear them here or at [email protected] . Thanks again!
Totally agree, the problem I see is that when you become big enough to be noticeable, websites start growing ban hammers at you for flagrant disregard of TOS. Working around that stuff becomes an art in of itself.
Are they "caching" responses or that offer is tailored to your user/cookie?
A couple comments in general:
1. Personally I think its better to be great at extracting one kind of data instead of average at many types. It makes sales and growth efforts easier. Pick one of those things (products, recipes, social, etc.) and just focus on that and get great at it.
2. I don't think you need the credit <-> request abstraction. Anyone using an API knows what a request is (I hope).
Now, a few comments regarding products specifically:
1. I got 500 errors on a couple random product URLs.
2. On an Amazon product that's on sale, I got back the original price but not the sale price.
3. If you truly want to be GREAT at scraping products, the 2 things most people in this space can't do are: (a) extract ALL high-res images for a product, and (b) extract a product's options and variant data (colors, sizes, etc. and availability for each combination)
Personally I think there are a ton of opportunities in this space. This is a good start and I wish you the best.