couple of additional points: - encoding - your input will have different encodings and its quite hard to guess the correct one, however you should at least try to convert everything into 1 encoding (f.e. utf8)

- by setting CURLOPT_ENCODING to '' you don't have to worry about (un)gzipping as curl ll do this for you (or it should)

- it might be a good idea to use url hash as url id (f.e. crc32)

- you should check content-length and content-type to avoid downloading huge files

btw your coding style is very disturbing. there shouldn't be spaces before or after ->