Re:Parsing short hexadecimal strings efficiently
by nostrademons on Wednesday April 17, 2019 @06:55PM
(#19687143)
Oftentimes the biggest data-ingestion tasks happen at organizational boundaries. You find this great open-source data set that's 77TB, or you contract with another company to dump a 50GB CSV file on your FTP server every night, or you pay for API access to a feed that generates 1G/sec, or another department dumps 1TB in your S3 bucket nightly.
There's a strong incentive to use standardized text-based formats (particularly JSON and CSV) for these. You can inspect sample data with a text editor, or just with 'head'. You can run really simply data analytics with just UNIX tools (I've found that a 'gunzip -c | grep | awk | grep | wc' pipeline has gotten me results in 15 seconds that would've taken 5 minutes with Python or half an hour in Java). Everybody already has a well-tested parser and serializer for the format in any language they might choose to use. Everybody's already familiar with them; you need only document field types & formats rather than giving a byte-by-byte description of the protocol. With JSON in particular, you've got 1-line parsing built into the browser instead of needing to ship a large library. If problems arise, you can dump the raw input record to the console for debugging without needing to pore over a hex dump, and you have tools available (jq, awk) for filtering out extraneous information.
If you control both sides of the protocol, than by all means use an efficient binary serialization protocol like Cap'n'Proto. But a lot of interesting data ingestion problems don't fit in this category. Oftentimes the system designer doesn't actually have control of the format that data is provided to the system.
There's a strong incentive to use standardized text-based formats (particularly JSON and CSV) for these. You can inspect sample data with a text editor, or just with 'head'. You can run really simply data analytics with just UNIX tools (I've found that a 'gunzip -c | grep | awk | grep | wc' pipeline has gotten me results in 15 seconds that would've taken 5 minutes with Python or half an hour in Java). Everybody already has a well-tested parser and serializer for the format in any language they might choose to use. Everybody's already familiar with them; you need only document field types & formats rather than giving a byte-by-byte description of the protocol. With JSON in particular, you've got 1-line parsing built into the browser instead of needing to ship a large library. If problems arise, you can dump the raw input record to the console for debugging without needing to pore over a hex dump, and you have tools available (jq, awk) for filtering out extraneous information.
If you control both sides of the protocol, than by all means use an efficient binary serialization protocol like Cap'n'Proto. But a lot of interesting data ingestion problems don't fit in this category. Oftentimes the system designer doesn't actually have control of the format that data is provided to the system.