Re:Ask HN: What roles have you had the most difficult time f...
by lastofus on Friday January 19, 2018 @06:52PM
(#16191026)
What are some if the types of problems that needed to be solved when cleaning data that required heavy tooling?
[ Parent
| on HN ]
Re:Ask HN: What roles have you had the most difficult time f...
by itronitron on Friday January 19, 2018 @11:05PM
(#16192004)
Lots of data typically means streams of data, which means processes running 24/7 moving data and files around. Streams, connectivity, and processes can cut out periodically which means you need some logic to reconcile and fill the gap and also restart the processes. You will also need some data QA as it is perfectly reasonable to get 'extra' data, either as duplicates or metadata bleeding into content.
If your data is from disparate sources then you may need to normalize timestamps across records from different sources, you may be dealing with different languages, identical tokens that mean different things depending on the source, different formatting of numeric fields, etc.
This is an incomplete list, the GP probably has a more exhaustive list of problem types...