Re:Ask HN: How are you cleaning and transforming data before...
by dataflowmapper on Wednesday May 14, 2025 @01:15PM
(#43986952)
Thanks for sharing that workflow, for more straight forward flows, that sounds like a decent approach. My main thoughts on where it could be improved, or where I see potential struggles, are when:
These are the areas where I'm exploring if a more structured way could offer benefits in terms of repeatability and accessibility for a wider range of users or complex scenarios. Appreciate the insight into your process and your thoughts on this.
[ Parent
| on HN ]
Re:Ask HN: How are you cleaning and transforming data before...
by sargstuff on Wednesday May 14, 2025 @04:51PM
(#43989075)
From doing historical data restorations (pre-json), using documented industry/standard structures & protocols, normalized data formats/layouts helps out quite a bit.
Note: awk/shell scripts, ~80's because relatively portable across platforms (mac,pc,dec,ibm (ignoring ebcdic), etc.)