News for hackers. Stuff that matters.

A comment on Loading a trillion rows of weather data into TimescaleDB. Parent comment

Re:Loading a trillion rows of weather data into TimescaleDB by dboreham on Tuesday April 16, 2024 @11:04AM (#40052889)
Quick note to observe that all of the above while true becomes less of a practical issue as CPUs become faster vs i/o, which they have done and probably will keep doing.
[ Parent | on HN ]
Re:Loading a trillion rows of weather data into TimescaleDB by simiones on Tuesday April 16, 2024 @12:32PM (#40054069)
Isn't the opposite mostly happening, with CPU's single-core performance mostly flat for the last decade, while I/O speeds have been improving every year (especially in networking, not as sure about disk I/O).
[ Parent | on HN ]
Re:Loading a trillion rows of weather data into TimescaleDB by pphysch on Tuesday April 16, 2024 @12:14PM (#40053855)
Feels like you're a couple decades behind. Single core speeds haven't gone up much in the last decade, especially for "server" CPU models.
[ Parent | on HN ]
Re:Loading a trillion rows of weather data into TimescaleDB by forrestthewoods on Tuesday April 16, 2024 @12:11PM (#40053819)
Wait what no stop that what are you talking about.

Transforming data and then transforming it back will always be stupid and costly – in more ways than one.

[ Parent | on HN ]
Re:Loading a trillion rows of weather data into TimescaleDB by relaxing on Tuesday April 16, 2024 @06:18PM (#40057972)
Not if you save transit time that can be more quickly made up in processing time.
[ Parent | on HN ]
Re:Loading a trillion rows of weather data into TimescaleDB by rjmunro on Wednesday April 17, 2024 @04:32AM (#40061933)
True, but in this case we are converting to strings that are longer because they require quoting and escaping etc. Transit time will be longer.
[ Parent | on HN ]