Re:Reducing logging cost by two orders of magnitude using CLP
by stingraycharles on Friday September 30, 2022 @09:05AM
(#33034317)
Yeah it’s silly, even when Spark is writing unstructured logs, that doesn’t mean that you can‘t parse them after-the-fact and store them in a structured way. Even if it doesn’t work for 100% of the cases, it’s very easy to achieve for 99% of them, in which case you’ll still keep a “raw_message” column which you can query as text.
Next up: Uber discovers column oriented databases are more efficient for data warehouses.
Next up: Uber discovers column oriented databases are more efficient for data warehouses.