News for hackers. Stuff that matters.
Re:Max severity RCE flaw discovered in widely used Apache Pa... by bri3d on Sunday April 06, 2025 @02:54PM (#43603943)
This seems to come from parquet-avro, which looks to attempt to embed Avro in Parquet files and in the course of doing so, does silly Java reflection gymnastics. I don’t think “normal” parquet is affected.
Re:Max severity RCE flaw discovered in widely used Apache Pa... by tikhonj on Sunday April 06, 2025 @03:22PM (#43604161)
Last time I tried to use the official Apache Parquet Java library, parsing "normal" Parquet files depended on parquet-avro because the library used Avro's GenericRecord class to represent rows from Parquet files with arbitrary schemas. So this problem would presumably affect any kind of Parquet parsing, even if there is absolutely no Avro actually involved.

(Yes, this doesn't make sense; the official Parquet Java library had some of the worst code design I've had the misfortune to depend on.)

Re:Max severity RCE flaw discovered in widely used Apache Pa... by twoodfin on Sunday April 06, 2025 @03:49PM (#43604367)
Indeed, given the massive interest Parquet has generated over the past 5 years, and its critical role in modern data infrastructure, I’ve been disappointed every time I’ve dug into the open source ecosystem around it for one reason or another.

I think it’s revealing and unfortunate that everyone serious about Parquet, from DuckDB to Databricks, has written their own “codec”.

Some recent frustrations on this front from the DuckDB folks:

https://duckdb.org/2025/01/22/parquet-encodings.html

Re:Max severity RCE flaw discovered in widely used Apache Pa... by dev_l1x_be on Monday April 07, 2025 @04:01AM (#43609018)
Unfortunately many of the big data libraries are like that and there is no motivation to fix these things. One example is the ORC Java libraries that had 100s of unnecessary dependencies while at the same time importing the filesystem into the format itself.
Re:Max severity RCE flaw discovered in widely used Apache Pa... by jeeeb on Sunday April 06, 2025 @06:03PM (#43605332)
The Apache Arrow libraries are a good alternative for reading parquet files in Java. They provide a column oriented interface, rather than the ugly Avro stuff in the Apache Parquet library.
Re:Max severity RCE flaw discovered in widely used Apache Pa... by amluto on Sunday April 06, 2025 @03:16PM (#43604120)
The documentation for all of this is atrocious.

But if avro-in-parquet is a weird optional feature, it should be off by default! Parquet’s metadata is primarily in Thrift, not Avro, and it seems to me that no Avro should be involved in decoding Parquet files unless explicitly requested.

Re:Max severity RCE flaw discovered in widely used Apache Pa... by bri3d on Sunday April 06, 2025 @05:21PM (#43605060)
To the sibling comment’s point, I suppose it’s not weird in the Java ecosystem. The parquet-java project has a design where it deserializes Parquet fields into Java representations grabbed from _other_ projects rather than either having some kind of canonical self-representation in memory or acting as just an abstract codec. So, one of the most common things to do is apparently to use the “Avro” flavored serdes to get generic records in memory (note that the actual Avro serialization format is not involved with doing that; parquet-java just uses the classes from Avro as the in memory representations and deserializes Parquet into them). The whole approach seems a bit goofy; I’d expect the library to work as some kind of abstracted codec interface (requiring the in-memory representations to host Parquet, rather than the other way around - like how pandas hosts fastparquet in Python land) or provide a canonical object representation. Instead, it’s this in between where it has a grab bag of converters that transform Parquet to and from random object types pulled from elsewhere in the Java ecosystem.
Re:Max severity RCE flaw discovered in widely used Apache Pa... by amluto on Sunday April 06, 2025 @06:09PM (#43605376)
I’d still like to see a clear explanation of where one can stick a Java class name in a Parquet file such that it ends up interpreted by the Avro codec. And I’m curious why it was fixed by making a list of allowed class names instead of disabling the entire mechanism.