Re:Ballista: Distributed compute platform implemented in Rus...
by lmeyerov on Tuesday January 19, 2021 @01:21AM
(#25830592)
We use arrow to do the validation 'for free': it has a typed columnar schema, so by passing the schema, we get the recordset validation without inspecting values.
we still need to do semantic validation, but w/ arrow, now we do that in bulk and on gpus :)
Webapps are dumb middleware that pipes data from the database into an http stream - but it needs to determine which database calls to invoke and sanitize all the incoming junk.