yeshengm

Re:Apache Arrow 3.0 on Wednesday February 03, 2021 @08:02PM (#26021331)

Since you need fast lookup, traditional transactional row store like RocksDB seems a better fit.

Re:Apache Arrow 3.0 on Wednesday February 03, 2021 @06:20PM (#26020563)

Are these execution engines internally using Arrow columnar format or are they just exposing Arrow as a client wire format? AFAIK Spark and Presto does not use Arrow as execution columnar format, but just data sources/sinks.

Ask HN: Why there are so many AI compilers?

Posted by yeshengm on Thursday May 07, 2020 @12:07AM
from the ask-hn dept.

There are a lot of AI compilers out there: MLIR/TVM/XLA/Halide/Hgraph/TensorRT etc... Do we really need so many AI compilers given that only PyTorch/Tensorflow/MXNet are mainsteam frameworks? Or do HW vendors/frameworks really have the incentive to have their own set of IRs?

( Read More... | no comments | on HN )

Re:Borg: The Next Generation [pdf] on Tuesday May 05, 2020 @11:15PM (#23087598)

Yeah. Likely a small amount of analytics jobs taking up most memory/CPU footprint, while critical prod OLTP jobs take very little resource comparably.

Re:What's Next for Scala on Saturday April 11, 2020 @04:27PM (#22843963)

Nowadays most Spark users use either PySpark or Spark SQL. Spark is designed in some sense to mimic Scala collection library. Rust is nice but its interop with other open source big data libs (Hadoop/Zookeeper/HBase) can be painful.

Re:What I Wish I Knew When Learning Haskell on Saturday April 11, 2020 @08:00AM (#22841204)

I'd say start with some eager functional language such as OCaml without the "object" part. Haskell code is hard to debug cuz you can't rely on good old print.

Monad itself is never a silver bullet. Type systems and Composability are the true power of FP IMO.