Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

Re:Composability in Julia: Implementing Deep Equilibrium Mod... by cs702 on Thursday October 21, 2021 @12:37PM (#28947079)

Very cool. I have only one question:

Has anyone successfully applied DEQs to larger-scale cognitive tasks or benchmarks, as opposed to MNIST, which is a tiny trivial task by today's standards?

Think ImageNet-1000, COCO, LVIS, WMT language translation, ..., SuperGLUE. There's a long list of datasets and benchmarks that regular boring fixed-depth NNs tackle with remarkable ease these days.

Has anyone anywhere applied DEQs to any of those datasets / benchmarks?

[ Parent | on HN ]

Re:Composability in Julia: Implementing Deep Equilibrium Mod... by avikpal1410 on Thursday October 21, 2021 @01:48PM (#28948250)

MDEQ work applies DEQ to some of the large scale benchmarks you mention: https://arxiv.org/pdf/2006.08656.pdf

[ Parent | on HN ]

Re:Composability in Julia: Implementing Deep Equilibrium Mod... by cs702 on Thursday October 21, 2021 @02:03PM (#28948514)

Thank you! The results don't look that great (e.g., EfficientNet models achieve greater accuracy on ImageNet-1000 with ~5x fewer parameters), but the works looks interesting and worthwhile. I'll take a look.

[ Parent | on HN ]