The Unreasonable Effectiveness of Deep Feature Extraction

Re:The Unreasonable Effectiveness of Deep Feature Extraction by stared on Thursday February 14, 2019 @02:39PM (#19165190)

A few caveats here:

- It works (that well) only for vision (for language it sort-of-works only at the word level: http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html)

- "Do Better ImageNet Models Transfer Better?" https://arxiv.org/abs/1805.08974

And if you want to play with transfer learning, here is a tutorial with a working notebook: https://deepsense.ai/keras-vs-pytorch-avp-transfer-learning/

[ Parent | on HN ]

Re:The Unreasonable Effectiveness of Deep Feature Extraction by mlucy on Thursday February 14, 2019 @02:42PM (#19165225)

There's actually been a lot of really good work recently around textual transfer learning. Google's BERT paper does sentence-level pretraining and transfer to get state of the art results on a bunch of problems: https://arxiv.org/pdf/1810.04805.pdf

[ Parent | on HN ]

Re:The Unreasonable Effectiveness of Deep Feature Extraction by stared on Thursday February 14, 2019 @03:04PM (#19165459)

Thanks for this reference, I will look it up. Though, from my experience people in NLP still (be default) train from scratch, with some exceptions for tasks on the same dataset:

- https://blog.openai.com/unsupervised-sentiment-neuron/

- http://ruder.io/nlp-imagenet/

[ Parent | on HN ]

Re:The Unreasonable Effectiveness of Deep Feature Extraction by samcodes on Friday February 15, 2019 @01:26AM (#19168898)

This is true, but rapidly changing. In addition to fine tuneable language models, you can do deep feature extraction with something like bert-as-service [0] ... You can even fine tune Bert on your days, then use the fine tuned model as a feature extractor.

[0] https://github.com/hanxiao/bert-as-service

[ Parent | on HN ]