It's useful to view the GitHub projects for these models as reference implementations. They're intended to provide a roadmap for reproducing the research and to aid in implementing production libraries.
Regarding the latter, take a look at the work by HuggingFace, the Flair project, Spark-NLP and others.
"While GPT-2 has gained mindshare for its generative capabilities, BERT and other pre-trained Transformer Encoder models are used for production workloads" You rephrased my point pretty well, while openAI search for "fun" tasks, deepmind and others allow progress on real world tasks.
You use BERT which is nice but do you consider using it's successor: XLnet?
"take a look at the work by HuggingFace, the Flair project, Spark-NLP and others." I was aware of Flair (from Zalando) but thank you for Huggingface and Spark-NLP, I will take a look!
And earlier this month they released a local version: https://www.tabnine.com/blog/local/.
Writing Bloomberg's "market wrap" articles.
If it was a high-return fruit somebody would be doing it. Not necessarily publishing papers about it or trying to beat useless artificial benchmarks on it.
Not necessarily publishing papers about it Yes.
trying to beat useless artificial benchmarks on it. Wtf is this bullshit? AI benchmarcks are what direct progress in AI and allow to quantify it. And they are less and less artificial and more and more real world: E.g quora, reddit, Wikipedia and Facebook datasets.
> AI benchmarcks are what direct progress in AI
Sadly this is largely true.
The AI benchmarks + culture around it are the bullshit.
What actually moves forward the field of AI is:
- accessible
- reproducible
- comprehensible
results done with some thought and reasoning which is explained well, published well, and justified by more than some #$!& "our F1 score went up by 2 therefore our approach makes sense" bullshit.
AI benchmarks have done as much to retard progress in AI as they have to promote it.
Current AI benchmark top scores are gamification for big companies to waste even more resources running algorithms they can't explain. They are not machine learning, they are machine pissing contests.
Well, then perhaps you should go work on them, instead of ranting here.
There's nothing to fix. People work on what they want to work on. Things that seem important to you are not important to me, and the opposite. I'm OK with that.
I'm not sure. I rarely have to do that explicitly in my head. Perhaps a model should learn to infer/guess them implicitly, from context, just like I do.
what exactly is more important?
In my opinion, having a world model (for common sense) and situational awareness (e.g. through sensor fusion, or from prior conversational history, or using some externally supplied conditioning) would be far more important.
GPT-2 does not generate meaningful sentences or only by accident
You think adding POS tags would help it generate meaningful sentences?
In my opinion, having a world model (for common sense) and situational awareness (e.g. through sensor fusion, or from prior conversational history, or using some externally supplied conditioning) would be far more important. Haha you basically want a general intelligence (AGI), I want it too! And not enough persons works on "architecting" such a thing. Opencog may interest you a lot then. But the reality is many other "simpler" tasks are needed to make this happen.
having a world model (for common sense) is an NLP task There are some interesting results https://github.com/sebastianruder/NLP-progress/blob/master/e... OpenAI does not work on this task sadly, at least for now.
You think adding POS tags would help it generate meaningful sentences? I would be clearly insufficient yet necessary. I believe they already use internally a POS tagger and a dependency parser.
Interesting. Where did you see that?
GPT-2 has not (yet) been trained to predict POS tags to my knowledge, nor BERT, or ernie 2 or xlnet has, but I think they have great potential to improve POS accuracy.
From my observations most SOTAs come from chineses researchers by far, followed by deepmind.
BTW isn't that a sad truth that not even one of all major AI actors has a draft of an AGI architecture, something comparable to CYC or opencog. https://wiki.opencog.org/w/CogPrime_Overview
Two other observations I would like to share: Many important NLP tasks have almost nobody publicly working on them it seems, on paperswithcode.com or NLP-progress (from github) some tasks have only one or two papers... And many others have not evolved since 2016. Most of the time it seems trivial to beat the old state of the art, just use BERT or XLnet on a task where nobody applied it before and hop, free state of the art for you! Yet researchers don't seems to chase those low hanging, high returns fruits. Also researchers seems to work a lot in isolation, many new generic improvements like new optimizers (RAdam for example) and new activation functions (Swish) allow to beat most of older state of the art on almost all task just by using them. Yet researchers will take years before using them because of an absurd inertia. Also unlike an open source program, BERT and XLnet have very low response and activity on github despite major open issues... </rant>