I haven’t done any significant projects using neural networks for sequence modeling or analysis, but when I do, I plan to start with a temporal convolutional network, based on [0]. They argue that RNNs being standard is likely an artefact of the history of the field, while they get superior performance from TCNs.

[0] https://github.com/locuslab/TCN

The truth is that the best architecture depends on the problem, and you get a lot out of "graduate student gradient descent", i.e., hyperparameter search and fiddling with the architecture fine grained details. You'll want to experiment with rnn, tcn, and transformer (t2t) models to find the best one for your problem.

Disclaimer, I wrote the TF RNN api.

Totally agree - it's hard to say which sequence architecture works best for your problem (are there long term dependencies, do they exist at multiple levels of scale, etc) and dataset size.

Convolutions are cheap to compute and can be more efficient on smaller data but it's also possible that a CNN outperforms when you have a small dataset and an RNN when you have a larger dataset.

For graduate students out there that would rather be doing research than "graduate student gradient descent" (or, high dimensional, non-convex optimization in your head), SigOpt (YC W15) is a SaaS Optimization platform that is completely free for academic research [0]. Hundreds of researchers around the world use it for their projects.

Disclaimer: I co-founded SigOpt and wasted way too much of my PhD on "graduate student gradient descent"

[0]: https://sigopt.com/solution/for-academia

I do a lot of sequence work (text).

While CNNs are tempting and fast to train I've never been able to get the accuracy I can from RNNs. In NLP, accuracy is important because for lots of tasks NLP is right at that inflection point of being good enough to be useful.. if it's good enough.

It's worth noting that this TCN paper gets a perplexity of 45.19 on WikiText-103. That was competitive in 2015.

The current state of the art is 29.2[1] - not their claim of 48.4 (unclear what that came from).

Still, CNNs are nice in an ensemble model if that's your thing. They do tend to pick up different things to RNNs, which can be useful.

Edit: I now understand why their reported metrics are so wrong. They use generic models to compare against. They list SOTA performance in their supplementary material (they still get them wrong though).

[1] http://nlpprogress.com/english/language_modeling.html

Thanks for digging into how their claims are misleading. As someone in an adjacent subfield, I was wondering if I was missing something. It does at least sound like an interesting potential component.

From my cursory understanding of TCN, it always outputs sequence of the same length as the input. If that is true, then its usage is severely limited compared to recurrent networks.

Please point to more substantive sources if I am wrong. I am very interested to make TCN work, as it is much faster.

This is the dangling carrot. Be wary of the stick in the rear.

> As much as I sometimes criticize FB and Google over privacy issues, they also do a lot of good by realeasing open source systems.

You realize that they do this so top researchers still want to work for them?

And is that is a bad thing? The work still gets released and can be used by anyone.

In a world with less perverse incentives these people whose education was largely funded by tax payers would be working on actual valuable projects rather than optimizing how to get people to click more ads for a private corporation.

I agree with the sentiment that ads are such a savage application, but...

* I'm confused by why you bring up tax-payer funded education... do you think these employees don't pay taxes?

* What is your definition of "valuable"?

Ads are core part of how today’s world economy works, like it or not.

What do you think would go awry, if we suddenly abandoned machine learning for advertising?

A lot of business would go bankrupt. And I’m not talking about adtech, but advertisers. There’s tons of them, from small mom and pop style to big companies that depend on ads to get customers. And no, they aren’t scams.

HN crowd doesn’t accept that, but most people here have no idea about how business works.

Has the cost of customer acquisition objectively dropped over the last twenty years? If so, where can I read about that? If not, why do you believe that?

Nobody says it's a bad thing (licensing traps aside, if any, which I didn't check). It just should be noted that when big corps/celebrities get struck by bad publicity they must react with some good move as a lever to counteract the negative press. I'm not saying they wouldn't have done this, but it's a very common move among politicians, corporations, celebrities etc. Their PR office is just doing their work.

Nope. And I’m not joking here, this is:

1. Give free tools to people 2. Drive adoptions of tools 3. ??? (Data-driven pivot) 4. Profit

Does the reasoning behind it matter?

The public benefits from it, and they get to hire researchers that want to continue that work. It sounds like a win-win?

It's a win-win, but with nasty side-effects.

It's like our economy and its effect on the climate. Win-win for consumers and companies, but (without intervention) a downward spiral for our planet.

Also, it puts us in a morally difficult situation because we are benefiting from the ones we criticize, and as such, it is hypocritical.

Of course everyone can do as they please, but in my view it is best to look for moral-issue-free software instead of using BigCorp's candy-ware.

> Also, it puts us in a morally difficult situation because we are benefiting from the ones we criticize, and as such, it is hypocritical.

What moral difficulties do you see? My opinion is these companies are despicable but taking advantage of their generosity is not hypocritical. Applications and motives can be immoral, tools without human action simply exist.

You are ignoring that their generosity exists for a reason.

Tools exist because of human action.

Also it’s now old tech for them - pushing it to the public domain allows the public to maintain it and improve it for free; not to mention having more people use their system is like free training before they hire the person.

However, pushing it out surely does enhance the public knowledge - in the same way that Carmack released his 3d engines that were out dated in industry by one gen but still helped the public.

Lastly - they need to push it out as they need to attract top talent, and need to demonstrate they have top tech there (and are willing to let their researchers claim credit for it once it becomes old enough).