State-of-the-art text classification with universal language models

Re:State-of-the-art text classification with universal langu... by Abundnce10 on Tuesday May 15, 2018 @07:50PM (#17078871)

This method dramatically improves over previous approaches to text classification, and the code and pre-trained models allow anyone to leverage this new approach to better solve problems such as: Finding documents relevant to a legal case; Identifying spam, bots, and offensive comments; Classifying positive and negative reviews of a product; Grouping articles by political orientation;

I'm starting a new project where I'm given many recipes and I need to take in a free form text of recipe ingredients (e.g. "1/2 cup diced onions", "two potatoes, cut into 1-inch cubes", etc.) and build a program that identifies the ingredient (e.g. onion, potato), as well as the quantity (e.g. 0.5 cup, 2.0 units). Could I use something like Fast.ai to tackle this problem?

[ Parent | on HN ]

Re:State-of-the-art text classification with universal langu... by timClicks on Wednesday May 16, 2018 @12:19AM (#17079986)

If you have sufficient labelled data, conditional random fields works well for this kind of problem. A technical team from NY Times have a great piece on it https://open.blogs.nytimes.com/2015/04/09/extracting-structu...

[ Parent | on HN ]

Re:State-of-the-art text classification with universal langu... by ioot17 on Wednesday May 16, 2018 @02:57PM (#17085361)

CRF works quite well, it's actually what I utilize right now to approach recipe parsing on https://cookalo.com/. It's based on CRFsuite with Python bindings for data training on already labeled recipes. If you build your own app and want to do some comparison, feel free to run some benchmarks against it.

[ Parent | on HN ]

Re:State-of-the-art text classification with universal langu... by Abundnce10 on Wednesday May 16, 2018 @07:35PM (#17087322)

Very cool! It sounds like you followed a similar approach that the NY Times used in their recipe parsing approach, correct?

How does your API handle ingredients with multiple options (e.g. "1 1/2 cups seedless red or green grapes")?

[ Parent | on HN ]

Re:State-of-the-art text classification with universal langu... by ioot17 on Thursday May 17, 2018 @04:59PM (#17095709)

Yes, that's correct, it's similar to the mechanisms NY Times guys were using and I've been focusing on the datasets to feed the CRF with as it's what drives the whole thing. This is the output I've got based on your example: [ { "unit": "cup", "input": "1$1/2 cups seedless red or green grapes", "name": "red grapes", "qty": "1$1/2", "comment": "seedless or green" } ]

Don't hesitate to try the API out by pasting some examples to the white box on the site and pressing the "Try it out!" button, it's interactive :)

[ Parent | on HN ]

Re:State-of-the-art text classification with universal langu... by Abundnce10 on Friday May 18, 2018 @01:08PM (#17103077)

Don't hesitate to try the API out by pasting some examples to the white box on the site and pressing the "Try it out!" button, it's interactive

Sweet, I didn't realize it was interactive. I'll give it a try!

[ Parent | on HN ]

Re:State-of-the-art text classification with universal langu... by jph00 on Tuesday May 15, 2018 @09:14PM (#17079254)

I'm not sure - what you're describing is information extraction. I haven't tried that yet, but I'm certainly interested in doing so (especially for medical data).

[ Parent | on HN ]

Re:State-of-the-art text classification with universal langu... by state_less on Tuesday May 15, 2018 @09:34PM (#17079324)

Similarly, could something like this be useful to extract out a command that a user wants to run from a transcription? For example, "Add a user named Jenny to our client list.", which results in the command, 'create user Jenny'. Or, "Could you add Jenny to our client list?", which results in the same command, 'create user Jenny'. Perhaps instead of outputting the next word, output the expected command from a set of commands?

[ Parent | on HN ]