Re:Show HN: NLP and NER-powered beautiful bank transaction feed
by omegalulw on Thursday October 28, 2021 @09:40AM
(#29025991)
What models are you using? Your problem seems simpler than general NER (just identifying a subset of entities). I would wager a good ol' LSTM or GRU can do it just fine.
Re:Show HN: NLP and NER-powered beautiful bank transaction feed
by ahmedahres on Thursday October 28, 2021 @09:45AM
(#29026059)
Hey! I'm the lead ML engineer behind the solution.
We have trained our custom NER to detect merchants using millions of transactions as training data rather than a general NER.
We have tested with a few general NER models but they weren't detecting merchants properly. We haven't tested with LSTM/GRU yet but that's a good suggestion!
Re:Show HN: NLP and NER-powered beautiful bank transaction feed
by i_have_an_idea on Thursday October 28, 2021 @11:24AM
(#29027188)
He was asking what algorithm you're using. Are you using a MaxEnt classifier, CRF, something else?
Our new approach consists of using custom Natural Language Processing and Named Entity Recognition models to predict a) what substring of a bank transaction string represents a merchant e.g. "FACEBK" and b) which canonical merchant e.g. "Facebook" this substring corresponds to in our merchant DB.
The toughest steps were to manually label thousands of bank transactions very carefully to use as training data, and to deploy the models in a production env where we need API response times to be under 200ms (usually what's required in order to incorporate this API in a payment auth flow).
We always optimize for accuracy (>99% currently), because we never want to return an incorrect merchant, but with this new approach our coverage is now at 80% of all bank transactions we see through our system.
We would love any feedback and comments, and also happy to answer any questions about the product or how we productionized it!