As someone that's worked on some rather intensive NLP implementations, Spacy 3.0 and HuggingFace both represent the culmination of a technological leap in NLP that started a few years ago with the advent of transfer learning in NLP. The level of accessibility to the masses these libraries offer is game-changing and democratizing.
I want to use AI to translate (localize) messages for free software, in my case, Ukrainian language. My plan to improve quality of automated translation is to translate from similar languages in parallel, i.e. give a same message in English, Russian, Polish, and expect message in Ukrainian as output.
Where I should start? Which libraries to use? How to connect them? How to train them?
https://huggingface.co/Helsinki-NLP/opus-mt-pl-uk
and English to Ukrainian
https://huggingface.co/Helsinki-NLP/opus-mt-en-uk
You can test the models with the form input on the right-hand side
And here's a site with docs and code examples for basic usage in your app
https://huggingface.co/transformers/model_doc/marian.html
You can also use Marian's training code on PyTorch with custom source and target texts.
Currently, volunteers manually translate from English to all other languages. It's possible to use AI to help, but quality of translation is too low. Dumb Translation Memory database is much more helpful than AI translator.
In the new system, one volunteer will translate messages from English to his native language, while all others will be able to use this translation as the additional constraint, to improve quality of translation to their native language. More input languages -> more constraints -> better quality of translation.
My best idea, so far, is to use a multilingual tokenizer (SpaCy looks good) and a linear transformer, because linear transformers are able to accept large inputs, with thousands of tokens. IMHO, I can input multiple translations of messages (they are short), and define loss as expected Ukrainian translation.
However, I'm completely new to this field. I completed just one AI project so far: recognition of animals at video. I have no idea how to start.
For example, I have no idea how to hint transformer that input messages are the same, just in 4 different languages. Should I interleave messages, like "Cannot Не можу open відкрити file файл : : ", or put them aside, via separator: "\0Cannot open file: \0Не можу відкрити файл: \0", or create few independent inputs? Or use a memory and improve quality of output message incrementally?
Any hints where to start?
My approach to AI is somewhat conservative because of working in a law-adjacent field where explainability is paramount. When it comes to getting "smart" I prefer forward-chaining logic over facts, and facts include predictions from models too. But at least there is a "judge"/engine to coordinate how the predictions from the ensemble of models maps to actions. I love me some pertained neural nets, but use them more as black box appliances.
There's also https://github.com/jruizgit/rules but haven't tried it yet.
Let’s say you wanted to build a system where you used an efficient bag-of-words text classifier to select paragraphs that might have information of interest, and then you wanted to run an entity recognizer and recognise relation triples between predicates and pairs of entities. When extracting the triples, you want to use the lemma of the relation word, so you want to map “dove” to “dive” when it’s a verb etc. It’s possible to build this system directly using PyTorch modules for the various model parts, but you’ll need to write the various bits of logic to string together the model predictions yourself, and for tasks like lemmatization that are pretty easy, you’ll struggle to find existing systems that you’ll actually want to use. A lemmatization system that’s published for PyTorch will probably be designed for languages where lemmatization is really hard, but for English it’s really easy.
spaCy has a good architecture and API for this system level stuff, where you’re putting together models into practical solutions. It also has a Doc object that makes it really easy to actually work with the system outputs, especially to relate multiple levels of annotation to each other.
Partly because orchestrating a number of models is kind of a hassle in lower-level frameworks, a lot of guides will encourage you to take entirely joint approaches to this type of system. In theory you can bypass problems like the lemmatization and NER if you take a sequence-to-sequence approach, and generate the relation triples as just arbitrary data. But this has a lot of limitations. It’s difficult to express structural constraints that you know should hold about the triples, the system will be much much slower, and might require vastly more training data. It’s also difficult to divide the task up between different people, and it’ll be difficult to analyse the system errors, iterate on individual parts, or inject rule logic to ensure certain invariants about the output. All these facts about joint approaches increase the risk of the project failing; they add large uncertainties that keep projects from getting out of the prototype stage.
Maybe there's a comp strategy i don't know about - but they've created SO much value for the world.
I will say that people are using spaCy for free because that is what we asked them to do. I chose to make the library free and open-source when I first released it because I had the idea that I would be able to make that work out for me, if I could make this thing that would be useful to people and if they could be convinced to adopt it. And in order to convince people to adopt it, we've been telling people that spaCy will stay free and that we'll continue to work on it. So everything's going to plan here. Even if things weren't working out well for us (and they are), the fault would be entirely ours. I don't think we'd have any right to suddenly say, "Oh none of you jerks are paying, how unfair".
(For the record, we make money from sales of our annotation tool, Prodigy: https://prodi.gy . If you're reading this and you like spaCy, check it out ;)
Your project was 1 of the 2 instrumental tools in my project to structure the transcripts of every word said on the floor of the New York State Senate over the past ~30 years in order to develop a topic-based “proximity” heuristic (based on CorEx, the 2nd instrumental tool) for which state senators were focused on which issues, based on the things they actually said on the record, not based on their press statements or their voting records (the latter of which doesn’t capture all the information you’d hope it would due to procedural nuances too obscure to detail here).
Thank you. Thank you, thank you, thank you.
Please invest more in SEO; I didn't find you guys two weeks ago when I researched different options for audio annotation :D.
Using Amazon ground truth which works fine (although seems quite MVP outside the core functionality e.g wrt reporting or user creation).
What tool have you had success with?
Spacy3 practically covers 90% of NLP use-cases with near SOTA performance. The only reason to not use it would be if you are literally pushing the boundaries of NLP or building something super specialized.
Hugging Face and Spacy (also Pytorch, but duh) are saving millions of dollars in man hours for companies around the world. They've been a revelation.