jo_kruger

Looking for review/feedback, ideas for new features, syntax improvements, etc.

Key features: high performance, zero memory allocation, up to 19 decimal places precision, fixed size memory layout (128 bits), no panics,..

Nats-mutex is a Golang distributed lock library using NATS JetStream.

It is like iceberg - website is just a tip... Website is interface to database of company profiles. Each of these profiles was build from corresponding corporate website.. all automatically. The process includes website crawling, page categorization, language detection, entity extraction, structured data composition, fuzzy comparison, etc... The key feature of the platform is 100% automated process which allows repeat the whole cycle for every profile on a regular base, which allows us to compare structured profiles and see all changes..

We have a platform running on more than 50 servers which includes crawlers, website categorizers, entity extractors, etc. It scans about 30 millions websites (categorized as corporate websites with content in English) on a regular base. The goal is to build a structured profile for every known company website.. and repeat this process every month in order to compare historical profile and generate "transactions" like "company X changed CEO", etc...

Yes. The most notable are https://www.aihitdata.com and https://www.happygrumpy.com First crawls corporate websites (~25 millions) and extracts key information such as people, contacts, etc. Second is a sentiment analysis tool.

I used RL3 to annotate words from multiple language dictionaries with total number of known words ~25 millions (i.e. 25m dictionary entries) - no problem at all

Sorry, we are working on a website - the project is quite mature (more than 10 years), but it was mostly for internal use. We just recently decided to make it public, which is not always easy to do. So we have a lot to do with documentation, etc.

Regarding your question -- there is no limitation on number of rules. But, I should say the definition of "number of rules" may be tricky. I don't see any reason to have a lot of high level rules -- i.e. annotators and asserts -- in most cases the number of annotators will be N time bigger than number of entity classes, or categories (in case of categorization task). On the other hand, there may be much more low-level rules (i.e. patterns and predicates used to form the high level rules).

Also, the built-in dawg dictionaries may help a lot - these dictionaries may handle millions of entries, behave same as other matchers (i.e. they can be used in same way as other patterns and regex matchers used), and work way faster than patterns. For instance, in RL3 you can define pattern \<{PERSON_FIRSTNAME},?\s{PERSON_LASTNAME}\> (which matches first name followed by optional comma followed by space and last name) where {PERSON_FIRSTNAME} and {PERSON_LASTNAME} are dictionaries..

This was actually one of the reasons we started RL3 -- make rules development not so expensive. Note, we started it more than 10 years ago as purely internal product. And just several months ago we decided to make it public (and currently it is free for personal / research and educational use). At the end, we were able to enable team of linguists (i.e. we actually searched for linguists - not even computational linguists) to write and support large library of NER patterns. Partially, it was possible due to named patterns -- i.e. we were able to develop sort of core library of patterns by small team of expensive resources (programmers and pro comp. linguists), and larger library of patterns based on this core...

We started RL3 (more than 10 years ago) because we had several projects with a huge number of patterns. We found other projects (present at that moment) were too heavy on a syntax, which make it complicated to support / manage large library of patterns. So, we tried to keep the power of regex, add new features (like named patterns, modules, templates and lookup dictionaries) but minimize additional syntax... as result we were able to enable team of computational linguists (i.e. not programmers) quite easily develop and support huge libraries of NER patterns and document classification rules.

Refer to email example (it demonstrate some features on a simple task) https://rl3.zorallabs.com/wiki/Extract_Email_Addresses_From_...

It is actually based on regex (on a low level). But the key difference -- it allows named patterns, modules, templates, and kind of classes. So, you may have a separate module with a definitions of related patterns - for instance, module for date and time patterns, module for location patterns, etc... then, you can include required module to your project and refer to these patterns (by name) from your own patterns.. like "{PERSON_NAME}[,:\s]?{JOB_TITLE}" may be a pattern which matches person name followed by optional punctuation and the by job title... where PERSON_NAME and JOB_TITLE are quite complicated patterns defined somewhere else.

Another key difference - it supports huge lookup dictionaries -- i.e. part of your pattern may be reference to dictionary which may contain thousands or even millions of entries... so, PERSON_NAME can be a dictionary.

...

It is free for non-commercial (personal, research and educational) use