Bistring – Bidirectionally Transformed Strings

Re:Bistring – Bidirectionally Transformed Strings by zawerf on Saturday July 13, 2019 @01:31PM (#20429121)

I was confused about the intended use case but there's more information in the docs folder: https://github.com/microsoft/bistring/blob/master/docs/Intro...

Apparently it's for machine learning where you want to pick out a span/substring in the original text but your model can only accept normalized text (I am guessing for stuff like transforming out-of-vocabulary words into UNK/unknown tokens). This solves that problem by keeping track of the index mapping between the original text and transformed text.

(picking out spans is very common task in NLP, for example see the SQuAD dataset: https://rajpurkar.github.io/SQuAD-explorer/explore/v2.0/dev/...)

[ Parent | on HN ]

Re:Bistring – Bidirectionally Transformed Strings by tavianator on Saturday July 13, 2019 @04:48PM (#20430054)

Yeah that's the idea. I'm not sure exactly how to communicate the purpose of the library in a few words. Possibly some animation that shows selecting spans of text and highlighting the corresponding one in the original text.

The formatted documentation can be viewed here btw: https://bistring.readthedocs.io/en/latest/Introduction.html

[ Parent | on HN ]

Re:Bistring – Bidirectionally Transformed Strings by bpicolo on Saturday July 13, 2019 @01:34PM (#20429134)

Took me a minute to comprehend (that the substring bit is the whole point was not obvious, because all string operations in Python are non-destructive regardless) but this is exactly what I imagined given that note. Glad that thought lined up with reality

[ Parent | on HN ]