Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor...
by ithayer on Monday August 18, 2014 @12:06PM
(#8193222)
Probably way beyond the scope of that project, but actually this is a pretty interesting and well-studied problem called segmentation, where you train a system to insert the correct breaks. This is used in natural language problems like OCR or machine translation.
Maybe google "machine translation segmentation" or OCR for references to papers on the topic, but methods for doing this are really successful (especially on english).
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor...
by adkatrit on Monday August 18, 2014 @01:24PM
(#8193570)
i bet you'd have better luck just removing all spaces and then resorting to a text segmentation algorithm. Peter Norvig has some great papers on how to do this effectively. I've wrapped his code with a Tornado web service: https://github.com/adkatrit/text-segmentation-server
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor...
by marktangotango on Monday August 18, 2014 @09:09AM
(#8192447)
Nothing related to analyzing free form text should ever be considered 'easy'. The wrinkle you describe was just the cherry on top.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor...
by arthurjj on Monday August 18, 2014 @08:05AM
(#8192272)
How did you end up dealing with the issue? My first idea would be to remove all the spaces and parse that
Oh, but their data warehouse thing that nobody knew what it was called, inserted a random space every so often and removed random spaces every so often making the text completely unreliable. Compiled with all the normal misspellings, fat fingered acronyms, slang and other nonstandard ways of writing things it turned out to be a huge job.
"Bicycle stolen at 0930 from residence on the 300 block of E 10th St, East Village"
would get turned into "Bicycle stol en a t093 0 from r es iden ceon the 300 block of E 10 thst, Ea st Vil lage."
Good luck with that.