News for hackers. Stuff that matters.
Re:Apache Tika – a content analysis toolkit by GEBBL on Monday September 14, 2020 @11:28AM (#24471302)
A timely link given there is a current discussion on the trouble with extracting meaningful text from pdfs in another thread on the front page! I look forward to reading the feedback on actual use of this
Re:Apache Tika – a content analysis toolkit by mumblemumble on Monday September 14, 2020 @11:33AM (#24471371)
Tika's PDF text extraction is fine if you're just trying to get searchable text. Which is what it's made for: Slurping doucments into Lucene. Fulltext search typically isn't terribly sensitive to getting the order of words right, and is even less sensitive to getting the formatting right.

If you're trying to get something fit for consumption by human (including via a screen reader) or an NLP pipeline, though, all the problems discussed in that FilingDB article still apply.

Re:Apache Tika – a content analysis toolkit by tonitosou on Monday September 14, 2020 @11:51AM (#24471587)
aspose can convert pdf to html
Re:Apache Tika – a content analysis toolkit by chaps on Monday September 14, 2020 @01:31PM (#24472753)
Firefox's PDF reader does the same. A few years back, I wrote wrote a pdf to csv converter with selenium -- it worked surprisingly well! Though, after I finished I found tabula and the code became immediately useless, hah.
Re:Apache Tika – a content analysis toolkit by rovr138 on Monday September 14, 2020 @01:44PM (#24472909)
When that happens to me, it's due to a keyword I hadn't thought of looking for. Then, while building the project, it came to me.
Re:Apache Tika – a content analysis toolkit by Ocelot20 on Monday September 14, 2020 @11:49AM (#24471568)
I used Tika to build a search engine prototype, and it was fantastic for getting us up and running quickly.

It's a really easy to use generic parser for a bunch of document types. The downside of being so generic and easy to use is that you end up lacking document-specific context that could be useful. For example: Do you consider the header/footer text to be important, or just noise (Page 1, Page 2, etc.)? Is the text contained in the Table of Contents or section headers important, or just the actual content? You won't find any ways to tweak the result, which could be a good or bad thing depending on your use case.

We ended up using it as our "fallback" parser, writing more contextually aware ones for document types of greater importance to our use case (PDFs were high on the list).

Re:Apache Tika – a content analysis toolkit by tonitosou on Monday September 14, 2020 @02:21PM (#24473347)
so how did u do to understand the "form" of the document such as table of contents and co.
Re:Apache Tika – a content analysis toolkit by nayuki on Monday September 14, 2020 @11:30AM (#24471326)
Re:Apache Tika – a content analysis toolkit by iav on Monday September 14, 2020 @12:36PM (#24472140)
I’ve found Tika’s PDF to HTML parser to be pretty good. My only complaint is that in a double spaced document where there is equal amount of space between paragraphs and normal lines, it labels every line as a separate paragraph.