News for hackers. Stuff that matters.
Re:What's so hard about PDF text extraction? by aidos on Tuesday March 03, 2020 @03:15PM (#22478326)
I’ve done a bunch of this work myself and while it’s a bit of a pain to do in general, you can make some reasonable attempts at getting something workable for your use cases.

PDFs are incredibly flexible. Text can be specified in a bunch of ways. Glyphs can be defined to the nth degree. Text sometimes isn’t text at all. There’s no layout engine and everything is absolutely positioned. Fonts in PDF’s are insane because they’re often subset so they only include the required glyphs and the characters are remapped back to 1, 2, 3 etc instead of usual ascii codes.

Re:What's so hard about PDF text extraction? by kbenson on Tuesday March 03, 2020 @03:29PM (#22478431)
> Fonts in PDF’s are insane because they’re often subset so they only include the required glyphs and the characters are remapped back to 1, 2, 3 etc instead of usual ascii codes.

I've actually seen obfuscation used in a PDF where they load in a custom font that changes the character mapping, so the text you get out of the PDF is gibberish, but the fonts displayed on rendering are correct (a simple character substitution cipher).

The important thing to remember whenever you think something should be simple, is that someone somewhere has a business need for it to be more complicated, so you'll likely have to deal with that complication at some point.

Re:What's so hard about PDF text extraction? by ccozan on Tuesday March 03, 2020 @03:45PM (#22478553)
That's why the only "reliable" way to extract text is to perform an OCR of the pdf rendering which is exactly what ABBYY is doing.
Re:What's so hard about PDF text extraction? by siftrics on Tuesday March 03, 2020 @03:52PM (#22478609)
I’m the founder of a startup that is doing this, as well. We strive to be as simple and easy as possible to use.

If you care to check us out: https://siftrics.com/

Re:What's so hard about PDF text extraction? by Defenestresque on Tuesday March 03, 2020 @04:51PM (#22479061)
Your website demo video is impressive and I can imagine there are many businesses that would save a lot of time and man-hours by incorporating a solution like this.

I've often thought about creating products like these but as a one-man operation I am daunted by the "getting customers" part of the endeavour. How do you get a product like this into the hands of people who make the decisions in a business? (For anyone, not just OP). PPC AdWords campaigns? Cold-calling? Networking your ass off? Pay someone? Basically, how does one solve the "discoverability problem"?

Re:What's so hard about PDF text extraction? by siftrics on Tuesday March 03, 2020 @05:40PM (#22479420)
Surprisingly, Hacker News has been our number one source of leads. We tried Google Ads and Reddit Ads, but the signup rate was literally three orders of magnitude lower than organic traffic from Hacker News and Reddit.
Re:What's so hard about PDF text extraction? by appleiigs on Tuesday March 03, 2020 @06:05PM (#22479591)
Is your product only on the cloud? My privacy/internet security team won't let me use products that save customer or vendor data on the cloud because you might get hacked. Only giants, like Microsoft, have been approved after an evaluation.
Re:What's so hard about PDF text extraction? by siftrics on Tuesday March 03, 2020 @07:37PM (#22480183)
More than half of our customers have asked to be able to skip our cloud and go directly to their database. We’re working on this right now. It’s scheduled to be released this week, so keep an eye open.

In the meantime, if you have any questions, feel free to send me an email at [email protected]. I’d love to hop on the phone or do a Zoom meeting or a Google Hangouts.

Re:What's so hard about PDF text extraction? by innomatics on Tuesday March 03, 2020 @04:11PM (#22478759)
Will check it out thanks, we had evaluated ABBYY but it didn't suit. Does your product do something like key value or table detection?
Re:What's so hard about PDF text extraction? by siftrics on Tuesday March 03, 2020 @04:13PM (#22478780)
We do table recognition and pride ourselves on being better at it than ABBYY. We can handle variable number of rows in a table and we take that into account when determining the position of other text on the page.

Feel free to email me at [email protected] with any questions. We can setup a phone call, zoom meeting, or google hangouts too, if you’d like.

Re:What's so hard about PDF text extraction? by 72deluxe on Tuesday March 03, 2020 @06:40PM (#22479829)
I like how your demo is clear and also in Linux! Surprising.

The page is clear and easy to understand, looks good. Well done.

Re:What's so hard about PDF text extraction? by siftrics on Tuesday March 03, 2020 @08:27PM (#22480405)
Thank you for the kind words!

I like to say that anyone with a good old ThinkPad and an internet connection can mint fortunes and build empires :-)

Re:What's so hard about PDF text extraction? by malikolivier on Tuesday March 03, 2020 @08:37PM (#22480460)
How is support for languages other than English?

I am especially thinking about Japanese. Our company could probably find good uses of such service if it had Japanese support.

Re:What's so hard about PDF text extraction? by siftrics on Tuesday March 03, 2020 @09:51PM (#22480803)
Yes, Japanese is supported! Almost all languages are supported.

If you have any questions or need help trying it out, please email me at [email protected]. We can hop on the phone, too, if you'd like.

Re:What's so hard about PDF text extraction? by tastyminerals on Wednesday March 04, 2020 @01:21AM (#22481607)
Gini GmbH performs document processing for almost all German banks and for many accounting companies. For banks it does realtime invoice photo processing -- OCR and extraction of amount, bank information, receiver etc. For accounting it extracts all kind of data from a PDF. Unfortunately, only for German language market. But here you go, ABBYY by far is not the only one. In fact ABBYY does only OCR and has some mediocre table detection. That's it.
Re:What's so hard about PDF text extraction? by zmix on Tuesday March 03, 2020 @07:40PM (#22480195)
I do not remember which of the two it was, but 'poppler' or 'pdfbox' (they may use the same backend) created great HTML output, with absolute positions. They also have an XML mode, which is easily transformed.

Of course, there is absolutely no semantics, just display.

Re:What's so hard about PDF text extraction? by aidos on Tuesday March 03, 2020 @03:46PM (#22478568)
That’s actually often just a consequence of the subsetting (I think). Believe it or not, you can often rebuild the cmaps using information in the pdf to fix the mapping and make the extraction work again.
Re:What's so hard about PDF text extraction? by kbenson on Tuesday March 03, 2020 @05:22PM (#22479291)
> That’s actually often just a consequence of the subsetting (I think).

I would believe that. It was a pretty poor obfuscation method as they go, if it was intended for that.

> Believe it or not, you can often rebuild the cmaps using information in the pdf to fix the mapping and make the extraction work again.

Oh, I did. That's the flip side of my second paragraph above. When there's a business need to work around complications or obfuscations, that will also happen. :)

Re:What's so hard about PDF text extraction? by speedplane on Wednesday March 04, 2020 @01:44AM (#22481689)
> PDFs are incredibly flexible. Text can be specified in a bunch of ways. Glyphs can be defined to the nth degree. Text sometimes isn’t text at all. There’s no layout engine and everything is absolutely positioned.

Can't stress this enough. The next time you open a multi-column PDF in adobe reader and it selects a set of lines or a paragraph in the way you would expect, know that there is a huge amount of technology going on behind the scenes trying to figure out the start and end of each line and paragraph.