News for hackers. Stuff that matters.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by kebman on Tuesday September 22, 2020 @10:39AM (#24554917)
I looked into coding PDFs once. Then I closed my MacBook (Pro) and went for a long walk into the ocean. I think I almost got to America, but then I turned and swam back again. Turnd out I had just fallen asleep and had a nightmare. I was actually just working with regular text files, and everything was fine.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by rvense on Tuesday September 22, 2020 @11:22AM (#24555544)
My favourite PDF fact is that it doesn't have to start at the beginning or end at the end of a file. Any sea of bytes that contains a PDF file is an acceptable PDF file...
Re:Embedded PDF viewer in Firefox 81 supports filling forms by TeMPOraL on Tuesday September 22, 2020 @11:39AM (#24555783)
Did anyone try to pluck out PDFs from /dev/urandom? How about from radiotelescope feed? Maybe the first evidence of extraterrestrial life will be some poor alien's tax form?
Re:Embedded PDF viewer in Firefox 81 supports filling forms by willis77 on Tuesday September 22, 2020 @11:56AM (#24556041)
The digits of pi contain every pdf that ever could and ever will exist.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by vcxy on Tuesday September 22, 2020 @12:44PM (#24556728)
"Find the earliest valid pdf in consecutive digits of pi"
Re:Embedded PDF viewer in Firefox 81 supports filling forms by paulmd on Tuesday September 22, 2020 @01:15PM (#24557199)
I mean, the answer is trivially zero, there exists a PDF-like structure somewhere in Pi, and the offset of that doesn't have to be zero, it can start or end anywhere. So the range [0, N] is a valid PDF.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by infogulch on Tuesday September 22, 2020 @03:00PM (#24558457)
"Find the last byte of the first valid PDF in the binary digits of Pi"
Re:Embedded PDF viewer in Firefox 81 supports filling forms by paulmd on Wednesday September 23, 2020 @05:36PM (#24572358)
Since the PDF also doesn't have to be end-aligned, the answer is trivially [0, infinity].

The first place a valid PDF could be ended, perhaps.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by infogulch on Wednesday September 23, 2020 @10:20PM (#24574663)
A pdf at [0, N] sorts before the one at [0, N+1], by "first valid pdf".
Re:Embedded PDF viewer in Firefox 81 supports filling forms by paulmd on Friday September 25, 2020 @12:17PM (#24591384)
No, both start at 0. Also, [0, infinity] and [0, infinity+1] are the same thing.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by infogulch on Tuesday September 29, 2020 @02:11AM (#24624615)
Re:Embedded PDF viewer in Firefox 81 supports filling forms by paulmd on Tuesday September 29, 2020 @01:24PM (#24630512)
your example fails to satisfy the invariant. 11 is less than infinity.

you're just pasting random python snippits at me now. It's time to move on.

again, just to summarize: PDF files do not have to be zero aligned, and they do not have to be end aligned. Therefore the answer to the question "what is the first segment of Pi that is a valid PDF file" is trivially (0,infinity). That is a correct statement. The non-greedy (in the regex sense) answer to that question will be different, however.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by infogulch on Tuesday October 06, 2020 @02:06PM (#24700645)
Why is this so hard? If the tuple (0,10) represents the range of a valid pdf, then the next tuple (0,11) is also a valid pdf. Or any after it up to and including (0,infinity).

Note the word "next", implying that (0,10) sorts before (0,11); you even say it yourself "11 is less than infinity". Where I'm from "first" and "less" are related (the first element in a unique sorted list is defined to be less than all other elements). So if there is any valid pdf in pi that can be identified by the range tuple (0,N), then the first valid pdf must occur before N -> infinity. Therefore (0,infinity) can never be the first valid pdf, even though it may be a valid pdf.

Maybe a picture would help:

    Potential pdf file ranges in pi: [(0,0),(0,1),(0,2),(0,3),(0,4),...,(0,N-1),(0,N),(0,N+1),(0,N+2),...,(0,infinity)]
    Is it a valid pdf?                 no    no    no    no    no  (no)  no      yes   yes     yes   (yes) yes
    Which one is first?                                                          ^^^
I thought linking to a python script that shows the order comparison of a tuple (0,N) as less than the tuple (0,N+1) would clearly demonstrate this, but it appears to have failed to communicate that to you. We don't need non-greedy regex rules to do a less than comparison.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by drevil-v2 on Tuesday September 22, 2020 @03:41PM (#24558914)
Please don't give them any ideas.. the whiteboard interview coding tests are hard enough as it is
Re:Embedded PDF viewer in Firefox 81 supports filling forms by gedy on Tuesday September 22, 2020 @07:05PM (#24561052)
How else will we weed out the fakers and people coasting for 10 years? Our CRUD SaaS app needs top people.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by ehsankia on Tuesday September 22, 2020 @07:01PM (#24561020)
Doesn't sound like that hard of a question, given you are provided the structure of the PDF header. I guess it really comes down to substring search.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by btown on Tuesday September 22, 2020 @01:06PM (#24557056)
Imagine if it was a PDF that simply rendered the number 42.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by ddalex on Tuesday September 22, 2020 @06:42PM (#24560868)
If that happens we know for a fact that we are in a simulation
Re:Embedded PDF viewer in Firefox 81 supports filling forms by mtzet on Tuesday September 22, 2020 @12:01PM (#24556116)
Well maybe. We don't know if pi is a normal number.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by poizan42 on Wednesday September 23, 2020 @10:20AM (#24567368)
Actually it only needs to be a disjunctive (or rich) number which is a weaker condition.

We don't know whether pi is that either for any integer base.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by SteveGoob on Tuesday September 22, 2020 @12:40PM (#24556671)
> We don't know if pi is a normal number.

Sure we do. There are plenty of proofs out there that pi is an irrational number.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by andreareina on Tuesday September 22, 2020 @12:43PM (#24556713)
Irrational does not imply normal. For example, 1.01001000100001... is irrational but it's certainly not normal.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by x3c on Tuesday September 22, 2020 @02:54PM (#24558395)
Technically, 1.01001000100001... can be normal depending on what ... stands for. :)
Re:Embedded PDF viewer in Firefox 81 supports filling forms by bscphil on Tuesday September 22, 2020 @06:08PM (#24560547)
Well, obviously. But presumably the ... is meant to imply that this is the summation of 1/(10^(x(x+3)/2)).
Re:Embedded PDF viewer in Firefox 81 supports filling forms by coddle-hark on Tuesday September 22, 2020 @03:55PM (#24559075)
Or what 1 or 0 or . stands for.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by landryl on Tuesday September 22, 2020 @03:27PM (#24558732)
Actually I'd argue the example you provided is normal, as long as you authorise a particular encoding where every number n you're looking for is encoded as a string of n zeros.

It's then trivial to see that every number you can think of is encoded in there, and therefore any data, piece of music or movie that ever existed.

(I'm not sure we're allowed to fiddle with the encoding, but since we allow ourselves to represent a piece of music into a number, we're already talking about encoding anyway, so it doesn't seem like cheating to me...)

Re:Embedded PDF viewer in Firefox 81 supports filling forms by enedil on Tuesday September 22, 2020 @03:36PM (#24558860)
Normality of a number is with respect to number bases, so your trick with encoding is invalid. Otherwise, every computable number could be considered normal - take an algorithm for generating of it, supply a random string (this is the encoding), disregard the random string, and you have a perfectly valid normal representation of your number. So it is cheating.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by NieDzejkob on Wednesday September 23, 2020 @07:41AM (#24565800)
I agree that normality is a specific formalized concept, but you could always require that an encoding function like this is injective.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by dheera on Tuesday September 22, 2020 @04:29PM (#24559493)
Encoding doesn't count. Normality is a very specific mathematical concept: https://en.wikipedia.org/wiki/Normal_number

Also, 1.01001000100001... is a good example of a number that is both irrational and transcendental but not normal.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by jimktrains2 on Tuesday September 22, 2020 @02:50PM (#24558354)
Normal in this sense means that all the frequency of all digits approaches a uniform distribution as the length of the sample increases towards infinity. Basically if we could see "all of" π and count all the 0s, 1s 2s, 3s, &c to 9 all the counts would be equal.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by gugagore on Tuesday September 22, 2020 @03:26PM (#24558721)
That on its own can't be right, because 0.12345678901234.....

According to wikipédia, you gave a definition for "simply normal", and for normal numbers the distribution of any sequence of digits is uniform. So 00, 01, ..., 99 each occur uniformally too.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by enedil on Tuesday September 22, 2020 @03:37PM (#24558870)
Moreover you need to consider it with regards to all other bases than 10 too.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by gorgoiler on Wednesday September 23, 2020 @12:23AM (#24562951)
Is this correct, mathematically?

I understand the point that PI contains every possible piece of information, theoretically.

However, the chance of finding a given string in PI depends on the string’s length. The longer the string, the more the probability tends to 0.

The paradox therefore is that PI contains every PDF, but you will never find them, so in what sense does it really contain them at all?

Re:Embedded PDF viewer in Firefox 81 supports filling forms by lsiebert on Wednesday September 23, 2020 @01:09AM (#24563203)
No, all strings theoretically exist in 𝛑 given enough digits, so longer strings don't reduce probability of existence, they just mean that it will take more digits to find them.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by tsbertalan on Wednesday September 23, 2020 @01:31AM (#24563318)
See Borell-Cantelli lemma.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by gorgoiler on Thursday September 24, 2020 @11:22PM (#24586270)
I looked this up but I’m not sure I grasp your point.

Are you saying that:

- given a long string, we might ask “can this string be found in PI?”

- the probability of finding a long string in PI is infinitely small

- the number of possible strings in PI is infinitely large

- it’s not possible to decide if the answer is yes or no?

Re:Embedded PDF viewer in Firefox 81 supports filling forms by rezeroed on Wednesday September 23, 2020 @10:58AM (#24567869)
If a tree falls in the forest and no one is around to hear it fall. Or a modern take, if a disease has no symptoms is it really a disease.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by nroets on Tuesday September 22, 2020 @01:05PM (#24557042)
<citation needed>

Including a PDF that generates the digits of pi

Re:Embedded PDF viewer in Firefox 81 supports filling forms by adtac on Tuesday September 22, 2020 @11:40PM (#24562704)
actually, if you find the citation, let me know, you might be in for an award
Re:Embedded PDF viewer in Firefox 81 supports filling forms by egocodedinsol on Tuesday September 22, 2020 @10:12PM (#24562240)
I'm not sure that's necessarily true. It is true (at least with a non-constructive proof) that if you pick a 'random' real number then it contains all possible PDFs with probability one ( or that the set of numbers for which this is not true has lebesgue measure zero). But I'm not sure it's known that pi has this property.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by dheera on Tuesday September 22, 2020 @04:21PM (#24559383)
Pi is thought to be normal but it hasn't been proven yet, so we can't say that for sure, but it's likely true.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by ducktective on Tuesday September 22, 2020 @12:54PM (#24556886)
I don't think that is a proven fact.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by rbonvall on Tuesday September 22, 2020 @05:27PM (#24560149)
Since a PDF can begin with non-PDF content, then pi itself is a valid PDF file.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by cgb223 on Tuesday September 22, 2020 @01:45PM (#24557594)
My favorite pdf fact is that the security flags for things like copy protection and passwords are on the viewer to implement so you can just turn them off and all the security is gone
Re:Embedded PDF viewer in Firefox 81 supports filling forms by BHSPitMonkey on Tuesday September 22, 2020 @05:53PM (#24560398)
Debian actually goes out of their way to patch those checks out in their PDF-related packages as part of their stance against DRM, like this example with "pdftk":

https://sources.debian.org/patches/pdftk/2.02-4/drm_fix/

Re:Embedded PDF viewer in Firefox 81 supports filling forms by r00fus on Tuesday September 22, 2020 @03:20PM (#24558646)
This is not entirely true, you can encrypt PDFs [1] since v1.3 of the spec but the cypher is often so weak (RC4 until v1.6) they can be bruteforced in reasonable amounts of time.

[1] https://www.pdflib.com/pdf-knowledge-base/pdf-password-secur...

Re:Embedded PDF viewer in Firefox 81 supports filling forms by Thorrez on Tuesday September 22, 2020 @10:55PM (#24562466)
You can encrypt them to completely prevent them from being opened. But cgb223 wasn't talking about that, cgb223 was talking about the ability to open them but not copy text, or not print.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by benmmurphy on Wednesday September 23, 2020 @07:18AM (#24565592)
You can make the text uncopyable by using non-standard font indexing. The reader will be able to copy the text but it will be gobbly-gook. It forces the user to OCR the PDF or reverse the font mapping.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by hnick on Wednesday September 23, 2020 @11:21PM (#24574926)
And ain't that a treat when clients send us PDFs to sort, print, and mail, but the address extraction fails completely.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by toomanybeersies on Tuesday September 22, 2020 @02:51PM (#24558360)
You can also circumvent copy protection on PDFs by taking a screenshot, or taking a photo of the screen with your phone.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by Drdrdrq on Tuesday September 22, 2020 @02:33PM (#24558168)
My somewhat less favorite pdf fact is that if you do that, you are still breaking protection, legally speaking.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by maxerickson on Tuesday September 22, 2020 @02:40PM (#24558241)
Seems to be a reasonable analogy with trespass, where you are violating the law when you cross an invisible line. The need for marking the line varies considerably.

And even places with strong roaming rights tend place limits on well marked land.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by smegger001 on Tuesday September 22, 2020 @05:54PM (#24560407)
So what if you open it in a postscript viewer instead of a PDF viewer? Because they are compatible formats except for some edge cases like security flags.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by mkl on Tuesday September 22, 2020 @06:17PM (#24560624)
Postscript and PDF are definitely not compatible formats. The drawing model is similar, but the structure and code are completely different.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by sp332 on Tuesday September 22, 2020 @11:56AM (#24556044)
On the other hand, this allows for some incredible polyglot files, like some of the tricks with PoC||GtfO issues where the file is a readable PDF but also a game cartridge and also a zip file with the proof-of-concept code in the issue. And the front cover has the MD5 hash of the whole file printed on it... but that's another trick entirely!
Re:Embedded PDF viewer in Firefox 81 supports filling forms by rvense on Tuesday September 22, 2020 @12:21PM (#24556385)
Yeah, next time I need a CV it'll be a single-file Ruby web server and PDF that's also an archive of its own sources.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by toomanybeersies on Tuesday September 22, 2020 @02:57PM (#24558427)
I'm currently looking for a job as a Rails Developer. I might just do that.

Probably won't send it to any recruiters, but it will be a funny anecdote for interviews.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by maddyboo on Tuesday September 22, 2020 @12:53PM (#24556866)
Can a PDF file contain a PDF file, and if so can that PDF file contain a PDF file?
Re:Embedded PDF viewer in Firefox 81 supports filling forms by betatim on Tuesday September 22, 2020 @01:53PM (#24557681)
Yes. Because the PDF standard specifies a mechanism that lets you "attach" files to a PDF :)
Re:Embedded PDF viewer in Firefox 81 supports filling forms by brailsafe on Tuesday September 22, 2020 @08:51PM (#24561771)
I worked for PDFTron on their WebViewer product earlier this year, and primarily spent time implementing this feature in JS. Understanding the spec on this was tricky, because standard PDF viewers need to be able to uncompress the stuff you jam in there. It kind of blew my mind that you can literally jam any arbitrary file into a PDF.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by layoutIfNeeded on Tuesday September 22, 2020 @01:56PM (#24557711)
Re:Embedded PDF viewer in Firefox 81 supports filling forms by folmar on Thursday September 24, 2020 @07:13PM (#24584770)
My stupid bank sends encrypted attachments as an encrypted PDF with HTML file attached.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by brailsafe on Tuesday September 22, 2020 @08:43PM (#24561714)
Yes.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by chuinard on Tuesday September 22, 2020 @08:35PM (#24561646)
I never understood that Google security blog post on how they could make 2 different PDFs with different content have the same SHA but now that you mention you can stuff bytes in a file unrelated to the PDF, it makes sense...
Re:Embedded PDF viewer in Firefox 81 supports filling forms by nostoc on Tuesday September 22, 2020 @02:49PM (#24558346)
It'll depend on the pdf reader you're using, but I'm pretty sure the PDF header needs to start in the first 1K of the file.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by mkl on Tuesday September 22, 2020 @06:20PM (#24560664)
Some readers won't need a header at all, I think. Near the end (usually!) of the file there's an index of objects (page data etc.) with byte offsets, which can point to anywhere in the file.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by agumonkey on Tuesday September 22, 2020 @05:03PM (#24559860)
I can never find the PDF hack talk where author explains all 100 ways to embed things in pdf or pdf into things
Re:Embedded PDF viewer in Firefox 81 supports filling forms by wffurr on Tuesday September 22, 2020 @06:02PM (#24560488)
It's hidden in a PDF in the digits of Pi.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by alexott on Tuesday September 22, 2020 @04:35PM (#24559571)
You can imagine the pain when you need to reliably detect PDF mime type on web proxy, or something like...
Re:Embedded PDF viewer in Firefox 81 supports filling forms by belval on Tuesday September 22, 2020 @10:44AM (#24554981)
As someone still working with PDF processing, I can confirm that it doesn't get easier.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by mettamage on Tuesday September 22, 2020 @11:47AM (#24555920)
What's your favorite PDF feature that causes a brain meltdown?

I've read a few comments on HN how PDF is, well, not developer-friendly. If people are interested in providing some more examples here, I'd be curious to know!

Re:Embedded PDF viewer in Firefox 81 supports filling forms by pierrebai on Tuesday September 22, 2020 @12:03PM (#24556141)
In the early 2000 I coded a PDF library for an industrial printer suite. (Print, proof, impositions)

I personally think the structural PDF format is a really great format. It's entirely ASCII-based, a pure text format, yet it can embed arbitrary binary data and compress that data. The actual structure is simple and support just enough functionality, like a tree of object, dictionaries and arrays, unicode strings, date formats, etc.

I think if you limit yourself to pure structural PDF woulde have been a great format to standardize upon, much better than JSON or XML. It';s richer than JSON, simpler and saner than XML. Again, it's top-notch ability to embed binary is great. It has other great characteristics, for example you can update anything just by appending.

The ugly bits are in the "semantic" PDF: the page descriptions, media, etc. Even then, the early version of PDF were nice, mainly just simplified Postscript.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by hnick on Wednesday September 23, 2020 @11:29PM (#24574966)
I'm of a similar opinion but I'll say the format is quite good, but the many and varied implementations are often not.

A common case is clients who use utilities that generate single customer documents then merge them into a bigger file for bulk print and mail (bills and statements, not identical copy). Without fail that results in thousands of similar but different subset fonts whereupon most printers I've encountered eventually fail due to memory issues.

Typically this leads to a discussion about "I can open it on my computer fine" and bending over backwards to find a workaround. Merging and consolidating these fonts doesn't seem to be a simple task, although some tools claim to work some of the time.

Something that scopes object resources for disposal could be nice in the PDF spec (maybe it exists), but something like a LRU caching mechanism on the printer would potentially resolve this too.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by izacus on Tuesday September 22, 2020 @01:17PM (#24557237)
Being able to do SQL queries to remote servers, upload form contents directly to a server, embedded 3D models and being able to have a fully featured page embedded Tetris game due to support for JS.

Having said that (and worked on a commercial PDF library), despite all the cruft that came with age, it's a well built format that survived the test of time with good reasons.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by dwd on Wednesday September 23, 2020 @12:02AM (#24562829)
Nice

I worked on a Pdf with inbuilt tracking solution that updated the form layout using ActionScript based on the workflow status and the role of the user (ie the line manager had a different group of fields in the form to complete like their signature while viewing what the initial requestor had entered) Lots of callbacks to the server saving in progress data and updating the status of who had the form, who was next based on department and emailing it to that person if it passed validation.

An initial fun discovery was that you could force the form to download and replace itself with the latest version even if they had just opened some old file they had on their pc.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by modoc on Tuesday September 22, 2020 @10:14PM (#24562245)
Can chat with you about PDFs? devon at digitalsanctuary dot com
Re:Embedded PDF viewer in Firefox 81 supports filling forms by jahewson on Tuesday September 22, 2020 @11:58AM (#24556081)
Acrobat can read fantastically corrupt PDF files none of which are covered by the spec. The endless surprises induce a special kind of madness.

Streams just suddenly end? That’s ok. Totally corrupt xref tables? Ok. Incorrect image headers? Ok. Unrecognisably mangled Type1 font formats? Fine!

Re:Embedded PDF viewer in Firefox 81 supports filling forms by belval on Tuesday September 22, 2020 @12:33PM (#24556562)
That's great because it creates client expectations regarding what my PDF application should support. Implementing the spec is not good enough, you have to do what PDFium or Adobe do.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by fgonzag on Tuesday September 22, 2020 @08:50PM (#24561766)
On the other hand, if they never supported those broken PDFs to begin with, we wouldn't have them in the wild and wouldn't have to deal with them.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by folmar on Thursday September 24, 2020 @07:17PM (#24584797)
20 years ago I were more surprised when I got a PDF with correct xref table than with broken one.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by nl on Wednesday September 23, 2020 @01:04AM (#24563173)
I think one of the biggest pain points that developers hit that hasn't been mentioned here is content extraction.

A lot of the time developers want access to the text inside PDFs. Unlike HTML or formats like MS Word (XML or old binary format) getting "text" isn't really possible.

Most "document" formats have the concept of words or strings: a set of characters separated by whitespace. PDF isn't a "document" format in that sense - it's a page description language. Instead of strings of text you have character glyphs positioned at a particular location.

If you want to "read" the text, you have to work out the orientation (which can change throughout the page - think of table header alignment), and use some kind of heuristic to guess the word spacing based on the font and character spacing.

There's also this whole thing with clipping, where some text can be hidden behind other objects (or off page) so you have to try to deal with that.

There's lots of libraries that try to do this for you, but there are lots because none get it right 100% of the time...

Re:Embedded PDF viewer in Firefox 81 supports filling forms by belval on Tuesday September 22, 2020 @12:31PM (#24556541)
- Remapping font tables to different characters for to reduce code usage.

- Clipping path logic, you can write text outside of it, which makes it effectively invisible yet it will show up if you try to extract the text.

- Anything regarding the graphicstate stack, it's a pain to debug.

- Extracting content from AcroForm/JS "XFA" forms

PDF is great format for printing, it's just a pain for pretty much everything else.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by aidos on Tuesday September 22, 2020 @12:38PM (#24556631)
This is also my list. Except for the forms, that's one I don't have to deal with.

My other one is the use of multiple subset fonts that are actually the same font with a different subset of glyphs that you want to merge back together.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by arh68 on Tuesday September 22, 2020 @07:43PM (#24561354)
"Identical" PDFs are not necessarily byte-identical; you can't just check equivalence through checksums. (AFAIK, it's been a while, feel free to correct) I don't remember a good way to normalize/disambiguate, at least I never attacked the problem long enough to have learned.

EDIT: oh yeah, I'm pretty sure it contains Mail, just like Zawinski said

Re:Embedded PDF viewer in Firefox 81 supports filling forms by amelius on Tuesday September 22, 2020 @11:55AM (#24556017)
At least it's probably better than MS-Word's internal format ... (?)
Re:Embedded PDF viewer in Firefox 81 supports filling forms by liversage on Tuesday September 22, 2020 @12:14PM (#24556282)
Microsoft Word stores XML documents inside a zip archive. There is a detailed specification of the format available: https://docs.microsoft.com/en-us/openspecs/office_standards/...
Re:Embedded PDF viewer in Firefox 81 supports filling forms by xaldir on Tuesday September 22, 2020 @12:32PM (#24556555)
I think he was talking about the classic .doc format which was a clusterfuck and not the open XML.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by akie on Tuesday September 22, 2020 @02:14PM (#24557958)
If I remember correctly the XML format was just an XML-encoded version of the binary counterpart. Including all or most of the bugs and weird hacks.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by amaccuish on Tuesday September 22, 2020 @02:35PM (#24558192)
with the previous format being essentially a memory dump, i'd say that's progress
Re:Embedded PDF viewer in Firefox 81 supports filling forms by alexott on Tuesday September 22, 2020 @04:39PM (#24559609)
That’s correct - I worked with MS team that documented old formats, and they said that sometimes they don’t have people left who knew what specific struct was intended for - although that was mostly for people PowerPoint and Visio, excel and word was better documented
Re:Embedded PDF viewer in Firefox 81 supports filling forms by tinus_hn on Thursday September 24, 2020 @10:40AM (#24579029)
Actually Excel was the only one that had official, freely available documentation for the old (now legacy) file format.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by patrec on Tuesday September 22, 2020 @05:07PM (#24559912)
You don't remember correctly. Word's docx format is far more intelligent than openoffice ODT, despite propaganda to the contrary. With one exception: word's zip files don't have a convenient magic header. The way it works with ODT, and a bunch of other formats is that you put an uncompressed identifier file (`mimetype`) as the first entry inside your zipfile. At byte 30 (of your zipfile) you then get `mimetype$THE_MIMEMETYPE`. This is a nice trick and works for any zip-based format. Sadly, docx does not do that so you have to go by file extension or look at (more of) the contents of the zipfile.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by yoz-y on Thursday September 24, 2020 @10:32AM (#24578941)
IIRC the original doc (and xls) formats were unwieldy mainly because of performance requirements. In order to save and load fast they were basically a bunch of binary dumped structs.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by DavidPeiffer on Tuesday September 22, 2020 @12:43PM (#24556715)
I'm not saying it's beautiful, but isn't Ms-Word's internal format basically a series of XML files that are zipped up?

The old .Doc and .xls files were a bad format, but my understanding is that since Office 2007 the format is generally much better.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by alexott on Tuesday September 22, 2020 @04:41PM (#24559644)
Ms office files prior to office 2007 were mostly memory dumps of specific components, wrapped into composite files aka OLE2 storage - their content varied depending on office versions and often locale
Re:Embedded PDF viewer in Firefox 81 supports filling forms by Polylactic_acid on Wednesday September 23, 2020 @12:03AM (#24562836)
Its a wonder alternative editors ever supported the format..
Re:Embedded PDF viewer in Firefox 81 supports filling forms by jhoechtl on Tuesday September 22, 2020 @01:32PM (#24557411)
To be fair the old doc format was conceived in DOS era and memory efficiency was a primer back in the days.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by smegger001 on Tuesday September 22, 2020 @06:06PM (#24560538)
and if they hadn't waited so long to update to a sane file format no one would complain but they waited until 2007 to fix the format long after the dos era memory excuse had long ceased to be an issue. even then they only did it to allow them to shoe horn their format in as the iso standard after one was already selected bribing there way through the process.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by divbzero on Tuesday September 22, 2020 @11:05AM (#24555294)
I had to handle action buttons for a PDF once. I swam out from America a long long ways before turning back. Might have spotted you middle of the ocean.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by meshaneian on Tuesday September 22, 2020 @02:28PM (#24558124)
Legit username warns of one PDF peril.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by moultano on Tuesday September 22, 2020 @02:20PM (#24558038)
It's kinda crazy that this is the format we've standardized on to carry all of the output of academia into the future.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by larrybud on Thursday September 24, 2020 @06:57AM (#24577326)
Actually, it’s not. The archiving standard is PDF/A, which is (as I understand it) much more structured and standardized.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by mjcohen on Tuesday September 22, 2020 @05:47PM (#24560333)
A lot of the input is LaTeX, so that's ok.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by MayeulC on Tuesday September 22, 2020 @05:48PM (#24560349)
And arxiv asks for the original latex source when submitting.

Well, at least, pdf is probably better than printed paper for that purpose.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by phpdave11 on Tuesday September 22, 2020 @03:45PM (#24558951)
It's really not that difficult if you read and understand the PDF specification. As a learning exercise, I created a simple PDF generator library that creates ASCII PDF documents (you can open them in Notepad) and includes comments about what each drawing instruction does.

https://github.com/phpdave11/davepdf

Re:Embedded PDF viewer in Firefox 81 supports filling forms by recursive on Tuesday September 22, 2020 @05:04PM (#24559874)
I'm sure generating PDFs is much easier than reading them, such that "it just works" with any kind of PDF.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by phpdave11 on Tuesday September 22, 2020 @06:15PM (#24560611)
Reading them is also easy. I wrote a library that reads PDFs and imports page(s) from an existing PDF into a new PDF as a Form XObject.

https://github.com/phpdave11/gofpdi

Re:Embedded PDF viewer in Firefox 81 supports filling forms by bjoli on Wednesday September 23, 2020 @02:29AM (#24563649)
I had a look through the XPS standard and had a similar feeling. I complained to a friend that had been involved in one of the bigger pdf libraries. He then made me compare it to his version of the pdf 2.0 (iirc) standard.

That is truly nightmare material. Especially considering a non-trivial percentage of pdfs circulating are non-conformant and people still expect them to render...

Re:Embedded PDF viewer in Firefox 81 supports filling forms by emiliosic on Tuesday September 22, 2020 @11:40PM (#24562701)
Years ago, I worked on a project that required generating PDF invoices. Used the FPDF library and I was shocked how small the files are (including a properly sized logo) when generated compared to most other PDFs 'rendered' from word or print processors.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by saagarjha on Wednesday September 23, 2020 @08:00PM (#24573750)
I just had to do tech support for someone whose 3 MB PowerPoint of text and some shapes became a 70 MB PDF that they couldn’t send through email anymore :/
Re:Embedded PDF viewer in Firefox 81 supports filling forms by ajsnigrutin on Wednesday September 23, 2020 @01:17AM (#24563243)
The last time i wanted to do that, i noped the fuck out, generated latex, and pdflatex-ed the thing, to get the pdf (we're talking generic reports, text, table, graph, email to customer on a schedule).
Re:Embedded PDF viewer in Firefox 81 supports filling forms by thehappypm on Tuesday September 22, 2020 @11:17PM (#24562573)
I am actually interested in doing pure JS pdf processing. All of the web interfaces for PDF processing are server side — which means it’s tough to process large files. The dream is a purely JavaScript solution that never leaves the local computer. I’ve got a few client-side success stories that do fairly significant image generation through the canvas. So far PDF seems reasonably manageable through manipulating the text, but it’s not the best format.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by anoopelias on Tuesday September 22, 2020 @11:28PM (#24562637)
> The dream is a purely JavaScript solution that never leaves the local computer.

Mozilla's pdfjs[1] project is a pure HTML/JavaScript solution for PDF rendering. This is the same code that ships in Firefox browser as well. This is standalone, AFAIK, it doesn't talk to a mothership.

[1]https://mozilla.github.io/pdf.js/

Re:Embedded PDF viewer in Firefox 81 supports filling forms by thehappypm on Friday September 25, 2020 @11:16PM (#24596149)
It’s not quite what I’m looking for—more of a viewer.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by schoolornot on Tuesday September 22, 2020 @03:20PM (#24558653)
I thought I found a nifty trick by using OpenOffice to create a form with a pre-filled value. I decoded the PDF using pdktk or one of the free tools, and then modified the value. Nope, that caused some kind of cascading/checksum error.

Ended up just making the app generate HTML before calling wkhtmltopdf.

The PDF spec is insane! But like all things what you get out of Word/OpenOffice is 100x more complex than if you wrote it yourself, which is indeed doable.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by 1vuio0pswjnm7 on Tuesday September 22, 2020 @07:07PM (#24561072)
Re:Embedded PDF viewer in Firefox 81 supports filling forms by gorgoiler on Tuesday September 22, 2020 @01:24PM (#24557316)
Only Forward

...a wondeful novel along these lines. They only get shot and kidnapped though. Nothing so bad as PDFs.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by godelmachine on Wednesday September 23, 2020 @01:21AM (#24563265)
I didn’t get it. Are you implying that coding PDF is an onerous task?
Re:Embedded PDF viewer in Firefox 81 supports filling forms by maushu on Wednesday September 23, 2020 @01:27AM (#24563303)
Check out the specs file for PDF and you will understand.

I will be extremely surprised if anyone (besides Adobe) has implemented 100% of it.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by saagarjha on Wednesday September 23, 2020 @08:01PM (#24573758)
I’d be surprised if Adobe has implemented 100% of it. With a format this complex, there’s bound to be discrepancies between the spec and the code they have.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by enriquto on Wednesday September 23, 2020 @02:16AM (#24563571)
Reading PDF files is certainly a nightmare. But you can easily produce a valid and simple pdf file just by printf-ing whatever needs to be printf-ed. There's an ugly header and the rest is essentially your text.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by Someone on Tuesday September 22, 2020 @12:05PM (#24556170)
OK, they added various ways of data compression, but PDF is, basically, a text-based format.

As far as I know, any PDF can be losslessly converted to an equivalent PDF that can be edited in any text editor, even Notepad. And yes, you could fill in the forms there, too (if you were stubborn enough)

Re:Embedded PDF viewer in Firefox 81 supports filling forms by roflc0ptic on Tuesday September 22, 2020 @12:15PM (#24556304)
It sounds like you either know a lot more than me or a lot less than me. The PDFs I've dealt with don't store text as strings, they store it as individual characters. This left me having to write a heuristic based algorithm to group the characters into words, words into lines, lines into paragraphs, paragraphs into columns.

Again, as far as I know, there are no heuristics good enough to get that right for all values of PDF.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by belval on Tuesday September 22, 2020 @12:24PM (#24556438)
He knows a lot less than you probably because there is absolutely no requirements for PDFs to be in text format and most aren't. The "text" he is editing could render to completely different characters depending on how the PDF document was created.

The default MacOS PDF printer will actually remap the font cmap making born-digital PDFs where the "text" is something else entirely (say "$" maps to "a").

Re:Embedded PDF viewer in Firefox 81 supports filling forms by yjftsjthsd-h on Tuesday September 22, 2020 @03:39PM (#24558888)
> The default MacOS PDF printer will actually remap the font cmap making born-digital PDFs where the "text" is something else entirely (say "$" maps to "a").

What? Why!? I've heard of doing that as a form of DRM, but I can't imagine Darwin defaulting to doing that.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by belval on Tuesday September 22, 2020 @04:18PM (#24559349)
I never dug deeper into it, so I am not aware of why it does that or if it's a specific version or whatnot, but take a PDF from which you can extract the text (with pdftotext/pdfbox for example). Open it in the document viewer and "print" it to PDF. If you extract the text again it is not readable anymore.

This wouldn't be an issue if it was a conscious choice, but when I parsed a lot of born-digital PDFs we ended up with a lot that were like that from various source. Try explaining that...

Re:Embedded PDF viewer in Firefox 81 supports filling forms by colejohnson66 on Tuesday September 22, 2020 @06:22PM (#24560686)
Could it be “compacting” the fonts? So if U+0000 to U+0007F aren’t used at all, remove those glyphs and set U+0000’s glyph to be what was U+0080? Yes, I know NULL doesn’t have a glyph, but I hope that gets the idea across.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by belval on Tuesday September 22, 2020 @07:09PM (#24561081)
That would be my personal guess yes, as there are other ways to "protect" PDF text like curve-based rendering.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by Someone on Wednesday September 23, 2020 @06:16AM (#24565198)
Where did I say PDFs have to be in text format? I said every PDF has an equivalent text-only representation. The format, like postscript and eps started as text-only. Compression, making the files binary was added as an afterthought, to make files smaller (much smaller). If it were binary from the start, its designers would not have made the table of contents at the end waste bytes by writing offsets in ascii.

See http://blog.idrsolutions.com/?s=%22Make+your+own+PDF+file for more info on how to write PDF by hand (also shows why I said you have to be stubborn to do that)

Re:Embedded PDF viewer in Firefox 81 supports filling forms by belval on Wednesday September 23, 2020 @12:19PM (#24568711)
"OK, they added various ways of data compression, but PDF is, basically, a text-based format."

You are trying to move the goal post here. The above statement is simply untrue, PDF is not a text based format and it's that simple.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by mkl on Tuesday September 22, 2020 @06:34PM (#24560789)
More. Go here and download the PDF spec.: https://www.adobe.com/devnet/pdf/pdf_reference.html

Look at Chapter 3, Syntax. The code is all text based. We are not talking about the visible characters in a PDF viewer, but the code of the PDF file itself.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by roflc0ptic on Wednesday September 23, 2020 @08:20AM (#24566111)
Oh, true. I misread, and also didn't know that. Thanks for the info!
Re:Embedded PDF viewer in Firefox 81 supports filling forms by hnick on Wednesday September 23, 2020 @11:39PM (#24575017)
Going by your original premise though, just so you know the reference does indeed have examples of text as strings e.g.

BT /F13 12 Tf 288 720 Td (ABC) Tj ET

This can be extended to include spaces so you can essentially mark up entire lines of text at one time. What it can't do is cohesive paragraphs and flow/wrap, you need to use the relative positions to work out what text is in one block (and usually I'd defer to something like pdftotext for simple cases).

Laying out individual characters is common though. It's probably due to kerning concerns.

Re:Embedded PDF viewer in Firefox 81 supports filling forms by quickthrowman on Tuesday September 22, 2020 @01:07PM (#24557069)
That’s news to me and Bluebeam’s PDF search feature! It turns out you can make PDFs (This usually happens with architectural drawings) that are comprised purely of images that are not searchable, and therefore you are wrong.

I silently thank every architect that provides searchable PDFs, it makes my job way easier

Re:Embedded PDF viewer in Firefox 81 supports filling forms by mkl on Tuesday September 22, 2020 @06:31PM (#24560768)
GP is right. The code that makes up a PDF is text-based. Those images can be encoded in the PDF file using the ASCIIHexDecode filter, i.e. as editable ASCII text code.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by yunohn on Wednesday September 23, 2020 @01:07AM (#24563185)
GP uses the phrases "losslessly convert to text" and "fill in the forms". I think you've misunderstood them, they're clearly talking about the display text, and not the code that comprises it.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by xfer on Tuesday September 22, 2020 @06:49PM (#24560918)
Yes just use a hex editor and every data is text-based.
Re:Embedded PDF viewer in Firefox 81 supports filling forms by mkl on Tuesday September 22, 2020 @08:20PM (#24561562)
Only the "binary" bits like fonts and bitmap images would need to be hex (and then only so Notepad doesn't mangle them). Everything else in an uncompressed PDF is already text. Go here and download the PDF spec.: https://www.adobe.com/devnet/pdf/pdf_reference.html

Look at Chapter 3, Syntax (and the rest of it, really).

I literally have scripts that use bash and sed, or Python, to modify PDFs by editing the text code. Doing it in Notepad is possible but tricky, as there's a table of object byte offsets near the end that it's easy to mess up by inserting a character.