It actually seems like a sane choice to me. PDF is good for rendering, but horrible for parsing. DOCX is a ZIP file with XML data. Maybe ODT or whatever would've been a better choice, I don't know what the format is like. But if you disregard the usual knee-jerk "but it's Microsoft!" reaction, it doesn't seem like a bad choice.

The Office Open XML file format is extremely complex, and takes up around 6,500 pages (compared to ~1000 for ODF). One thing you notice when reading the DOCX spec is that they designed it with the sole constraint that DOC files could easily be converted to DOCX. For example, you'll frequently see compatibility tags like "autoSpaceLikeWord95", "footnoteLayoutLikeWW8", "useWord2002TableStyleRules", and "lineWrapLikeWord6" that expose internal implementation details. Rather than creating a useful standard allowing all users to store their documents in a clean, portable way, Microsoft decided to make their standard faithfully reproduce all of the quirks and bugs of their legacy binary formats. It's so difficult to correctly implement the Office Open XML standard that even Microsoft took until Office 2013 to do so (the standard was approved in 2006).

So true. "It's XML, so it must easy to parse and manipulate" is such a naive, even misleading attitude. If what you do is take a byzantine, legacy-encrusted implementation and just serialise its data strucures to an XML representation, very little has been gained.

[edit: but I will grant that almost anything is better than attempting to parse useful content from PDF.]

I had a CTO who told me that "anytime someone says something will be simple because it uses XML, just repeat what they said and use the word text instead of XML. Then tell me if it sounds smart."

*Uses regular expressions to parse binary XML*

In fairness though this a really good point.

Microsoft "adopted" XML purely to assuage various regulators that their new format was totally gonna be open, interoperable and standardized. It is a pure fig leaf but judging by all the comments in here mentioning "XML", it seems to have worked.

I think they pretty much had to do that to preserve the formatting of existing documents for users who are force-upgraded by their employers to new Office versions. But it seems a scraper that just wants the information in the document can ignore almost all of those tags.

edit: ninja'd.

> "autoSpaceLikeWord95", "footnoteLayoutLikeWW8", "useWord2002TableStyleRules", and "lineWrapLikeWord6"

I expect that whatever tooling the USPTO uses can probably just ignore those things. They're extracting metadata, not actually rendering it.

Interesting! How do they compare feature-wise? I feel like there must be things each of them support that the other one doesn't, but I don't know how consequential they are.

I don't know how thorough or accurate it is, but Microsoft has a list.

https://support.microsoft.com/en-us/office/differences-betwe...

The list was only a few lines the last time I looked years ago, so maybe they're actually trying to make a complete list.

Was just about to post this. Unzipping DOCX and parsing XML is much easier than accurately processing PDF submissions.

Why not make yet another XML standard for patents and generate PDFs from it?

Or just use Docbook and the standard, open pipelines for publication to print and web.

I was saying it half-sarcastically because of how many "universal" XML standards are out there and how useless they are.

I wrote DDEX parsers in the past and I wish it never existed...

OCR has come a long way, so much that visually interpreting a PDF is about as error-prom as parsing XML output from Microsoft in non-microsoft software.

Try extracting tabular data from a PDF! With XML it's trivial, but for PDF you need highly specialized software packages to do this. One of the best, pdfplumber, is largely based [1] on a Master's thesis titled Algorithmic Extraction of Data in Tables in PDF Documents [2].

[1] https://github.com/jsvine/pdfplumber/blob/stable/pdfplumber/...

[2] https://trepo.tuni.fi/bitstream/handle/123456789/21520/Nurmi...

This was mostly aimed at the various ways the XML document may or may not conform to any number of XSD types. What we 'see' as a table might not be described and stored as a table in the same way in XML. And with XML I mean whatever XML office (the one from microsoft) generates.

A 6000-page spec and attributes that specify if the data is tabular data based on various properties (be it columns and rows or just plain text with start and stop pointers) and then may or may not render it visually as a table is error prone, even on first-party implementations (first-party desktop versions within Windows vary, as wel as on macOS, Android, iOS and their web offering).

If there was one simple data structure describing the table and all other aspects being optional, then yes, a XML based format is easier than OCR. But that's not the case I was pointing at.

> for PDF you need highly specialized software packages to do this.

Not really, or at least not all that specialized. You need:

a: a pdf-to-raster-image converter (ie any working PDF viewer, plus maybe the X server it talks to)

b: a reasonably decent OCR system capable of scanning tables (definitely nontrivial, but hardly "highly specialized" since things other than PDFs display data in tables).

I never said it was a bad choice, only that they didn't say why they chose it.

I guess, but you were replying to "I was wondering why docx format would be chosen instead of PDF" seemingly unconvinced, so I assumed you thought PDF would've made more sense.

Neither of these are sane. PDF and DOCX are both built to ape analog paper processes. It's been 40 years of this computer thing, it's time to get rid of the 70s office facsimiles.

I think the knee-jerk is against any alternative to Office, not against Office. Statistically speaking, trying to use anything reasonable that doesn't genuflect to Microsoft's monopoly is what seems to be met with a knee-jerk reply such as yours. As in, there's probably more people who don't care but hate the complaining about libre stuff than their are advocates for libre stuff.

> But it doesn't really say why they chose to build on docx.

Having worked directly with their teams in the past (although not on this), a lot of their systems seemed to evolve naturally over time based on the needs present. In that industry, a large majority of the documents being passed back and forth are DOCX. So my semi-educated guess is someone built a system to handle some simple intake tasks for DOCX applications because a large majority already were, it evolved over a few years, and when they finally decided to fully automate the process, they decided to build upon what they have which only supported DOCX and it was cheaper/easier to mandate everyone submit in that format than to build a new system or add support for others.

I get that you worked with them it seems, but would argue that your hunch is wrong here.

Regulatory processes, business systems, and international integration are plagued by PDF OCR complexities. OCR creates systemic issues and an anatomy of complex system architectures. Im sure XML is a typical downstream for parsing anyways. Use DOCX to enhance quality of the overall scope of integrations.

They could just use standardized application forms the way they do for research reports they require (the "ISA ###" forms). Those forms are easily parsable by things like pdftk and don't require any OCR.

I don't necessarily disagree with your point (since it makes complete sense), just wanting to point out that they already have a system in place for this using other means (although even there they are moving toward XML instead, likely because of what a pain it is to deal with text that exceeds the area of the input in PDFs)

This makes the most sense.

As a few others have mentioned, the parsing alone means DOCX is a huge win over PDF. I had to parse a bunch of PDF data related to COVID and it was always a PITA. Every time they changed their layout even a little bit I had to rewrite parts of my extractor. The worst part? The headers/metadata showed it was all made in Word so they could have exported to DOCX as well as PDF if they wanted to but they only provided PDF.

>But it doesn't really say why they chose to build on docx.

  Is requiring the DOCX format just adding another step in the process for applicants?
  Actually, it's the opposite. The USPTO conducted a study and found that over 80% of applicants are authoring their applications in DOCX format (through writing tools such as Microsoft Word). Because the files are originally in a DOCX format, uploading the original file eliminates the step for the applicant to convert the document to PDF prior to submission. Instead, the applicant is able to save the step of converting because our system will do that automatically.

Wait, what? So they convert from DOCX to PDF after you upload? But you have to upload DOCX? This sounds like the worst of both worlds.

Not really for their own purposes, no. They do convert to PDF (and TIFF!) for various archiving and display needs.

I guess you have little exposure to the industry. My experience is the vast majority already use Word or something else which supports DOCX. I cannot think of another format which practitioners have easy access to and would use. PDF just needs to go away for this process.

I actually do have a lot of exposure to patents, and I know everyone uses DocX already. I'm just saying that that web page doesn't say why they chose DocX, only why you should use DocX.

Sure it does. I'll give you it does not say why they chose it over other alternatives which I'm thinking is what you are looking for. Are there really any alternatives? The only real alternative I can think of is OpenDocument Format and I don't consider it alternative. As they say on that page, 80% of their users already deal with DOCX so 80+% of them will have to convert to ODF. I can't imagine ODF having any sort of benefit worth requiring most people to convert their documents before sending.

To me, the salient question is why is the government officially adopting a proprietary file format? Why is it important to optimize for the trivial convenience of patent applicants?

It seems more like rationalization than reason.

Isn't docx technically an open file format? https://en.m.wikipedia.org/wiki/Office_Open_XML

That's a standard for a file format, but that doesn't necessarily mean it is a free/open format. I don't think the FSF, for instance, considers it open (maybe because of patent issues). I'll leave that for you to decide, but just mean to suggest a standard doesn't automatically mean free/open.

Interestingly, a section in the wiki article linked mentioned the standard proposal was controversial because ODF already existed (and ODF was considered less complicated as a specification).

Nevertheless, good point. It depends on what you mean by "open".