News for hackers. Stuff that matters.
Re:Congress votes to make open government data the default i... by TheAceOfHearts on Sunday December 23, 2018 @05:27PM (#18748774)
Do you have any tool suggestions or general advice for someone trying to do this? A while back I was trying to extract text from some government PDFs in order to make the information more accessible for others, but I became a bit overwhelmed when I started reading up on PDFs.
Re:Congress votes to make open government data the default i... by _bohm on Sunday December 23, 2018 @08:43PM (#18749699)
Sure! In terms of raw text extraction (for documents that don't require OCR), the most useful tools I've worked with have been pdftotext [0] and PyMuPDF [1]. For extracting useful details, really, my best advice is to make sure that your regex skills are sharp. I've been meaning to explore the possibility of using NLP tools for named entity recognition, but unfortunately I don't have much of a background there.

The rest kind of it kind of just comes down to using good software engineering practices to help keep yourself sane. Find useful abstractions for common tasks you need to perform and build a library around them, make sure that your data processing pipeline is designed with enough flexibility to handle inputs in different formats so that adding or modifying parsing logic becomes trivial, etc.

[0] https://www.xpdfreader.com/pdftotext-man.html [1] https://pymupdf.readthedocs.io/en/latest/

Re:Congress votes to make open government data the default i... by ocrcustomserver on Sunday December 23, 2018 @09:21PM (#18749855)
pdfminer is another good library (Python).