alexcg1
Search nested data with Python, DocArray and subindices
I'm working at an open-source project that builds an AI-powered search framework [0], and I've built some examples in very few lines of code (for searching fashion products via image or text [1], PDF text/images/tables search [2]) and one of our community members built a protein search engine [3].
A good place to start might be with a no-code solution like (shameless self-plug time) Jina NOW [4], which lets you build a search engine and GUI with just one CLI command.
[0] https://github.com/jina-ai/jina/
[1] https://examples.jina.ai/fashion/
[2] https://colab.research.google.com/github/jina-ai/workshops/b...
Search PDF text, tables, images with Python and CLIP
Summarizing text to create storyboards with DistilBART model and DiscoArt
The next level of multi-modality in DocArray
Text/image product search, using Python and CLIP
Using Karpenter to manage GPU nodes with time-slicing
IT Threat Detection Using Neural Search: Part 2
On the other hand, Tensorflow or MLNet are machine learning frameworks, to achieve the task you can choose whatever you want to build the model components.
Jainammm/TableNet: Model for table detection/extraction from scanned documents
Genshin Impact Fine-tuning CLIP for anime search
Seriously though, I agree. The fact that a file format can stick around that long is impressive.
Mars Express Images Show What's Inside the Biggest Canyon in the Solar System
In terms of processing the PDFs and extracting that data. idk. That depends on a lot of factors - e.g. do you need to OCR the PDFs or can just extract text directly? Either way, should be possible to write a module and then easily scale it up (Jina supports shards/replicas). Anyway, lemme know. I'm in talks with folks about this kind of shitshow...uh...use case now.
Jina supports multiple vector database backends, like Weaviate, Qdrant and others. For others (like Milvus), suggest you ask on the Slack [0] - responses tend to be fast.
Agreed on the rest. PDFs don't store machine-readable data. Often just pixelated scanned hot garbage dumpster fire text.
I hate PDFs but have to work with the satanforesaken things. Hence the notebook. It's my little way of trying to give my little PDF-bespoked-hellscape a tiny little glow-up.
[0] based on typical HN demographics, no assumptions here
Opening up the correct page? I don't know of any standardized PDF reader that supports that kind of thing. And the format has such a history that even if it were supported (technically by Adobe - don't even get me started on what PDF readers support what formats), there's no guarantee the file itself would even have that cooked in.