Arimbr

I like the YAML abstraction. This should make it easier to programmatically try and evaluate multiple configurations for the whole AI pipeline (not just the LLM) against a dataset or real users through an API deployment.

Some feedback: It would be great to see in one place all the supported fields and values for the YAML config.

What is an hybrid index?

Nice, thanks! I was reading https://pathway.com/developers/user-guide/deployment/persist.... If I understand correctly you persist both source data and internal state, including the intermediary state of the computational graph. And you only rely on the backend to recover from failures and upgrades. So if I want to clone a Pathway instance, I don't need to reprocess all source data, I can recover the intermediary state from the snapshot.

Is it the same logic for the VectorStoreServer? https://pathway.com/developers/user-guide/llm-xpack/vectorst...

If all the pipeline and the vector index is keep in memory... does Pathway still persist state somewhere?

The AI Connector Builder from API docs is insane! Which API doc specifications will it support? Or does it even matter?

Interesting implementation! For complex stream and text processing, I also prefer processing data in memory with Python (ETL) rather than SQL in the warehouse (ELT).

I see the ingested documents in the data folder don't have an id field, only a doc field.

{"doc": "Using Large Language Models in Pathway is simple: just call the functions from `pathway.stdlib.ml.nlp`!"}

What if I pass two contradictory statements? Is there a way to remove (or better update) a document with a new version?

For example, if I am ingesting some public docs, and I update a doc page. How do I make so that it only takes the answer from the latest document version?

Hi, interesting!

> Then it processes and organizes these documents by building a 'vector index' using the Pathway package.

What is the Pathway package?

Nice list of resources!