pierrebrunelle

Building an infinite-memory, context-aware, multimodal chatbot is as easy as 600 lines of Pixeltable code.

Why? Because Pixeltable is the only multimodal data infrastructure that unifies storage and orchestration for your AI workloads.

Agree...as simple as:

@pxt.query def search_documents(query_text: str, user_id: str): sim = chunks.text.similarity(query_text) return ( chunks.where( (chunks.user_id == user_id) # Metadata filtering & (sim > 0.5) # Filter by similarity threshold & (pxt_str.len(chunks.text) > 30) # Additional filter/transformation ) .order_by(sim, asc=False) .select( chunks.text, source_doc=chunks.document, # Ref to the original document sim=sim, title=chunks.title, heading=chunks.heading, page_number=chunks.page ) .limit(20) )

For instance in https://github.com/pixeltable/pixeltable

I fully agree and this is why context engineering matters and unifying storage and orchestration and treating agents as just another function call is significant and getting full visibility into the pipeline to easily iterate and with with the I/O. This is a good sample implementation of that: https://github.com/pixeltable/pixelbot

It really depends on the use case. For Pixelagent, I don't really look at competition as mentioned by the fact that this is a reference implementation.

People should build their own; it's easy. What's difficult is the underlying data infrastructure (storage and orchestration).

It all started with https://github.com/pixeltable/pixeltable which is the core project we are actively working on and the problem space of working with video and frames for Computer Vision teams who struggle with the explosion of data and maintaining lineage and versioning for frames of video.

The same problems arise with LLMs for Documents and Chunks, Audio and Samples, etc.

If you are into that:

- Object Detection on Video Frames: https://github.com/pixeltable/pixeltable/blob/main/docs/note...

- Text and images similarity search: https://github.com/pixeltable/pixeltable/tree/main/docs/samp...

LLM is just a function call. We are agnostic to whatever framework, library, and data format are used; we are focused on solving the data plumbing issues (mulitmodal storage and orchestration) with a simple python sdk: https://github.com/pixeltable/pixeltable.

I like PocketFlow. You beat me on the # of lines of code! But does it provide parallelization, caching, orchestration, versioning, observability, lineage, multi-modal support?

As you just showed, building an agent SDK is easy, so what's interesting to me is tackling:

- Infrastructure Sprawl: Juggling separate systems for vector search, state tracking, multimodal data handling, and monitoring leads to fragmented workflows and high operational costs. - State Management Nightmares: Reliably tracking agent memory, tool calls, and intermediate states across potentially long-running, asynchronous tasks is incredibly difficult. - Multimodal Integration Pain: Integrating and processing images, audio, video, and documents alongside text requires specialized, often disparate, tooling. - Observability Gaps: Understanding why an agent made a decision or failed requires visibility into its state and data lineage, which is often lacking.

And doing all of that while finding the right abstraction layer to leave all the application and business logic to the dev/users so they don't feel limited. It's difficult!

Besides, I don't know where you see a commercial offering? Everything is Apache 2.0/Open Source from A to Z.

Congrats! Take whatever you want :)

Yes: Memory, observability, versioning, and lineage being built-in is a derivative of unifying orchestration and storage: https://docs.pixeltable.com/docs/datastore/computed-columns.

Thanks!

https://github.com/pixeltable/pixelagent/tree/main/examples/... for multiple tools/agents. A tool is just a UDF. You can have as many as you want.

The goal of this reference agent SDK is to showcase the flexibility of Pixeltable (the underlying unified storage and orchestration system) that is Open Source and Apache 2.0.

This is where storage is defined: https://github.com/pixeltable/pixelagent/blob/main/pixelagen.... These are very simple examples.

Here's how an Agentic Reddit bot would work for instance: https://github.com/pixeltable/pixeltable/tree/main/docs/samp...

You can indeed turn anything that you want into an MCP server, e.g. https://github.com/pixeltable/pixeltable-mcp-server.

Pixelagent is a reference implementation for a multimodal agent framework to show that an agent class is easy to build and users should be empowered to build their own from scratch for their use cases.

Regarding Memory, to me it's just about Data Storage, Indexing, Orchestration, and Retrieval and I don't know why we should abstract Memory away from users. Memory will mean so many different things for many use cases.

Let's say you want:

- Working memory: Holds current context and immediate interaction history within the agent's context window -> this is just about passing Q&A pairs to maintain context alongside with roles.

- Episodic memory: Stores specific past experiences and interactions -> this is just about indexing past exchanges and having semantic search on it.

- Semantic memory: Organizes specific knowledge in structured formats -> this is just about building a custom logic (udf) to decide how and what to extract insight from and then retrieve it.

I've implemented them all in this example: https://github.com/pixeltable/pixelbot

Hi HN - I'm sharing Pixelagent (https://github.com/pixeltable/pixelagent), an open-source agent engineering blueprint we built to tackle the core infrastructure challenges of agent SDK development.

I see agents as the intersection of an LLM, storage, and orchestration. With this reference implementation, I show how you can unify them into a single declarative agentic framework.

Many frameworks focus on orchestration; for me the real pain points are in data infrastructure. This is why you see so many abstractions around memory for instance and people re-inventing the wheel around state management. All of these are just data storage, indexing, retrieval, and orchestration.

With Pixelagent, I show how you can get:

- Multimodal data handling: Processing text, images, audio, and video through a consistent interface

- Persistent state management and built-in observability: Automatic tracking and versioning of agent memory, tool calls, and context...

- Provider agnostic: Support for OpenAI, Anthropic, and AWS Bedrock with the same API.

- Declarative Model: A type-safe framework

- Agentic Extensions: Add reasoning, reflection, memory, knowledge, and team workflows.

Fork it, make your own or whatnot! I hope it inspires people to build the abstractions they need.

Onward,

Pierre

---

Get Started:

- Pixelagent GitHub Tutorial: https://github.com/pixeltable/pixelagent/tree/main/blueprint...

- Pixelagent Documentation: https://docs.pixeltable.com/docs/libraries/pixelagent

- Join our Discord: https://discord.gg/QPyqFYx2UN

We got sick of fighting with YOLOX's dependency every time we needed to update the integration for a decent object detector demo, so we forked it and fixed the annoying issues.

pip install pixeltable-yolox

Key stuff fixed:

- Works with Python 3.9+

- Compatible with current PyTorch

- No more version mismatches

- Actually runs in Colab without hacks

Will keep the Apache 2.0 license. No architectural changes to the model, just focused on making it usable in production.

ChatGPT directly into Jupyter Notebook with Notebookgpt by Noteable, notebookgpt.com as a plugin: http://notebookgpt.com/

Also a different one: https://github.com/noteable-io/genai with additional magics.

Nothing breaks the flow of experimentation in a notebook like an exception. You’re deep in thought, pulling data, and then suddenly everything comes to a halt.

To empower you to fix exceptions quickly we're releasing genai: an open-source library that brings the power of ChatGPT into error handling within Jupyter notebook.