Ephil012
Disclaimer: I work at Tonic
Show HN: I Built a GitHub Action to Monitor LlamaIndex Performance
Hi all, I had been working with tools like LlamaIndex at work and realized that there were not many good options for monitoring RAG systems for performance. So I built a pretty simple Github Action + PyTest setup to measure the performance of LlamaIndex over time. The setup basically uses a library called Tonic Validate, which scores the quality of your RAG system’s answers (Disclaimer: My...
Haystack: A Great LangChain Alternative for RAG
Validating the RAG Performance of OpenAI vs. LlamaIndex
Shameless plug for the library: https://github.com/TonicAI/tvalmetrics
No, OpenAI won't kill RAG startups
Analysis here if anyone is curious https://news.ycombinator.com/item?id=38280718
Here's my post with the analysis. https://news.ycombinator.com/item?id=38280718
OpenAI's Assistant API can't handle multiple files
Evaluating Rag Parameters Using Tvalmetrics
One thing we have considered is some forms of evaluation could be replaced simply with using the embeddings of the question, context, and answer instead of using the LLM model for analysis. The idea is you could compare all the embeddings to get a rough idea of the performance based on similarity. That should in theory reduce costs. The only other alternative is just to use less advanced models which are cheaper.
Show HN: Tonic Validate Logging – an open-sourced SDK and convenient UI
https://github.com/TonicAI/tvallogging
Hey HN, Joe and Ethan from Tonic.ai here again. Alongside last week’s announcement of Tonic Validate Metrics (https://news.ycombinator.com/item?id=38012126), we’ve also released an open-source SDK for logging the performance of Retrieval Augmented Generation (RAG) applications during development, Tonic Validate Logging. Tonic Validate Logging is used to log your RAG responses to the Tonic...
We also have a UI for visualizing all of these metrics that is free at the moment with no paid tier at all. We are planning to add more features into that UI for better visualizations along with improving the ability to keep track of different versions of your LLM system for comparison.
Show HN: Tonic Validate Metrics – an open-source RAG evaluation metrics package
https://github.com/TonicAI/tvalmetrics
Hey HN, Joe and Ethan from Tonic.ai here. We just released a new open-source python package for evaluating the performance of Retrieval Augmented Generation (RAG) systems.Earlier this year, we started developing a RAG-powered app to enable companies to talk to their free-text data safely.During our experimentation, however, we realized that using such a new method meant that there weren’t...
Let me give an example. On Kagi, I searched up "Overcoming addiction to Reddit". On Kagi, most of the top results are actually just Reddit posts about drug addiction / addiction in general and not Reddit addiction [1]. If you scroll down, the is a result about internet addiction, but that isn't what I asked for either. Only once you get to the very bottom do you actually get links about Reddit addiction itself. Even then, there aren't that many.
If you do the same query on Google, I get much more results that are actually about Reddit addiction [2]. On top of that, the relevant results are way closer to the top than on Kagi. It even includes high quality links like a Hacker News post on Reddit addiction. I believe Kagi included the same link, but it was hidden all the way on the third page as opposed to being somewhat near the top on Google. Don't get me wrong, Google's results aren't perfect and it has a lot of irrelevant stuff show up in my query about Reddit addiction. But it's still better than Kagi.
My theory is that Kagi just saw the words "overcoming addiction" and "Reddit" and assumed I wanted to look for links about addiction in general on Reddit. Basically it's almost like it did "overcoming addiction site:reddit.com" for the top results. Albeit, after the top results it does show links from other sources. Meanwhile, it seems Google did a better job at parsing the query and realizing I was looking up a specific type of addiction, a Reddit addiction. Kagi did not do this.
My next problem with Kagi is just the lack of results in general. It seems Kagi likes to return a limited number of results per search. It seems it usually gives 20-30 results and sometimes gives more (~70 results) if you give it a very generic query. The problem with this is it makes Kagi terrible for research. In daily life, I never even read past the first few pages on Google. But when researching a niche topic for a paper, you sometimes need to go way past the first page to find what you're looking for. This might be because when doing research on something vague or niche it's hard to get accurate matches on the first page of search. As a result, having a large number of search result pages in general is useful for when this happens. But Kagi simply doesn't do this. It's limit on how many results it returns just kills the ability for you to actually go more in-depth without issuing a modified search query. I seriously doubt Kagi's index is literally so small it can only return around 70 something results at max, so I just have to think this is some self-imposed limitation which ultimately harms the search quality.
Another problem I have with Kagi are lenses. Lenses get hyped up by them, but in practice I found them utterly terrible. I searched up Lo Mein without lenses on and my first results were some pretty high quality recipes. I then turned on the recipe lens and the quality noticeably decreased. Instead of returning Lo Mein recipes from actual recipe sites, it instead returns a Reddit post that links to a gif of someone making Lo Mein. Issue is that gif is actually on gfycat which is shut down. So the top result is useless. The rest of the results aren't even Lo Mein recipes, but instead chicken recipes for instance. So in the process of trying to use lenses to increase my search quality, it actually made it worse.
Also, another tiny nit is that Kagi isn't as good at getting quick answers. If I look up a question on Google, it will usually display some quick summary at the top with the answer to the question. Kagi has quick answer too, but you have to manually click on it to get Kagi to generate the quick answer. I get they're a startup trying to save money, but this is just another reason for me to use Google over Kagi.
As a whole, I haven't found Kagi to be useful at all like many commenters claim it is. I really wanted Kagi to work out as again I love the idea of Kagi. But ultimately, I can't justify using it over Google because the quality has just been so much worse in my experience. Ultimately, I am paying money for Kagi, yet I am getting a lower quality product than Google which is free. At that point, it doesn't make sense for me to continue with Kagi.
1. https://kagi.com/search?q=overcoming+addiction+to+reddit
2. https://www.google.com/search?q=overcoming+addiction+to+redd... (Google may show different results to you because Google might be customizing results based on past searches)
As far as the Apple support app, I think that Apple does have a legitimate app under that name. However, usually find my is disabled under the find my app, not the Apple support app (unless Apple stores use a custom version with it in there or it’s hidden in the regular app where I can’t find it). It’s also possible the support app just redirected you to the find my app. Either way, if you don’t trust the app, you could just go to a web browser at home and disable it via the web app for find my so they can proceed with the repair. I don’t know if third party repair shops will also make you disable find my, but I know Apple usually asks you to disable find my first