Neither does the github "System requirements" section, which I find disappointing. Ideally, it should give minimum memory requirements and rudimentary performance benchmark table for a sample data set, across a handful of setups (eg, Intel CPU, Apple M1, AMD CPU, with/without a bunch of common GPUs). With that information I would know whether or not it's worth my time even trying it out on my laptop.
Edit: lol, I went through 2 pages of the issues on the github page and most of them could be avoided by putting this basic information into the system requirements:
https://github.com/imartinez/privateGPT/issues/174 https://github.com/imartinez/privateGPT/issues/179 https://github.com/imartinez/privateGPT/issues/104 https://github.com/imartinez/privateGPT/issues/141 https://github.com/imartinez/privateGPT/issues/282 https://github.com/imartinez/privateGPT/issues/316 https://github.com/imartinez/privateGPT/issues/333
... And lots more! Some of these people have 128gb memory and 32 cores, and still find it "very slow". Others having memory pool errors. Some of the answers hand-waving at needing "a more better computer"
I reckon a lot of these issues could be closed and linked to a single ticket for proper hardware requirements in the readme.
Locally ran LLMs are far worse.
I don't like it either, but for now, if you want good RAG, you have to use GPT-4
Although until these things can do my laundry none of them deserve any of my compute time either.
Once the vectors are created tho, you're completely off the cloud if you so choose.
You can always swap out the embedding generator too, because LangChain abstracts that for your exact gripes.
Everything else is already using huggingface here and can be swapped out for any other model besides GPT2 which supports the prompts.
Even if the alternative is free, how much do you value your time, how long will it take to set up an alternative, and how much use will you get out of it? This is also why it's never been the year of linux on the desktop, and why most businesses still don't use Libre Office and GIMP.
I can't speak for quality; even if I used that API directly, this whole area is changing too fast to really keep up.
Most of the other options just work with the transformers library.
And from my personal experience Ada embeddings are not the best. They are large (makes aproximate searching harder), are distributed weirdly, and zimply put, other embeddings give better results for retrieval.
Another advantage is that you are not an OA's whim: they just announced the deprecation of some previous model. What are you going to do when they will deprecate Ada v2 and you've built a huge system on top of it? You'll have to regenerate embeddings and hope everything still works just as well.
https://huggingface.co/spaces/mteb/leaderboard
Of course, that's far from saying that they're the worst, or even headed that way. Just not the best (those would be a couple of fully opensource models, including those of the Instructor family, which we use at my workplace).
qa = ConversationalRetrievalChain.from_llm(OpenAI(temperature=0.1), db.as_retriever())Ehr no? You'll need to also create an embedding of your query, which makes you totally dependent on OpenAI. If you swap out embedding algorithm you will have to regenerate all the embeddings as well, they might not be even the same size.
Here is my premise: We are past the wonder stage. I want to actually get stuff done efficiently. From what I have tested so far, the only model that allows me to do that halfway reliably is GPT-4.
Am I incompetent or are we really just wishfully thinking in HN spirit that other LLMs are a lot better at being applied to actual tasks that require a certain level of quality, consistency and reliability?
OpenAI have had many years to craft their dataset down from the noisy public datasets, and GPT4 is (supposedly) a mixture of 8 "expert models" each of which is 220B (5x+ larger than the Falcon 40B) with a total of 1.7B parameters (3x+ Google's huge 540B PaLM). The hardware and software to train networks of that scale is also a deep moat. Relatively speaking, the model architecture ("gpt from scratch") is the easiest piece.
Models are getting commoditized, well executed ideas are not.
- Pay openai less than $50mo
- Manage cloud gpus, hire ml engineers > $1000/mo
- Buy a local 4090 and put it under someone's desk, $no reliability +$1500 fixed
Any larger business will need scalability and you still can't compete with openai pricing.
Maybe one of you startup inclined people can make an openllama startup that charges by request and allows for finetuning, vector storage
And more models coming soon.
Vector storage isn’t on the roadmap (what stops using a separate vector store from working well? Could add to roadmap but want to add understand more first), and we could add fine tuning if it’s a common request.
If people can drag and drop some files from their nas, you parse them with apache tika or similar https://tika.apache.org/ , they can start using personalized branded bots. It also lets you do things like refusing to answer, if the vector database returns nothing and the use case requires a specific answer from the docs only (not the llm to make stuff up).
Issuer: McAfee OV SSL CA 2
Expires on: Aug 3, 2023
Current date: Jul 8, 2023
PEM encoded chain: -----BEGIN CERTIFICATE----- MIIGfzCCBWegAwIBAgIQKt9VNrFtaozA1bILX1OcfzANBgkqhkiG9w0BAQsFADBk MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0
Will your pricing be competitive with Replicate?
I've had great fun with the " Easiest 1-click way to install and use Stable Diffusion on your computer."
https://github.com/easydiffusion/easydiffusion
And while Whisper is OpenAI, it is trivial to use locally and extremely usefull
The use case is based on public information on the internet. News articles, PRs, social media posts, etc.
LLMs are used to extract info from text in a structured format. It used to have several classification and NLP models to do the job, but now a single LLM can do it faster and with better accuracy.
The instant a company has sensitive data, this becomes very viable.
I'm currently building www.lalamon.us specifically to provide a fully hosted open source model experience. One slight difference is that I'm providing a private chat instance for each user, so charging based on hours of active chat usage seemed to make more sense. Per-request charging seems more unpredictable for users, but I'd be interested in hearing the case either way.
Feel free to reach out with more questions if interested; my email is in my profile.
They're not here to release an actual product. They're here to release part of a CV proving they have "OpenAI" experience. I'm assuming this is the result of OpenAI not actually having any homegrown certification program of their own.
A bit off topic but where are certifications (e.g. Cisco, Microsoft) useful? I am sure they are useful (both to candidates and companies) because people go to the effort to get these certs, and if they were useless everyone would have stopped long ago. I don't assume people do it for ego satisfaction.
But I've never worked anywhere where it has come up as an interview criterion (nobody has ever pointed it out when we are looking at a resume, for example). Is it a big business thing? Is it just an HR thing?
To be a partner, the company would need a certain number of certifications among their employees, so there was tangible value to companies who either used a ton of Microsoft licensing or Cisco/Dell hardware, or resold those to their own clients (better discount equating to higher margin).
In some cases, getting the higher level certifications like Cisco CCIE was a virtual guarantee of a good job.
I feel like this has become less of a thing in recent years, but I’m not involved in that space anymore.
I'm so horrified that they are going to take away the ability to ask medical questions when the AMA comes knocking at their door.
paperai is a sub-project focused on processing medical/scientific papers. https://github.com/neuml/paperai
Disclaimer: I am the author of both
This commanding attitude on HN seems to be getting worse lately. Not a fan.
Do you know of any FAISS / open source / one-click install w<windows app here I can search in my PDFs via vectors? I can see Secondbrain.sh will have the function in the future, but currently it does not.
I have around 500 documents I want to be able to search in.
Stop doing that.
You will have way more users if you make OpenAI (or anything that requires cloud) the 'technically possible but pretty difficult art of hoops to make it happen' option, instead of the other way around.
The best way to make these apps IMO is to make them work entirely locally, with an easy string that's swappable in a .toml file to any huggingface model. Then if you really want OpenAI crap, you can make it happen with some other docker secret or `pass` chain or something with a key, while changing up the config.
The default should be local first, do as much as possible, and then if the user /really/ wants to, make the collated prompt send a very few set of tokens to openAI.