kristoo

The benchmark result is for 100 concurrent users

Thanks! Great question. The seconds vary a lot on a task, model and input basis.

For example, if you do something quick like text vectorisation on a couple of sentences then it is less than 100ms per call. That would be 10,000, 50,000 and 200,000 calls (0.0005, 0.0002, 0.0001 per additional call) respectively.

On the other end, if using GPT-2 Large to generate around 40 words then that takes around 2500ms per call. Giving 400, 2000 and 8000 calls (0.0125, 0.005, 0.0025 per additional call) respectively.

This is a PyTorch based library my team and I have been working on for the last few months, with the goal of making finetuning and using models as easy as possible for devs, even without extensive ML experience.

We've currently got support for text and image-based tasks (classification, generation, q&a, etc.), with wrappers around models like Google's T5, OpenAI's CLIP, GPT-2, Facebook's BART, and others.

We've got some features that make deployment easy, but for full transparency, it is through a paid platform [0] we've developed that is by no means necessary to use the library.

We're happy with the progress we've made, but we're curious to hear what people think so we can keep improving.

[0] https://backprop.co

Nvidia GPU support on Linux is great. I meant the lack of the support on Mac. Apps that are lacking and I frequently use off the top of my head is just Teams tbh.

But, as we know, Mac has great app support. The same level of polish and availability would be great. Just to have the options whenever the need arises. Currently finding good webapps is the best alternative.

I have not. I use a MacBook Pro as my laptop.

That would be a dream come true (if it had Mac level app support and additionally supported Nvidia GPUs). Probably next to impossible, unfortunately.

Nice! Out of curiosity, did you build it to learn/practice the low level concepts of deep learning? If so, was it worth it?

I use PopOS as my OS on a high end PC and it is pretty good. There are some things which I find annoying, such as Bluetooth headphones going to a headset mode at the start of calls. Just tiny things. Would I pay to have a smoother experience? Probably, - I would like the OS to be as smooth as possible.

Although, I don't think that's the main problem with a Linux distro. Most of the friction still comes from third party apps, such as Teams (which I use for calls multiple times a week and it absolutely sucks).

You might get some really promising results with finetuning.

If anything, you could build writing assistance that almost automates responses.

I've been co-authoring a library that lets you finetune such models in a single line of code.

https://github.com/backprop-ai/backprop

In specific the text generation finetuning example should be what you are looking for: https://github.com/backprop-ai/backprop/blob/main/examples/F...

Hope this helps, happy to chat more about it. Pretty curious about the results.

We have exposed the large GPT 2 model as an API.

https://backprop.co

Anyone can access it and our pricing is usage based. If you use less than 1000 seconds of inference a month then it's completely free.

That would be one incredibly expensive Q&A system. As rightly suggested by others, some search + a finetuned Q&A system is what you need.

I've been building a library that makes solving tasks like conversational Q&A easy.

https://github.com/backprop-ai/backprop

If you need some pointers, we have example notebooks and I'm happy to chat about it as well. Would love to know if it helps!

For better or worse that's pretty standard practice for banks. Usually a single phone call is enough to clear up the matter.