Interesting; I've build a similar application for assisting with the translation of localization files (mozilla's project fluent) and for the same reason as well.

I build this stuff for fun; and because I needed something like it. I don't expect to be making money of it so I want to minimize operational overhead and hassle. So it suits me to not have to build and run an application server for this; even though I'm well capable of building such a thing.

It's available here: fluent-ai.jillesvangurp.com if people want to play with this. It uses openai in the browser and probably can work pretty easily with claude as well. Bring your own key. It's all open source if people want to play with this.

Could ads even support such AI heavy use cases?

I'm kind of out of touch with current AI API pricing and ad revenues, so i'm curious how the economics work out.

Yes, we're now at an inflection point where this is starting to become possible. gpt-4o mini costs $0.15 per 1 million input tokens, and $0.60 per 1 million output. This is cheap enough that it can, at least in some cases, be funded by ad impressions.

Of course, the implications here are mixed. If you want to build an ad-supported tool that actually helps people, that's great. But it also means it now makes clear financial sense to fill the web with AI-generated garbage with the assumption that that ad impressions will pay for it.

This is also pricing for customers, not what it actually costs to run.

Yes, but that's what's relevant, right? If I create a service, I don't really care if OpenAI makes a killing or is subsidizing my cost with venture capital, what matters to me is how much I pay to OpenAI, and how much revenue my service generates.

It's relevant as far as the providers of said LLM inference could always swoop in, undercut you and take your business if they feel like it. But they could do that anyways, no matter if they do it at a profit or not.

This assumes the service developer adds no value to this process. Because otherwise its like saying nVidia can swoop in on EA's business.

Edit: s/App/service.

Another implication is that you're dependent on it remaining cheap enough. You risk VC money running out and them having to jack up prices, or them doing so because they managed to capture the whole market.

I don’t think it’s cheap because VC money is subsidizing losses on every token. It’s getting cheaper because models and infrastructure are becoming more efficient.

And I really don’t think any of the AI API providers can “capture the whole market”. There are at least 3 of ballpark equal capability, so I don’t see how dramatically raising prices is compatible with dominant market share.

Even if the inference is getting cheaper, all the frontier companies are running massive losses building and serving it. That has to come back eventually, that's just how capitalism works.

Just remember that Netflix didn't start really jacking up the price till after the other players entered the streaming war, when they were pioneers it was dirt cheap. The existence of Disney+ didn't stop them at all.

One major difference is that Netflix has a monopoly on much of its content, but LLMs are fungible.

Dell was never able to jack up its prices, even when it was dominant in the market, because people would just go to another vendor. I think OpenAI is closer to a Dell than a Netflix.

if new models won't require to have significantly better hardware then I think eventually many cases will be run on locally. Currently you have macbook pro with even 128GB RAM - of course almost nobody has it but those hardware will be mainstream common in ~5 years. You already can have ollama - you just need to have equivalent preinstalled by Microsoft/Apple/Google on their all new devices - again most likely this will happen in <5 years.

Haha, maybe Gen AI needs will finally drive up memory needs and the memory installed in most device, but i doubt it.

Apple still happily takes 1800$ for 8GB laptops. Wouldn't be surprised if that were their most sold SKU ans the laptops sold today will be around for quite some time.

Fixed costs vs variable costs. That’s how capitalism works.

Probably not.

I do the same now for a firefox extension I wrote (automatic form-filler that works way way better than anything else out there).

So it's also "bring your own keys" but then how do you monetize at all?

I personally don't like "bring your own keys" at all from a user-friendlyness perspective. It means that you exclude the vast majority of potential users, because they don't know what that even means. Even "create an account" is more user friendly.

"Bring your own keys" can mean a "log in with OpenAI" button. Having users navigate through a third party's arcane dev portal isn't a good experience, but that third party can make it painless for users if they so choose.

I absolutely think that OpenAI or Anthropic should provide such integration. It’s very similar to how Apple Pay centralises your subscriptions and makes payments secure and simple. Would be nice if AI labs had an equivalent portal where each authorised app gets its own key and I can cancel any time and control my spending.

> Finally that might enable some kind of monetisation if OpenAI or Anthropic give developers a cut, e.g. 10% mark-up that goes to the authorized app.

I was totally against you until this, but it’s an interesting idea. Its still early, but seems like OpenAI hasn’t succeeded any more to be broad consumer product past the core Chat experience. Building an AI OAuth platform would be an interesting way to be sticky and avoid being a commodity. But it’d give developers more leverage vs their custom-GPT product, and it’d shift charging per-use for an API to “unlimited” per month for a single subscription fee.

Generally, a product shouldn’t tie themselves to an API provider (eg OpenAI) when it could’ve been an implementation detail. If you hide the actual API from users, you can swap it for cheaper or better ones as the market evolves. If you give up the account access to a providers OAuth, and you give up control over that implementation, you risk being really stuck to a market loser and no direct relationship with users.

Getting paid for it though…. That would be an interesting twist. But I’m not sure it’d make sense as anything but a bulk discount. The problem is that it doesn’t make sense to pay a developer to use your paid product, unless you get a relationship with the end users like Google Search defaults in browsers. But again, it doesn’t make sense to give OpenAI that relationship if you don’t have to.

They don't; OpenAI maintains a separation between "general population" ChatGPT Frontend and the LLM API on purpose. It's arguably a good purpose.

And the issue should really be inverted: it's not about excluding less technically savvy users - it's about recognizing a market niche of more sophisticated users, that really want that feature, can likely pay more for it, give you word-of-mouth marketing for free if you execute well. It's a niche so underserved that you don't even have to compete with scammers and shovelware all that much.

> So it's also "bring your own keys" but then how do you monetize at all?

TypingMind.com is a "bring you own API key" (obviously, being a LLM frontend), that's also successfully monetizing users. The secret is that it's actually a very good product; until recently, it was far ahead of the official tools (I mean, they had plugins for like half a year before OpenAI started talking about "GPTs"), so paying for the license feels worth it (definitely was, when TypingMind was strictly better than ChatGPT Plus subscription).

It's also not a subscription - another reason "bring your own keys" apps are interesting, because you likely already have a paid subscription with the API vendor; adding another one on top of that needs some good justification.

Anyway, "bring your own key" users are a different market from general audience, and unlike the latter, it isn't already saturated with fly-by-night garbage and scam extensions, so you can both charge more for that feature, and have smaller costs marketing it.

I think with some good UX that people suggested in different post (such as OAuth etc) it could work even with non tech savy users. Similar deals happens very often e.g. when you are renting or leasing a car you still have to pay yourself for gas. Same with renting long term for accomodation you pay utilities yourself.

> So it's also "bring your own keys" but then how do you monetize at all?

Why do you need to monetize? The original comment you replied to talked about making something for the world and sharing it. They said they didn’t want to maintain it, they didn’t want to be obligated to care for it. You can’t make that choice if people are paying you (or at least shouldn’t…).

I don’t understand the BYOx use case for a monetized product. If you’re BYO api, you’re essentially missing the opportunity to monetize a spread on API requests. The more a customer uses your product (because it’s good), the more you’d make. That’s the best case scenario because it means everyone is finding value.

> Why do you need to monetise?

Some people, myself included, are trying to earn a living creating software that helps people in some way. Just like any other physical or digital service, it’s fair to charge for a useful tool

> I don’t understand BYOx use case for monetised product

In my case, BYO keys turns out way cheaper for the end user. For instance my tool calls an LLM API. If I were to host the keys myself, I’d be charged $X for Y calls.

By getting the user to bring their own key, in my case the user easily fits into the free tier of the LLM (Gemini in my case) so the product costs me $0 to run, and I just charge a small service fee for me having created the product.

This allows me to keep building useful tools, some free (7 of 8 projects so far) and some paid (1 of 8)

BYOK frees application developers from being inference resellers and enables generous free tiers where you convert users because they love your app and want advanced functionality, not because it has a 7 day free trial then they can't use it anymore.

Also, subscriptions are a garbage business model from the user perspective, it's literally a dark pattern. They make sense for things with recurring costs to provide, but for instance, I should be able to buy a copy of Cursor and plug my key in and use it forever, and only shell out if I want upgrades. It's a subscription service because they're trying to bleed their users dry, and I'm sick of it.

Bring your own keys to minimise costs, you can still charge a subscription or one off charge for the base service if it’s a SaaS extension or similar

What's the name of the extension. I was looking for something like this recently

Outdated proof of concept version (can only use super slow local LLM using Ollama with llama3.1): https://addons.mozilla.org/en-US/firefox/addon/llm-form-fill...

Will probably update it in the coming weeks with one that can use LLMs through openrouter (Claude and others). I'm testing that now and fills forms in < 5 sec, usually better than existing tools (roboform, Bitwarden, ect).

I do the same, I make chrome extensions and my most recent extension uses a byo api key model for calls to an LLM.

Means I can offer the service for free and not worry about hosting keys, serving ads, and storing users keys in a db. Everything can be done client side.

Good to hear I’m not alone in the endeavour, what software are you building?

I made two one-page react apps

https://www.livetranslate.net/

https://www.subsgpt.com/

Second one is more refined but both are functional.

I was pleasantly surprised somebody made a YouTube tutorial in Japanese about the latter https://www.youtube.com/watch?v=8gAkvZYayEc - feels like retro internet where people share things on their personal webpages.

Curiously the first one also landed me a contracting opportunity for a company that wanted to add live captions to their product and we went live with my help.

I also made a decentralized twitter dapp ages ago but AI apps definitely have had more interest.

Do you think it would be possible to attach other language SRTs in the context? For example when translating English to Polish, the LLM has no idea whether the lines are spoken by a man or a woman, so the polish translation will be very confusing. However, if I could give the model both English and French subtitles, the gendered words from French would let the model avoid the confusion and the polish translation could be much more accurate. Does that make sense?

I offer both in https://kidzfun.art. If you're non-technical, you can buy packs of 100 images and it uses my key to access Dall-E, or you can provide your own key and pay nothing to me. The vast majority of users go the simpler way, but it's a nice bonus for technical users to just reuse their own key. The difference with your approach is that I store an encrypted copy server side as I do all the AI generation on the server.

I don’t understand why you’re being downvoted. I think this is a reasonable approach. If you want convenience, you pay for it – otherwise it‘s BYOK.

Not saying it isn't reasonable, but I'm guessing people might downvote because of storing secrets server-side rather than passing them on from the frontend and saving them there instead. People get worried as soon as secrets are stored anywhere :)

I think the biggest issue is that the vast majority of all Internet Users, including 'techies' really dont understand Secretes, Security, risks, non-risks etc...

I think that What HN (the site) is actually lacking is any kind of formal education [section] on the state of tech. Esp. given how much of SV tech zeitgeist flows through the frontpage of HN and the folks in its orbit - HN is missing out on a service that could look like a "tech News podcast" where Khan Acadamy meets OpenCourseware CS level snippets...

As an example - there have been a flurry of tools and launches and shows to HN recently that if there was a 15 minute video explaining the TechLego - and you could watch all these announcements and little educational doo-dads for the various tech componentry and tooling being shown here - a scrappy motivated modern version of 20-year-old [Every Grey HNer] could build wonders with...

We need to give people a solid grasp of all these concepts and issues, best practice, and the WHY we think the way we think about things such as secrets, auth, security. (the boring layer in OSI for most)

Yesyesyesyesyes I didn't realize how badly this is needed and how much I would like to work on this until you brought it up.

It's wild how much useful information flows through the HN Zeitgeist! I singlehandedly attribute my career success/position to keeping up with it all

I think it was probably the gratuitous inclusion of the url

It wasn’t gratuitous at all and the “self-promotion police” are an insufferable plague on this website.

I put together https://nfriedly.github.io/contributor-locations/ a while back, which has the same idea except that the access token is optional. GitHub's API provides a small number of request without one, but adding an API key will enable it to do more.

Yes I couldn’t agree more. I wish there was more support for this; like for example a system where users can be sure that the key cannot be stolen by the app.

Another benefit of BYOK is it simplifies the implementation. You don't need to spend as much time protecting against the attack vector where they rack up a big bill against your key.

Yeah, I also love this style of web app, for the same reasons you note!

I made https://github.com/pickledish/cardi as a kind of dynamoDB-based bookmark keeping tool in this style :) though I haven't worked on it in a couple of years.

The UX could be so much better and more secure. This type of use case is a perfect fit for OAuth2.

Current UX:

1. User hits your app

2. You tell them to go to Anthropic and generate an API key. You'll probably need to give them instructions on how to do so, which will become outdated over time as Anthropic makes changes to their website.

3. User goes to Anthropic and generates an API key

4. User manually navigates back to your app and pastes the key

OAuth2 UX:

1. You redirect the user to Anthropic

2. The user approves your app getting access

3. Anthropic redirects the user back to you and your app starts working.

For the life of me I don't understand why orgs don't implement OAuth2 for basically everything. Yes it is more complicated on the developer side, but that's for good security reasons. And it isn't that bad, and well worth the moderate time investment.

You have to remember that dev time is always at a premium. Even if it takes one week to build OAuth2 support, that's one week delay on other features. It's always a matter of priorities.

Sure but the alternative we are talking about here is encouraging developers to paste their API keys into random web pages.

I don’t think we should accept the argument that since implementing a respectable authorization scheme might take a bit of effort, it’s okay for sites to ask users to just hand over their password.

Wait... can't they use Claude to build this for them? :)

Devs here often don’t even really understand JWT tokens as evidenced by the hundreds-of-comments deep arguments I’ve witnessed over them.

JWTs are orthogonal to OAuth2. Tokens in OAuth2 are opaque to the client applications. JWTs are one way to do it, though with significant tradeoffs.

While this is nice, I don't think the dangers are discussed enough. A user has no guarantee their key isn't just being sent to some malicious third party. Normalizing this seems dangerous because it only takes a couple bad actors.

Sure you could try to get people to issue / delete keys every time they use an online app but it seems unlikely most users will do that.

The browser doesn't need to facilitate this.

They could generate an application specific key. Could do that every time one uses the application by forwarding though the website issuing the key and back.

I want government id to work like that. You authorize the website on the .gov then the website only gets a key, no further information. The only thing to knows about the key is that each citizen gets to generate one key for each [registered] domain.

probably not a good idea for your government to know their each citizen what kind of websites uses. However there are similar idea currently in some crypto projects that trying to solve it to make it distributed, so that if you e.g. like gambling the government won't know about it but website that provide gambling services can still check that you are from non banned countries and adult without any more detailed KYC

> A live transcription and translation app that uses microphone input. This is useful for watching proprietary content and facilitating communication.

It would be funny to transcribe someones speech, improve the grammar and play it back in their own voice.

I love this pattern as well, but as some comments have pointed out, there is a pitfall of security risk (you don't know where your API key is going). I think it would be really cool if more services let you cap API key usage, so it's almost like a virtual credit card number. I could then put in API keys to sites without worrying and knowing that at most, I can lose, say, $10 or something.

This is absolutely the thing I most want: it should be trivially easy to create a dedicated API key with e.g. a $5 maximum spend that I can share with a "bring your own key" tool.

Being able to issue those via an OAuth flow as opposed to copy-and-paste would be nice, but the fundamental thing I want is per-app spending limits.

Obviously you aware, but let me still say it: openrouter is for me a trivially easy way to create dedicated keys with max limit https://openrouter.ai/credits or am I missing something?

oh this is interesting thanks for linking gonna check em out

This is pretty much exactly what an oauth2 flow is. You generate a single-use-specific token and constrain it with the limited permissions you choose. If a site wants to offer an oauth token flow where users can put a time or credit usage limit on the token they absolutely can do so.

OpenRouter.ai does this pretty well. It has a pretty reasonable flow. It supports enough free models that you can do at least _something_ without a person having to pay anything (even if quality is a bit iffy). They have user-set spending limits, and each key is bound to the service. Their privacy policies seem decent.

The other option is Gemini, which has enough free credits to really be free for any small app. Unfortunately you can't use that through OpenRouter, you need your own developer account to be able to take advantage of Gemini's free offering.

I’ve always thought this would be a cool thing to offer, one thing that I’ve asked myself though is why should I trust you to use my key? Who guarantees that you’re not going to abuse it or use it in an inefficient way?

I’m unfamiliar with how these keys are priced but I guess they still use number of tokens that you input? What happens if one day your software has a bug that sends 1k tokens multiple times and I get charged for it because I’ve used my key?

Note that this is not a complaint and I would definitely love a “bring my key to use for free” tier, just a legitimate question that I asked myself from time to time.

On the other hand, I trust you very much in doing your best work to avoid any bug like the one I’ve presented above and squeeze every little penny out of your key to maximise earnings.

Finally, how do you make people entrust you with their keys?

The browser runs the code, which is open-sourced. You can look at the network tab of the debugging console or isolate the browser completely by sand boxing it. I agree that UX can be improved but that’s more on the browser side.