> I'm not sure "you and a swarm of a thousand digital clones" counts as fair use

It obviously does not. You’re a commercial client.

> You’re a commercial client

No, I'm broke and have no income. But about a year ago I wrote some 20 line Python programs that use GPT (feed it text from web searches) and was surprised to find they worked significantly better than Bing's AI search...

I used it at a small scale with Python's (unofficial?) DuckDuckGo API, but I was looking for a way to scale up (ideally for free or very cheap). At the time, Bing's API was the best option I found, and it was pretty expensive.

I think it would be really cool if I could have an autonomous agent doing research on my current interests in the background 24/7, but it seems my best option is a very low volume of (free) web searches to seed a small web scraper, plus possibly a custom-built search engine (see also https://news.ycombinator.com/item?id=38703943 )

You say you're broke witbout income, but from reading your comments you have at least some skills to pay the bills. And I bet that if you put your back into it you can do it without misusing the only good search we've got. Or anything else for that matter. Keep it up!

Thanks. And I'm sure you can succeed without implying your prospects are crooks.

Edit: from your tone I assumed you worked for Kagi, but now I'm not sure

Don`t work for kagi, and definitely not implying you are a crook. I believe your use case is not in good faith, and I also believe that you know that, but given your position it should be ok. And that is where we disagree, because I also believe you can do better. We all can. And sometimes that is something that needs to be said. I really wish you the best, and please do not take my words as an insult. It`s just the observation that my perspective grants me.

>regularly accuses me of being a bot!

>swarm of autonomous agents.

So....Google is right?

I haven't launched the swarm yet. Google is reacting to (as I said), my "human" search usage.

(Also possibly Brave browser's anti-tracking, which their captcha system hates and actively punishes.)

Apparently the average person does 3-4 Googles per day.. I checked my search history and I'm somewhere in the 100-200 range.

I've sometimes had a similar issue with searching at www.startpage.com. It became increasingly frequent during work-from-home, blocking me with a captcha or a false message titled "Just Checking…" (implying they're inspecting something, but actually requiring that you fill out a freeform text box explaining yourself). After a few dozen times, I started appending to the textbox an appeal for a paid version.

When I switched to eu.startpage.com, it stopped treating me like a bot. But eventually it started misbehaving in other ways: more than a page of ads (more than two, on mobile!), inserting the ads at the last moment so that your eyes and sometimes your click or tap would land on an ad. Occasionally, I'd win by searching for something so esoteric that no ads displayed. But later they started inserting results specific to my realworld location for searches that returned few or no results.

I reluctantly removed the adblock exemption for StartPage, which causes them to now insert a plea to support them by enabling ads. I contacted their support, who refused to believe that the had pages of ads, and they had no interest a paid version.

Then I found Kagi, and tried it every once in a while until it matured. I still sometimes goto StartPage's EU site on other computers, but on my own I mainly use Kagi.

There was actually an outage due to this (their first major outage!) a few weeks ago. Someone was scraping via their API a little too hard, which had unexpected effects on upstream systems. So 100x human search volume is likely to be an issue unless you discuss it with the devs first.

The page with the post-mortem for that outage now gives a 404 error, but was archived by the Wayback Machine: https://web.archive.org/web/20240121000935/https://status.ka...

(It was discussed on HN at the time: https://news.ycombinator.com/item?id=39019119)

We switched to a differnt monitoring system and the URL changed to https://status.kagi.com/clrnl9zwl97290beoine8zlvzx

yes https://kagi.com/settings?p=billing_plan

To rephrase my question, was the unlimited plan not available about a year ago? Otherwise I don't think I would have disqualified kagi at the time. I recall the volume being too low for my needs, even without hooking it up to any automated systems (which for me at least, was the whole point of looking into paid search engines).

Iirc it was available but more expensive as part of the Ultimate $25/mo plan.

They made the $10/mo plan unlimited in September 2023

was it not available about a year ago? I'm not sure. Maybe the wayback machine can tell you. But I have a gut feeling that there is a slight misalignment between your use case and their intended use of their service :)