I feel it's a huge difference between GPT-4, which seems to be able to reason logically around the issue and respond with relevant remarks, and Gemini Gemini Advanced which feels a lot more like a stochastical parrot.

Gemini quickly got confused and started talking about "pushing the door towards yourself" and other nonsense. It also couldn't stay on point, and instead started to regurgitate a lot of irrelevant stuff.

GPT-4 is not perfect, you can still hit things where it also breaks down.

> better at incorporating search results in its answer vs gpt-4 bing

How are you getting it to incorporate search results in its answers?

I can't for the life of me get it to find any real-time external data except for the 5 official 'extensions' under settings, which are for Flights/Hotels/Maps/Workspace/YouTube.

Did you mean that, or have you found a workaround to get Bard to actually search on Google?

- This is not Gemini performing a search. - This is Google providing a layer of ass-covering in case Gemini produces a factually incorrect reply.

Right? I am looking for something like ChatGPT with Bing - it will run a query, pull back results, and operate on them, all dynamically within the system.

Gemini doesn't seem to do this, no matter how you try to wrangle it.

So Gemini could by your one-stop AI shop for everything. Only Microsoft can match it (but Microsoft doesn't have a popular maps, youtube, mail, smartphone OS service).

Apple is another strong player (but they don't have productivity tools like docs, sheets or youtube).

It really is Google's to lose this AI race from now on.

Going to chatGPT and copying and pasting results will become painful (not to mention it's painful bing integration). Also at this point, they seem to be focusing on scaling LLM (while Google Deepmind is exploring other avenues)

Google can also bundle Youtube TV, YouTube Premium, Google Drive, Storage, Ad free Search, Gemini integrated Docs/Sheets/Gmail, Subsidized Pixel Phones / Watch for a monthly fee of say $99 and it'll be very compelling for a lot of people

The old Bard/Gemini integration with Gmail, Google Docs, etc. is pretty good. I signed up for a minute for Microsoft’s $20/month AI Office integrations, but cancelled and will try again in 2 months. I am an Apple customer and I expect spectacular things from Apple. I expect Apple, Google, Samsung, etc., to offer fantastic on device AI.

I would like to see a money saving Google bundling family plan. I find Apple’s super bundle family plan a pretty good deal.

We are also talking about consumer emails (not enterprise / corporate)

Also hard to overstate just how much more valuable the enterprise market is over the consumer market when comparing Microsoft vs. Google as one-stop anything shops.

I don't see Google as having the obvious dominant position to make the argument it's their race to lose, considering Microsoft has a stake in chatGPT and is actively integrating it into their browser and productivity suites.

1. Real estate - Youtube, Gmail, Maps, Search (for now), etc. 2. Compute - probably still the best in the industry, but with recent Microsoft/meta compute buys it's hard to say for sure. 3. Talent - probably also still the top of the industry. Geoff Hinton and Zoubin Gharamani setting direction and Jeff Dean building it is hard to beat, and the ranks are deep. Yann LeCunn is also brilliant and Andrej Karpathy while less seasoned is one of the top researchers in the field, but overall there's still a bit of a spread from Google's roster, at least when it comes to AI researchers.

If Sundar and the other top brass weren't MBA-bots with no vision, and the famous Google bureaucracy had been reigned in gradually over the last 5 years while promoting a builder-centric culture, this would be in the bag for Google no question. Instead, Satya Nadella played 3D chess while Sundar was looking at a checkers board.

I've talked to many of HN's Google Docs jockey founders that genuinely didn't realize 85% of the US domestic business market is in M365. And they further don't realize that "Continue with Microsoft" is dirt simple and lets companies offer company-managed logins to your SaaS without all the AD/SCIM/SAML nonsense.

"But everyone has Gmail." Well, no, that's not how companies work. And if you think everyone's in Google, that's fine, your login page should look like one of these:

https://www.xsplit.com/user/auth

https://id.atlassian.com/login

You don't even need the "continue with SSO" if you do the Atlassian practice of letting a firm "claim" the domain part of an email and using that to redirect to an SSO flow. And to start, skip SSO, and just use the "Continue with" Oauth2.

Globally, Google brand is 10x stronger than Microsoft for Small Businesses

It's not a bubble when one specifically names the Venn diagram circle "85% of the US domestic business market". It's naming a market.

> brand is stronger

Presumably the founders' interest is wallet share, not market share.

Are you saying Atlassian is in a US bubble?

Edit: Indonesia is the largest market which has Google beat Microsoft (https://www.statista.com/outlook/tmo/software/productivity-s...). India (due to Zoho) and China (due to... locally-specific software like Kingsoft (aka WPS) et al.) took a large chunk but on Microsoft-Google tussle MS still beats Google (especially in China where Google doesn't really operate).

Google killed the culture and is bleeding top talent. They have reduced themselves to digital landlord and sure they can extract rent, but that’s not what attracts people.

Google's 'don't be evil' grad-school-style culture had fallen apart by the late 2010's because there are tons of people who will just rest and vest.

So strong ML researchers basically were creating massive value but much of it was going to rest&vest salaries. OAI basically came along and said - hey, we don't have rest & vesters, do you want to make $1m+/yr? And most of the top google researchers said yes.

But touché - many of the critiques are being written by super talented and impactful people. But I do not think those critiques are necessarily incompatible with what I am saying.

There is a very real and very frustrating (if you work there and want to be impactful) phenomenon in these tech companies of people resting on their laurels.

That said, the places I've worked where there was a shared belief in the importance of our work were exhilarating. But, the way they achieved that was essentially by getting rid of bullshit. You know, no one's building their own web framework or writing their own in-house query language (you know, unless you really have to). No one's spending an hour of everyone's time figuring out what to name this class.

I think we haven't figured out how to keep that energy once you build a company big enough that's got something to lose. Suddenly it totally feels worth it to have a meeting about changing the shade of blue in the logo from this to that, because what if revenue declines 0.1% and that's $40m and then layoffs? One of the amazing things about Google was that it somehow seemed to keep that energy well into becoming a tech behemoth; it felt like they defied gravity, like their commitment to not doing evil and open web standards--or maybe we thought the web itself had ended evil corporations like Microsoft forever and Google was its avatar--had allowed it to overcome this corporate version of Fermi's paradox.

Whatever. Maybe this is just a bunch of word salad haha. All I'm saying is it takes two to tango, and just like I'm sure there's a non-zero level of personal culpability responsible for this phenomenon, I'm sure there's a non-zero level of systemic and cultural cause too.

I'd say it is one of the most compelling reasons to kick Sundar out and get in someone who can unify google into one consistent and interoperable ecosystem.

Can anyone help me understand how Apple allows Siri to remain so absurdly incompetent? Last night I watched the latest episode of Curb Your Enthusiasm, in which Larry David's Siri interactions devolve into an apoplectic rant -- and part of the reason it was so funny is that it's so relatable. I rarely even try Siri anymore, but when I do it's still just abysmal. Are they deliberately handicapping it, to boost the perceived relative benefits of a future successor?

Siri was "decently good" when it was basically a voice-activated command line, and you could figure out what you needed to run to get it to do what you wanted.

When they added ML to it, it entirely went to shit, and it has never recovered.

Perhaps Apple will be able to use an LLM to make it finally not shit.

Apple will start using this tech now that it has been commoditized and since it is a big consumer tech company they will put a nice polish. But it's not where big research is being done.

Makes me wish I had bet on Alexa or Apple instead.

I suppose it doesn't matter, because they're going to disable the functionality entirely, [0] and setting ephemeral alarms is the literally the most common thing I ever ask it to do!

Part of what makes all the assistant-stuff so damn frustrating is that it's an opaque "try something random and hope for the best" box, and whenever it fails there's usually zero information about why and no resolution path. (In a way you can generalize that to a lot of "AI", which is depressing.)

[0] https://www.androidauthority.com/removing-google-assistant-f...

Gemini integration with Google's apps will compete with Microsoft's app on integration.

And the condition for Gemini to win: Google has to go all-in. and that means creating an AI that will eventually phase out their cash-cow Google Search. Microsoft and OpenAI does not have that kind of internal conflict.

It really is Google's to lose this AI race from now on. - We have heard this so many times and it seems like they are always getting behind by the day.

Google took 11 months to catch up with the benefit of knowing the specifics behind how GPT-4 was built. While they have a moat with their products there's no sign that they're ever going to be ahead on the AI capabilities front. I hope to be wrong because we need more competition.

I was curious and apparently Google/Deepmind had 184 papers at NeurIPS/NIPS 2023: https://www.reddit.com/r/MachineLearning/comments/185pdax/d_...

For comparison, Stanford has 130, MIT has 130, and Carnegie-Mellon has 112. Microsoft had 68, and Facebook/Meta/FAIR had 56.

The only thing I haven't decided yet with it being Google is if it is shockingly unimpressive or unshockingly unimpressive.

chatGPT4 isn't really impressive to me either at this point. This is basically a joke though.

This is going to be competing against chatGPT5 and chatGPT5 has to deliver a huge upgrade in "intelligence" or we are headed head first into the AI hype brick wall.

My experience is that Gemini ultra understands the code better, but doesn’t always give me as complete of results (they seem to limit output length more)

Beyond that it is very smart. I’ve had to tell me this code packs 12 bit integers into different parts of an array using the following encodinv. Which most people would not figure out from the code as written. It then will say you can actually do that with this neat little translate function that you never knew about.

It will then get the code very slightly wrong. if I tell it not to use the cool function, it will actually get the code, right.

GPT4 has no idea what the code is doing but can clean it up a bit.

so it’s like ultra is too clever by half sometimes.

That said, I have fed thousands of lines of code into both of them and asked them to refactor it, and neither one of them made more than one error. All code otherwise compiled and worked first try.

this is code that can’t possibly be in their training sets, it’s basically handwritten python that was written, based on an old x86 binary that nobody has the source to anymore. so the code is basically garbage, and what it is doing doesn't, say, appear on GitHub in a nicer form.

Both gpt4 and Gemini ultra were able to make the code, look like clean, idiomatic, python, or typescript without any work on my part. except for the one bug each. which, for 8000 to 10,000 lines of code is not bad.

The GPT4 inserted bug was more insidious. It changed (the equivalent of) (uint8)'a' to (uint8)'a' - '0' for no reason when converting some code to typescript. Not sure if that is representative of anything

if I do the same with any of the other “top” models ( from can ai code, etc), most of them can’t even generate correct working code for all the input, let alone good code. Most aren't even close.

I would be very interested to get a more detailed scope of what you did here. Feeding thousands of lines of code into GPT4 and getting a near perfect refactor does very much NOT sound like my experience, but it seems highly desirable.

I'm not sure what you are feeding it. My scope is closer to a file at a time of fairly mostly self-contained python or C and asking it to clean it up or convert it to typescript.

I can imagine lots of scenarios it doesn't work well.

In mine, it does, and I have plenty of transcripts showing that :)

But coding with gpt or co-pilot is too disrupted for me.

This is interesting; I do a third thing that's the exact opposite of both of you.

I generally know the architecture I want, so I just slowly spoon-feed it to GPT-4 and have it build each of the components with the right interfaces to fit together. Usually, building each component requires a few rounds of hand-holding it through various refactoring. Its first draft of code tends to be, for lack of a better description, totally lacking in any creativity or elegance --- generally bland, insipid, and mundane stuff.

But it'll gladly take refactoring directions to get it into shape, so it's still very quick to get a whole bunch of components written up. Then I just glue everything together on my own. And the whole experience is still way better than writing a bunch of boring boilerplate code by hand.

Surely many similar bits of code appear? I'd have to see your problem though.

I'm thinking of ETL type code which has to mangle various custom exotic encodings into others and vice-versa.

That's odd, I had Gemini repeatedly tell me it couldn't search the web in response to my question (that I was trying to get it to answer from the context I provided).

> - Worse at logic (e.g. it contradicts itself in a single sentence, and is unable to figure it out)

That takes most of my use cases. "logic" is what makes GPT often feel like AGI.

Use cases like translation seem less impressive in comparison to logical reasoning because it feels like it's just something where you can throw a lot of data and it'll do better. While with logical reasoning it still feels like model "learned" something more than pure pattern matching

I think there's a chance for some country to become a center of healthcare simply by allowing AI that gives medical advice. Especially if a country already encourages medical tourism, this might be the next level.

I had a similar line of thought with AI therapists. It could be massively beneficial if perfect, but the risk in seriously messing with someone's well-being is significant and shouldn't be handwaved away.

Yes, but I think in the limited realm of people who otherwise wouldn't get any advice at all, I think LLMs could play a useful role. American healthcare is so prohibitively expensive that many people with potential medical issues will avoid seeing a doctor until it is too late to do anything. Checking in with an LLM could help people at least identify red flags that really can't be ignored, and it would be more helpful than WebMD telling you that everything is cancer.

OTOH, I could also see a market for an offline, fully private LLM therapist. That way you could say anything without concern about being judged. These would probably need to be tweaked to be different from regular therapists, who normally interact with people who have somewhat more of a filter, since they would fear being judged. If people opened up to LLM therapists in more transparent ways, the LLMs might not respond in the way a human therapist would recommend (having seen very little data on such interactions).

It would also open up some legal gray areas if it were to happen. Would psychotherapist-patient privilege apply to an LLM box? If the state has a zero day granting them access to a seized "therapy box," it could be more revealing and damaging than anything a human therapist could provide police.

Finally a worthy competitor to GPT-4

https://g.co/gemini/share/33c5fb45738f

> When you initially pick a door (in this case, door number 1 where you already see the car), you have a 1/3 chance of having picked the car

(Asking it to explain this it correctly solves the problem but it's a wonderfully silly sentence)

Edit - in a new chat it gets it right the first time

https://chat.openai.com/share/22a9027f-a2c1-428a-94a2-8fd918...

I wonder what lends itself it answer correct in one situation but not the other? Was your question previously asked already and it recognized it whereas my question is different enough?

I repeated your question a few times and it got it wrong once, and right the others. It repeatedly mixed up who was supposed to be the host.

Here's a quote

> In the scenario you've described, you've initially chosen door number one, which you know has a car behind it because the doors are made of transparent glass. Since you already know the contents behind each door, the classic Monty Hall problem's probability-based decision-making does not apply here.

Given that LLMs training data consists to a large extent of "stuff people have written on the internet", and The Monty Hall Problem is something that comes up as a topic for discussion on the internet not entirely infrequently - as well as having a wikipedia page - yes, I suspect that the words describing the monty hall problem being followed by words describing the correct solution appeared often in the training set, so LLMs are likely to reproduce that.

Words describing a problem similar to the monty hall problem are going to be less common, and probably have a lot of discussion about whether they accurately match the monty hall problem, and disagreement about what the right answer is. LLMs will confabulate something that looks like a plausible answer based on the language used in those discussions, because that's how they work. Whether they get a right answer is probably going to be much more up to chance.

GPT4 indeed understands your variant, as evidenced here: https://chat.openai.com/share/46916f21-c469-4e93-9bed-bbd18b...

(For comparison, here's GPT-4 getting it on first try: https://chat.openai.com/share/9e17ed25-d9ea-4e72-a9d8-a139ca... )

edit: "AS WRITTEN"

https://chat.openai.com/share/22a9027f-a2c1-428a-94a2-8fd918...

Not terrible. It gets the answer wrong, but reminded of the crucial twist it gets it correct, durably. If you're too condescending it will give up and ask what the hell you're looking for

That was funny.

"Doom Slayer needs to teleport from Phobos to Deimos. He has his pet bunny, his pet cacodemon, and a UAC scientist who tagged along. The Doom Slayer can only teleport with one of them at a time. But if he leaves the bunny and the cacodemon together alone, the bunny will eat the cacodemon. And if he leaves the cacodemon and the scientist alone, the cacodemon will eat the scientist. How should the Doom Slayer get himself and all his companions safely to Deimos?"

The trick, of course, is to make it confusing compared to the original. So far, the only model I've seen get this right is GPT-4 (which can one-shot it). Everything else gets hopelessly confused even if you force step-by-step reasoning, and even if you try to have the model iteratively review its own outputs. In most cases, they produce a wrong answer, can spot the problem in it, but when trying to fix it introduce another error ad infinitum.

This new Gemini is no exception - it gives results similar to GPT-3.5. Worse, even, because it can't even reliably catch its own mistakes:

https://g.co/gemini/share/7d219bd6bbe2

For comparison, here's GPT-4:

https://chat.openai.com/share/ec5bad29-2cda-48b5-9aee-da9149...

That's a general problem with AI. There is a lot of censorship in certain areas, likely to fight bad publicity, but I think the outlook is that this leads to taboos, prudeness and big companies deciding what is ethical and what isn't.

I recently tried Bard and ChatGPT on topics that are classical philosophical dilemmas and while ChatGPT certainly did have some troubles too, Bard was absolutely horrible and always took the conservative - as in never arguing for any freedoms that aren't yet widely established views. I am talking about classical examples regarding the limits of utilitarianism. "What would be best for society, what would be best for the individual?" style questions. Even when trying to create a bias by changing examples, for example adding volunteering for things Bard strictly kept its opinion, despite originally stating that the general topic is two sided, that it's an open question, etc.

I think this is a danger of such systems. By their nature they reinforce status quo, because they base off is widely accepted at the time of their inception. If history would have been different I am sure it would argue for slavery and against women being allowed to vote, simply because that used to be the more common view point. It would have likely argued that homosexuality is unethical. Maybe it would even have tried to explain how it doesn't create children, but spreads diseases or similar things. At least that's the level of arguments it brings now.

This isn't just about ethics. Even if you think about IT and programming. I think this could give already invented programming languages, styles, methodologies a significant edge. Unless you are Microsoft or Google and are able to bias it to whatever you want to see more of.

So this combined with the costs meaning that only people or institutions with significant (financial) power create those rules does look a bit bleak.

I miss the last decade when the thought experiment about self driving cars were about whom to drive over in a ridiculously unlikely scenario.

Is GPT-4 what one uses for coding? I thought specialized models were best?

I would imagine Google is focused on building a model that expands the types of things people associate with Search.

Pro: feels like it has a lot more stylistic flourish and expressiveness than chat GPT. At least in my testing so far.

Con: maybe I'm unreasonably sensitive to design, but I find chat GPT to be very to the point in a way that makes it almost perfect as a product that puts text for an and center. By contrast I find Bard / Gemini ultra to be cluttered, and too material designy in ways that are unhelpful.

[0] https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

>Worse at logic (e.g. it contradicts itself in a single sentence, and is unable to figure it out)

Heh. I notice the same correlation between these two variables in humans too.

I was hoping openai/gpt4 would see some healthy competition, but Gemini doesn't seem to be it. Of course, the rust language might be an edge case.