[1]: https://www.home-assistant.io/integrations/openai_conversati...

It’s neat that you can intermix general chatting with HA commands but you’re probably going to find that the old assist is more reliable for commands. What I do like is that you can use a template as your system prompt so you can provide the state of a number of entities and then ask for them with natural language. That works well.

I have an Alexa/Echo voice announcement system set up and have recently tied that into assist so I can do automations like if the garage opens I prompt for “what is the state of the garage?” and announce the result. Makes it feel more humane than the same plain announcements all the time.

I mean honestly, how is it possible Amazon, Apple, Google and Microsoft[0] all keep screwing this up for over a decade now? I literally spent 15 minutes hooking up GPT-4 to the Home Assistant integration, and I was able to semi-reliably[1] control actual devices[2] like air conditioners and smart lights, in a completely natural and ad-hoc way, by talking to my smartwatch on the go, or to a phone, whatever was more convenient at the moment.

It's a really magical experience, a step closer to Star Trek reality. And what makes it possible is not just LLMs being able to deal with natural language, but more importantly, "bring your own API key" model allowing to cut away all the bullshit that FAANG assistants are stuck in.

--

[0] - Ever since they dropped MS Speech API in Windows, and did the Cortana thing. Some 15 years passed, at this point, and I'd still prefer to work with the Speech API than to touch any of the FAANGs' voice assistant - it worked, and worked off-line!

[1] - Works ~90% of the time; some 5% of the time the voice model (from Home Assistant Cloud) misunderstands me, and 5% of the time the LLM gets confused. It's still worth it, because I can actually talk to it like to a person, without thinking of style or grammar or magic keywords.

[2] - Which, given the level of integration of Home Assistant companion app with the phone, can be easily turned into an equivalent of on-phone voice assistant that can do more than the one I got from Google. Critically, there are ways to couple Home Assistant app and Tasker, so it's not hard to make it do arbitrary things on your phone. And, if you don't like low-ish code Tasker experience, you can trivially shell out from Tasker to Termux, at which point sky is the limit. Point being, an enthusiastic non-developer with minimal tech aptitude can beat Google and Apple at the voice assistant game today.

Since that seems to be an increasingly niche desire (at least as far as the product managers are concerned), I've been looking more and more seriously at setting up my own local voice assistant. My main barrier has been hardware—the mic arrays in the Home devices are surprisingly good and hard to beat with cheap off-the-shelf components, and you need a good mic for good STT.

The hardware is really well done but the software is either over or under-engineered to a stupid degree.

Teach a man to fish!

https://www.theverge.com/2024/8/14/24220178/halide-camera-ap...

If you actually peak under the hood they just pass through weighted selectors, no different than a switch statement

Nah.

Smartphone cameras stopped being shitty a while ago, long before AI and computational photography hacks.

What you mean to say is they without AI, you'd know sooner that the smartphone maker put a cheap, shitty camera in your "premium" phone.

>People do care about that. AI isn't just LLM chatbots.

Yeah, it's also fake image generation featuring humans with a funny number of fingers.

What AI isn't is a camera.

I'm having a horrifying realization that all of my pictures are "fake" in the sense that they don't match what I saw/experienced. Maybe it's time to get back into Polaroids.

I'd caution that judicious/proper post-processing is actually needed if you want that result, because of the differences between the sensors.

Your human experience comes from many small pictures taken by a set of lenses panning across multiple points in a scene with constantly adjusting exposure times and focal lengths, all biologically composited into what feels like a single moment.

Trying to fully replicate that with a single artificial picture is going to be deficient in certain ways.

---

Separately, a pet peeve of mine: Too many people have been subtly brainwashed into conflating the "like I was really there" with " like a Hollywood film camera was really there." Then the next thing you know your medieval fantasy game has lens flares in it for no good reason.

But knowing that the phone does a lot of software tweaking to get a picture to look similar to how good a full camera is made me want to switch. I think this was around the time that article about Samsung basically replacing a photo of the moon.

But those pics will be probably further from perceived reality than those enhanced by software (lets not get retarded here and don't brand every data processing as 'AI'). Distortions not only of barrel type, waning brightness towards edges, moire, heavy vignetting, tons of noise, over/underexposition, maybe some dead pixels... thats not how I see my days go by.

Some personal examples of how I find it more useful (love to hear yours) and fun to use....

- Wanted to go on a hike an hour away from where both my friend (lives two hours west of me) and I live. Asked GPT what are some good hikes an hour drive away from both of us to meet & hike. With Google I have to do Many searches where GPT just provides the answer right away.

- I count calories and eat out everyday. GPT knows the calories of everything i eat as I eat at chains mostly (Cava, Panera, Starbucks, Chipolte). I tell it via voice what i just ate for my 1st meal, it calculates my calorie count and later I'll tell it what im having for my 2nd meal. It can also recall my calorie count from days ago. It does all this quickly vs. Google i'd have to do oodles of searches.

Usually Im using GPT the most when driving via voice and unlike Siri, GPT understands me and i can have whole conversations with it to get things done while driving.

I can't speak to your personal entertainment experience, but AI chatbots are generally a slower, less accurate way of getting information than a google search. (Though Google polluting search results with a big, often inaccurate, AI result at the top narrows this a bit.)

These blanket statements lead to flame wars.

For me with the two research examples of using GPT I gave/use it for the information is accurate. My friend and I have driven an hour away (for both) a few times (different spots) and hiked. Same goes for calories GPT has in it's knowledge base for well known chains. If it wasnt a chain restaurant GPT might not have it in it's knowledgebase or possibly have it wrong.

To say it is faster to get information from Google than the latest update to Sonnet is simply absurd to me.

I might have even agreed a few months ago but certainly not now.

Wake me up when I can say things like…

Hey, Google, are my custom license plates ready for pick up at the tax office?

Hey, Siri, ask my doctor to refill this medicine.

Hey, Alexa, how many charging stations are broken at the gas station on 16th street?

Hey, Google, why is this plant dying?

Hey, Siri, why are there so many people in my neighborhood today?

Hey, Alexa, did anything ever get done about that story in the newspaper from a couple of years ago about the Chinese slave labor being used to grow pot on illegal farms on the Navajo reservation?

"AI" just doesn't have access to the information required to do anything interesting or useful. And because so much of its information comes from the web, which is already so polluted on certain subject (gardening, travel) as to be useless, the AI becomes useless.

Imagine an AI that popped up when you are reading something and warned you that information is false.

For instance, imagine something like https://theconversation.com/can-ai-talk-us-out-of-conspiracy... helping people discern about news and propaganda.

try asking chatGPT to only give you true and accurate answers and not make anything up.

Now, more seriously, it'd need to put together a coherent argument and back it up with reputable sources, as just citing sources is very ineffective. The article I cited gives more details on possible approaches to that.

With your argument, the problem happens when given person goes to Fox news in the first place. Selection of the source has already been made, with its biases. Not much you can do or expect after this point.

Also, who curates the curator? Again an age old problem with no real, long term working solution in sight. No, you should not expect some statistical model to hold your hand through vast internet, while giving up any form of critical thinking, reasoning, or I guess any cerebral process altogether. Ultimate laziness. Since we know how much money there is in diet fad business, its safe to say this above will find its non-tiny desperate crowd.

There are no sides in objective reality. You might offer competing hypotheses and evidence for those, but we don’t need to do that to know that climate change is real, that it’s caused by humans, and that vaccines work.

The less fraught one is warning users that they’re being scammed: https://arstechnica.com/gadgets/2024/05/google-wants-ai-to-l...

That's even without considering the efforts that would be made to undermine any system that showed any effectiveness. A lot of misinformation, probably most of the stuff that gets traction, isn't random, it's serving a purpose and being pushed for that reason.

It's an unfortunate fact that the people most sick will refuse treatment until it's too late. Seen from the inside, insanity looks like lucidity.

It's still pretty terrible though

AI is anything automated it seems, and now they’re being subcategorized into niches as to what they do, e.g. “Agentic AI”, “LLM backed AI systems” etc.

If it’s not real intelligence then it isn’t really AI, and I wish the world at large would call it out.

LLM, Machine Learning, Neural Networks etc are all great but none of them have true spontaneous intelligence or learning ability.

Please, someone point out how any of these systems have organic spontaneous learning ability for a subject it was not pre-data seeded on. This is a generally accepted measure of higher level sentience as far as I’m aware

Hence the predicated “artificial”, and hence the downvotes you are currently receiving.

Your message is largely, if not entirely, a strawman.

There has been considerable success in programming computers to draw inferences, for example, but not actual reasoning. You can mimic some forms of reasoning but you can’t take one ML set - like recognizing photos with mountains, then expect it to correctly identify a similar geographical element - a hill. It can’t do that. It may correctly identify that it’s not a mountain but that isn’t the same thing as actually learning it’s similar to a mountain but not the same, which would be a rudimentary definition of a hill that an intelligent entity could conceivably use if it knew what a mountain was but not a hill.

Machine Learning was always a more honest place to have This discourse. I am indeed pushing back on the idea that we should be calling ChatGPT or anything like it intelligence.

It’s Machine Learning, clever algorithms, Large language Models, among other things, that are trained on ways to mimic certain aspects of intelligence, but it does not actually possess any real intelligence. Look at the LLM hallucination problem for example. It can’t be self corrected because it’s not an intelligent system.

Moving the goal post on what AI means (and pushing AGI as some new goalpost) is disingenuous, and relatively recent.

I’d care not if it wasn’t for the fact there is so much misinformation around capabilities and the future of AI, that it’s already negatively crept into policy making for example.

Seems to me the problems are (1) the "assistants" aren't anywhere near good enough to be trusted to make the right decisions, and (2) a trustworthy assistant isn't compatible with the adtech business model, so it's unlikely facebook or google would produce such a thing.

Yes, why wouldn’t I? I could also give them parameters like “if I’m more than 20 minutes late please re-schedule this” or “if my flight is delayed please let everyone know it’s delayed”

Why wouldn’t I do that? Presumably the person hired is competent to make determinations within parameters specified.

I could also let them know when it’s inappropriate to do this. Again, they should be competent enough to discern the differences between when it is and isn’t appropriate.

This could honestly be done by an algorithm if you give it the correct inputs and outputs and it could be fed updates, the only real limit is the fact that some of this isn’t exposed via an API either in a timely fashion or at all

That years of training is what we are missing. I don't think modern AIs can be trained in the way the assistants of old could be, at least not yet.

Being able to collate the requisite inputs from outside sources is the real problem. If you can’t do that reliably it’s simply hard to build an algorithm around it. Flights for example would require your calendar program to reliably pull data from an API regarding the flight information that is current and effectively real time. That’s the actual hard part, and this expands across services.

For all the advances we have made with computers and smartphones in particular they suck at meaningfully exposing a way to collate data sources and create actions around them reliably

Having one running locally helps but it's still necessarily storing information that you might not want to have stored where someone could potentially retrieve it, either via some sort of exploit or by forcibly compelling you to give it up.

The CEO would trust the personal assistant to do this if they have a deep trust in the assistance competence. They would also need to know the assistant has a deep enough understanding of the their preferences to not do something they don't like. AI can mirror that.

More importantly though there will be consequences if the human assistant makes a big mistake and books the wrong flight. They would have to take responsibility for the mistake.

The LLM is always just going to write in text it is sorry if it makes a mistake. That is never going to be good enough for anything of consequence. The LLM would practically have to be omniscient in a way that is not going to be possible in a world filled with uncertainty.

So much of human activity is built around the network of trust that another human takes the blame if something goes wrong. So much activity involves coin flips and that someone takes the blame when the coin lands on heads but we bet on tails.

Today, this is nearly available, nearly. Probably only something Google/apple can realistically offer. Apple “intelligence” has started to read your notifications and rewrite them for you, so it shouldn’t be a big leap to listen for a United App notification and decide it’s urgent enough take action. Should be “trivial” for Google to do as well, and they could even run it server side to help without a phone present.

2. How does it know which contacts to contact? Does that acquaintance you talked to for some professional reason need to know your flight got rescheduled? What about that travel agency you talked to last night to confirm the flight?

If they had an app then an AI assistant should be able to tie things together. Where things seem to be going is apps provide an intent-based API wrapper plus UI widgets to interact with it. That way assistants can operate them too.

I use reminders often so I suppose it is a low failure rate.

But, when they first made the change to Gemini I had to switch back for a few weeks/months before it could set reminders properly.

ETA: The joke is I don't even have Google Tasks installed, so I am not sure what effect those Tasks items would have. They might only surface on the side pane of Gmail.

True, but it is an external service with all the privacy concerns that entails. I appreciate eg. Apple pushing local AI but at the same time I don’t think it needs to ship with the OS. Just provide an AI API apps can hook into then I can decide which models I want and where they run.

Apple's approach of using current-boom AI to help you navigate and digest your own private trove of multimedia content (photos, videos, apps, notes, structured data, etc) is absolutely useful to people as well, and for some of us, one of the only personal uses of this AI that seems compelling at all.

I'm much more excited to have help finding that goofy picture of my cat by describing what I remember, so I can share it with a friend, than I am to have some chatbot dialog about entry-level Python with a hallucinating parlor trick.

But these features have to work, and work well, and work fast, and be widely known to work, before they'll really win the market. But that's going to take a minute and it might not even happen.

Hasn't Google been doing this forever? I can search random things in my photos (like pictures of an old car I owned).

The new LLM-ish tools promise that users can be more vague and casual in what language they use and more elaborate in how specific they mean to be; and that the queries (and operations) can span more diverse data sources.

https://blog.google/products/photos/ask-photos-google-io-202...

I have my sister’s dogs named in my google photos library. Every time I a take a picture of either dog, they are automatically tagged and added to a shared album I set up for my sister.

I have nieces and nephews with photos from newborn age to 10+ years old, and it has managed to organize them across their growth and ages. It’s incredible. I can search for “<niece name> <vacation area>” and get every photo of her on a certain vacation to make a family scrap book.

Apple photos search and tagging is pitiful in comparison.

But that whole find me something on my mutliple gigabytes of storage on my device account definitely seems like the mass appeal

This is such a mundane use of AI, but unsurprising Apple would sell it as revolutionary.

That is in contrast to a lot of fancy AI demos which are a great party trick, but fall apart in actual usage, with their reliability being “maybe it will work this time, maybe it won’t, just keep retrying :)”.

Apple is pretty well-recognized for usually being a bit late to the party, but at least delivering stuff that’s polished.

Just look at this thread of people sharing how Gemini broke all their commands and automations. The Apple Intelligence Siri on the other hand works just fine (even if new features are arriving slowly).

The difference between whether someone has 4 or 5 fingers, whether they're holding a gun versus a random object, or whether they're mixed-race or caucasian, all seem like they would be pretty important things. Likewise, car number plates, signage in the photo that help identify where it is, metadata of the image itself (often more useful than the image), are all incredibly important. All of those are things that AI is absolutely terrible at lmao.

I'm not sure how a bunch of probably-correct keywords that miss a lot of the important aforementioned details in the image is more useful than the image itself, or it's metadata. Both of which would be lost. My point applies equally with respect to image classification, too.

Literally every thing I used it for got answered via "I cannot do that yet" after it randomly opted me into that. Pure garbage.

I'm not, I pretty much just accepted that Google doesn't care about usability whatsoever and haven't prompted it in a very long time.

To be clear, the only time I've ever used it was via "ok Google" in contexts in which I'm unable to interface with the phone directly, i.e while driving. If it doesn't work you'll learn that you can't start driving before queueing the navigation anymore. The voice assistant was a nice feature, but not important enough to waste my time trying to figure out which feature they opted me into and how to get back out of it.

That said, at some point it started working better, but there was a good 6-12 months where it was a tire fire.

This assumes that the capabilities and use cases are unchanged. Yes, for the AI features available today, I suppose ChatGPT can do much of it -- I wouldn't know because it's not interesting or useful to me, so I don't use it.

But: If I'm deciding whether AI features are important to me in making a decision to spend money on a future phone, it's those future AI features that I will be assessing.

95% of my everyday needs for an external intelligence (besides my own) are covered by e-mail, text, and phone calls with other humans, with a trivial portion covered by nascent AI features. As this changes, and AI gets more capable of replacing human intelligence in these interactions (TBD if this happens in the next smartphone generation, or the next human generation, or further in the future), then I will /very much/ care that the electronic device that I use most often day-to-day has access to these capabilities, and will very much use access to those capabilities as part of deciding where to spend my money.

Weird! Mine can still create reminders, as well as set timers and alarms.

Also, being able to talk to my mailbox asking questions about subjects mentioned in my e-mails would be a huge time saver.

Imagine a purely local Microsoft Recall-like thing that could answer questions about things you saw, or that read the news articles you went over quickly and answer complicated questions about them much later, at a time you just started to regret not having bookmarked it for future reading.

Like when a timed alarm is making the phone buzz in my pocket while I'm driving, I'm telling it to silence the alarm, and the response-voice is regretfully informing me that there are no alarms going off right now.