- rainbow dripping arrows

- angrily pointing arrows

- flame-surrounded text boxes with particle effects

- the ability for the agent to play airhorn.wav at max volume, overriding existing volume or mute settings

Maybe isoprophlex will add that to his PR!

It's already got the power to be obnoxious at you

https://hyperties.org/cabinet/symelec/

PIXIE and FORTH on PDP-7 Cabinet Emulator with Type 340 Vector Graphics Display:

https://www.youtube.com/watch?v=lo8kdY-5i6c

She doesn't have to call you anymore.

That would be a godsent for those of us with aging parents.

Right yeah? I mean I have freagin remote desktop installed on all my parent's devices. I am the agent for them every time they call frantically for help.

It's Time to redesign the apple genius bar for the modern aging boomer using this.

For web apps, WebMCP is a way where you can make a frontend way more discoverable to an LLM rather than that it is going to read all your code. I sometimes create this in my apps at home and sometimes in my apps at work and it makes LLM assistants way more usable. It also helps if they can highlight certain things of a visual.

We humans can do this too on paper. We can point at something, we can highlight something with a marker. Why not give an LLM these capabilities as well?

What terminal / agentic workflows spawn the demonstrated dialogue boxes that require the User to click/reject the action? Aren't most such flows actually inline UIs? And finally, if the core issue is that these confirmation boxes are tied to the terminal that triggered them, which does not autofocus, what's the point of said "big arrow" that is also lurking behind without focus?

If the said "big arrow" automatically gains focus, isn't the real fix here to just make the dialogue boxes themselves gain focus automatically? Both are similarly disruptive anyway.

"Teach me to use this app myself" kinda stuff. Guiding agent rather than doing agent.

Honestly, @franze, if you're reading this, maybe update your screenshots with that bent (showing us an agent in tutorial mode on some complicated app)?

We built something to help with that [1].

[1] https://www.happysupport.ai/en/in-app-messaging

Does one need 4 programming languages to draw something on a mac?

Grim. If you're just there to click sudo buttons for the bot, you might as well give it root access and be done.

    --norms-taboos-off

  %  @(#)handy.ps
  %
  %  Handy Pointer
  %  Copyright (C) 1989.
  %  By Don Hopkins. ([email protected])
  %  All rights reserved.

PSIBER Space Deck and Pseudo Scientific Visualizer Demo:

https://youtu.be/_fqCeuue5Ac?t=213

The Shape of PSIBER Space: PostScript Interactive Bug Eradication Routines — October 1989:

https://medium.com/@donhopkins/the-shape-of-psiber-space-oct...

~guywithnopowertodisallowit

As a human stochastic parrot, don't you find it embarrassing and humbling to be so easily outdone by an LLM?

Whenever I see these vibecoded apps, my move is to just do what you've described: point an agent at it and tell it to reproduce it (usually with my own little customizations). It's a bit absurd, but I don't trust that they vibecoded the app as well as "I" could lol.

Hopefully someone smarter than me is working with trying to develop a new programming language given what we know to be possible with LLMs

The open models that are chasing the frontier labs will stop being open once things slow down and there is less incentive to undercut the front runners. Time will tell if GPU compute gets cheap enough to run stuff locally.

I think the next generation of exploit will be putting up code all over the internet that does x, y and z and then when asked to one shot whatever that code does the agents bake in the exploit that was indirectly included in their training corpus.

Everything your building is now streaming through a third party who could steal it, the same way Amazon steals open source software and repackages it.

All you need to do is look at palentir's forward deployed engineers. Once you let a third party observe your processed, you're cloneable and replaceable _as an entity_.

I think we can look at our own human psychology as it developed into social groups. If our brains were obviously deterministic, it would be easy for people to prey upon us via structured inputs. Some level of randomness in our thought process is almost evolutionarily required to avoid becoming part of the larger super structure, similar to an ant colony.

So these AI companies are definitely more dangerous than just the "AI is super intelligence". They're basically a intelligence platform everyone is wiring into.

That's today. Tomorrow, they'll be run by an MBA which is the enshittification.

If you can make a facebook clone it doesn't mean you can steal facebook's lunch. there are exceptions of course.

Now even if you want your idea to compete with all the other todo apps with your special sauce, youre giving a third party your recipe. That third party ostensibly has no reason to steal it, their business model does.

So, sure, if your app is a cult of personality and nothing special, i admit it doesnt matter.

But the number of software home runs is dwarfed by all other auccess on individual products

At their scale I'd expect it to be only worth stealing potential home runs and even so, imagine they did this a couple of times, how would that make them look? Do you think it would be worth the bad blood and mass exodus that would follow?

Not really. Ants are superorganisms, they share more than half of their DNA with each other. You could think of each ant as being like a cell in a human.

So atleast pare of our collection of genes has added value as with random decision factors rather than find some supergene that everyone benefits from.

https://github.com/seb3773/ntfs-repair-rfc

https://github.com/Kotivskyi/screenshot-tool

I think it was more popular around the beginning of the year to mid-year

Again, I think it heavily depends on if people are writing comprehensive specs with sufficient detail.

It makes sense to not trust AI models with the ability to read and alter your screen for many many many reasons, but the developer of the tool knows that as well. A tool that lets your ai do something doesn't also let it do anything.

So, as always, it's a question of if you trust the software producer. You're giving them the ability to draw on your screen too. Would you have the same fear if there were no AI involved?

For stuff like this I imagine AI misbehavior as a novel failure trigger not a novel failure type. It can only do the harm that the software was able to do on its own already.

> "It's this window, not that one." You have 14 Chrome windows. The agent knows which one it means: --window "Google Chrome:Pull request". It even picks the right tab: --app "Google Chrome:Pull request".

> Remote help. "No, the other gear icon." Point at it instead of describing it.

I found this awesome vibe-coded Pixel News Network project recently. How fucking cool is this?? Would never have been created otherwise. https://pnn.watch/

I actually somewhat enjoyed the times when vibed projects were a broken mess, lots of accidental comedy in that era.

Why does this need to be a skill? Literally just ask Claude to “call out actions in these screenshots with numbered stylized arrows” and you get the same thing without burning tens of thousands of tokens in context.

> burning tens of thousands of tokens in context.

~1400?

> Literally just ask Claude to “call out actions in these screenshots with numbered stylized arrows”

Is there a macos native thing it uses? Or are you talking about screenshots? This draws on your screen instead.

A big arrow to an interface element which ... has a label that explains what it does?

So...a label for a label?

Helping people deal with bad UX.

Everybody who has worked in a traditional corporate environment has encountered poorly tutorialized, poorly labeled, counterintuitive user interfaces where the only way to know how to get something done is to have somebody sit over your shoulder and tell you which buttons to click.

Oh, you want to figure out which OIM entitlements your coworker has that you don't to debug an access issue? You just need to know that you have to click the "compliance" button to get there. Why is it called "compliance" when what it actually does is let you export comparisons from the user tables? Because fuck you that's why. Memorize it.

An AI assistant that can read the procedures for you, or sometimes even read the source code, or the website code, can show you how to accomplish something that the UI on its own fails to do.

For the first experiment, Claude stepped me through how to edit its settings.json to remove warnings from its configuration: a little clunky as I had to respond to Claude's CLI prompts to get it to auto run.

In the second experiment, Claude walked me through how to use Apple's Compressor.app to speed up a video. It walked me through dragging the file into compressor and clicking on the appropriate UI elements to do this.

So, one use case for the bigarrow skill enables Claude to step users through a relatively complex UI to accomplish a task.

There are likely more efficient ways to accomplish the tasks at hand (e.g. ffmpeg for the second experiment), but this use case is interesting if using a GUI tool is required.

To me this feels like it takes away from what the human is supposed to do (read, understand the consequences of the action, then.. consent or abort)

There is a reason your AI Agent won't automate these clicks for you

But I feel dumber just by looking at it. If this is how this will look like in 10 years, why to make desktop at all. Just connect mic and speaker to your PC or talk to the phone: 'I need new pair of socks. Order 10 for me. In your favourige color it will be 21.37. Should I charge your credit card?'.

It is not like most of the people enjoy computers. I am pretty sure they do not. They just need them to operate systems they need: government websites, banks, maps, restaurant menus etc. If some agent will do that for them, why bother looking at screen at all? Rich people have it with their own personal assistants.

Whenever I use web tools that don't have "deep linking" [0], I love to throw out this quote:

"Have we thought about using hyperlinks? They are these SUPER useful things that were invented in Switzerland back in the 1990s."

0 - https://en.wikipedia.org/wiki/Deep_linking

I absolutely loathe this phrasing now. I don’t even know what part of it is good or notable.

Brilliant. This is exactly the kind of content I seek when I visit Hacker News.

Truly art.

It's just the epitome of living in an era where code is so cheap to generate that we don't invest any effort into even fully fleshing out the idea. It's just "Claude, make me a program that draws arrows on screen". And HN, I guess, is dominated by two groups: one still stuck in the mentality of "code = effort" and trying to recognize that; and another stuck in the mentality of "AI = cool".

Those who code will be left behind. I code. I don’t like it. But I have to face reality.

https://laughingsquid.com/yahoo-neon-billboard-in-san-franci...

(OP, nice project! Sorry for my rant.)

But it's a subpar experience compared to just opening github on the browser.

People have been saying "we don't need docs", "nobody reads docs", "they'll figure it out". They experience the pain of a new employee onboarding but never treat this as a signal for the user experience. The user can't just walk over to the developer's desk. But luckily we got stack overflow, blogs, and Google. Somewhere someone explains it! The problem became finding it!

But hey, why do things the "hard" way when you can just throw money at the problem? Or better yet, dismiss the problem by calling the user an idiot.

You're right. What a time to be alive. We've created a world where this product is useful. And not even to because dark patterns exist, but simply because we've always wanted to avoid taking a step back and thinking or getting an outside view. Because it isn't obvious if a robot has to teach you what button to press. If it does, you probably should be ashamed (there are, as always, exceptions)

That said, I do think there's still a lot of utility you this. Those exceptions aren't uncommon. This'll help people learn programs like FreeCAD or Blender, or whatever. Where the complexity is naturally high. But also I do think we should recognize the silliness of many problems that this does solve that shouldn't be problems in the first place.

Garage sale signs are a good example of this. The person making the sign knows what it says, so they can "read" it from farther away than someone who doesn't already know what it says.

Even the data centre using this probably uses less net energy, and costs less, to build one of these dumb arrows than the aggregate sum of the energy used to keep you alive while you Googled for the answer. (Including heating/cooling the building, powering the equipment growing the food you eat, etc.)

Nonetheless why not let an LLM make a guide and use that in the release, I don't see how that's wasteful

Obviously I'd prefer to ask specific questions to my expert human friend sitting at the next desk over, but sadly they don't exist.

I just hope there is a future where this tech exists without such big downsides.

I don't see how this is anything like that. The point here is to make it easier to give the user indications which has been a staple since forever, hell this is so much of a problem that there are entire solutions engineered to letting one user take control of the computer of another user through the net.

Beyond that, I'm sorry but looking for help in the context of videogames is extremely prevalent. That external datacenter to run the tutorial levels is sometimes called "Youtube" and others "Wikia".

An agent is not at all like this. The freaks who made this disingenuous machine don't even know how or why it does half the stuff it does. We're all just throwing shit at the wall and seeing what sticks. Absolute nonsense take my man, I'm sorry.

Indeed, what a time to be able.

When we got a form of such search, it searched all of the internet alongside with the functions of your software on your computer. You had to tell it manually what software you have and when it found something you had to fish out the description of what to click to get the functionality you are seeking and then click through some dumb ui to actually invoke it.

Agents are the first thing that can lead you directly from "I know what I want my computer to do." to "Actually doing it."

Success of agents is founded on the profound and sustained failure of all of the software designers and developers ever.

apropos ?

apropos switch desktop to dark mode

and

aprops set dns to 8.8.8.8

   $ apropos dns
   [...]
   resolvectl (1)       - Resolve domain names, IPV4 and IPv6 addresses, DNS resource records, and services; introspect and reconfigure the DNS resolver
   [...]
   $ man resolvectl
   [...]
     dns [LINK [SERVER...]], domain [LINK [DOMAIN...]], default-route [LINK [BOOL...]], llmnr [LINK [MODE]], mdns [LINK [MODE]], dnssec [LINK [MODE]], dnsovertls [LINK [MODE]], nta [LINK [DOMAIN...]]

           Get/set per-interface DNS configuration. These commands may be used
           to configure various DNS settings for network interfaces. These
           commands may be used to inform systemd-resolved or systemd-networkd
           about per-interface DNS configuration determined through external
           means. The dns command expects IPv4 or IPv6 address specifications
           of DNS servers to use. Each address can optionally take a port
           number separated with ":", a network interface name or index
           separated with "%", and a Server Name Indication (SNI) separated
           with "#". When IPv6 address is specified with a port number, then
           the address must be in the square brackets. That is, the acceptable
           full formats are "111.222.333.444:9953%ifname#example.com" for IPv4
           and "[1111:2222::3333]:9953%ifname#example.com" for IPv6. The
           domain command expects valid DNS domains, possibly prefixed with
           "~", and configures a per-interface search or route-only domain.
           The default-route command expects a boolean parameter, and
           configures whether the link may be used as default route for DNS
           lookups, i.e. if it is suitable for lookups on domains no other
           link explicitly is configured for. The llmnr, mdns, dnssec and
           dnsovertls commands may be used to configure the per-interface
           LLMNR, MulticastDNS, DNSSEC and DNSOverTLS settings. Finally, nta
           command may be used to configure additional per-interface DNSSEC
           NTA domains.

           Commands dns, domain and nta can take a single empty string
           argument to clear their respective value lists.

           For details about these settings, their possible values and their
           effect, see the corresponding settings in systemd.network(5).


   [...]

People can do and do better with googling. Proportion of people who found the answer to that question through apropos to people who found that answer googling is probably less than a million to one.

Press super and type “dark mode”

What happens?

I’m not sure I understand the desire to control computers with conversational language. Computers are a collection of on off switches. I personally would much prefer being able to control exactly which on off switches to switch, instead of just telling the black box to do something and for it to make no mistakes. It just doesn’t strike me as a good idea to give the incomprehensible matrix weight the ability to make decisions about what to do on my machine. There’s just far too much ambiguity for my comfort here.

I mean, Android settings has global search because the settings menus for each Android phone is different (wtf) and some settings aren't even accessible except searching.

Actually they’re the second thing. The first thing was studying computer science.

/s

Do you remember when PCs used to come with a completely soup-to-nuts tutorial that would talk to you like you had never seen a PC before? Things like this: https://www.youtube.com/watch?v=3ScS4OYDfHE

This kind of baked-in interactivity could really help in a training or disability context.

Wrong target audience I guess, but I’m surprised by the sheer short-sightedness on display here.

I remember taking your SEO course many moons ago.

I don't know why anyone would ever willingly want this.

I'm wondering if the AI has been trained on Franz. Which would also explain why the Terminator ends up with Manner Schnitten.

the first versions looked way to professional and lost that "yes, that, exactly that, i mean it" arrow feeling that i am usually going for