It can?! It does that?

https://github.com/openai/openai-cookbook/blob/main/apps/web...

This is only the case for now, however. I expect this to change in favour of LLMs as time goes on and they become easier to deploy and better at their job.

It seems to be at least a not-yet-true claim, but let’s ignore that for now. It’s interesting if LLMs actually could do this. As I understand it, LLMs are trained on texts and source code among other things. But lacking… let’s name it a reasoning apparatus, can they really look at a DOM tree and tell what it is/does? It’s not a text, and barely can be a well-structured source code that was ever discussed (it’s a result of either bundling or componentization). This is almost on par with “LLMs will look at any .exe and be able to integrate with it immediately”.

Yes, if you encode the DOM as a list of options for ChatGPT to choose from. In fact I developed a proof of concept of this for a client. https://jarvys.ai/ although they seem to have pivoted from automating just the browser to automating all software.

Not sure if I understand this, does it mean you have to pre-cook DOM in a specific way? If yes, then isn’t the answer to my question “no”, like “no, it can’t take any site and use it as is”?

So if you assume that you start on google.com, then your options are like 1.) Input with name "search", placeholder "search anything", value "" 2.) Button with label "I'm feeling lucky" 3.) Button with label "search"

Obviously, doing just one of these doesn't achieve the objective - it just needs to pick which one it thinks has the most "value" for completing the objective. If you repeat that enough times, then it can actually do what your overall goal of the session was.

I'm just giving a simplistic answer, and if you implemented only what I've written, then it's going to get stuck in a loop more often than not. But that's the gist of how you could encode the DOM into something that GPT can interpret and make decisions/take actions based on.

In fact, the problem of HATEOAS is exactly what LLMs seem to be good at - inferring the interface at runtime, from dynamically received metadata. This should even be easy to try in practice today - HATEOAS can be trivially mapped to the "function calling" feature of OpenAI's GPT-3.5/GPT-4 APIs.

A good example would be a misguided approach at making a bunch of labels with values that are aligned. Someone told this poor developer that <table> is bad, so they figure hey, let's use CSS to lay it out. They make a dictionary of the key/value pairs and iterate over all the keys in the first column into the first div and then output all the values in the second div.

div - label 1 - label 2

div - value 1 - value 2

If there's 100 key/values it's going to be hard for a human to figure out which value is for the 76th item, and LLMs have proven to be very bad at indexing problems like that so I wouldn't expect it to be a better story there.

(Not saying this wouldn't work in some cases, just couldn't be a general solution given the crap out there)

The LLM will looks at any .exe and determine if it halts.

https://en.m.wikipedia.org/wiki/Halting_problem

> The halting problem is undecidable, meaning that no general algorithm exists that solves the halting problem for all possible program–input pairs.

It feels like at that point you need an AGI that's superior to humans and can figure out something we can't, rather than an LLM that more or less repeats human knowledge, not even necessarily reasoning about it.

Do note that the halting problem is fundamentally true, no AGI will realize some new way around, unless are mathematics are flawed to the core.

So we will probably never practically solve the halting for LBAs due to the fact, the quest for this AGI that should be able to solve this, is not just day dreaming, it's rooted in the theory.

An LBA is effectively "just" a Turing machine that has a finite tape.

A typical current computer is an LBA only if you disallow all IO of any sort or bound that IO and include it as part of the system you analyse and so fix the values which will be provided as IO, which of course is a very unusual situation, and so that constraint does not really make the halting problem more tractable in situations we usually care about.

If you can represent a program with an LBA, it is by definition a context sensitive language and decidable. You could also show it by making a equivalent gramma for the language, that accepts the same input as the program. This gramma must be constructed in a certain way, and then you know it is a context sensitive language.

We can certainly look for AGI that can do better at deciding the halting of decidable programs, but even for current computers the general halting problem is undecidable without adding artificial constraints.

So, no, you can not construct a machine for every conceivable input without imposing an artificial constraint on the input size.

Put another way: For every tape you construct and analyse, there is a tape one segment longer that might contain a symbol that can alter the outcome.

  hailstone :: Integer -> Integer
  hailstone n
    | n `mod` 2 == 0 = n `div` 2
    | otherwise      = 3 * n + 1
  
  collatz :: Integer -> Bool
  collatz n
    | hailstone n == 1 = True
    | otherwise        = collatz (hailstone n)

But yeah, it is not even trivial to say whether it has a bounded max memory, so in case of an arbitrary precision int type, it may not be LBA, but Turing?

We can actually do it in O(1) space complexity in exchange for higher time complexity.

For i in range 1..n, compute n_i, the ith number in the hailstone iteration, and then continue the hailstone iteration n_(i+1)..n_n to see if n_i is equal to any of them. This takes us to quadratic time complexity, O(n^2), but only requires storing a single n_i at a time for cycle detection.

But then again, if you actually loop for n_max iterations without crashing, you had to reuse a number somewhere, so the explicit cycle detection isn’t really important.

Humans are also limited by the Turing model’s limits, we can only ever determine computable functions as well. With all due respect, it is stupid to assign more capabilities to ML than what we know is fundamentally the limit..

Of course ML has use cases where traditional tools are less fit, my gripe is the hype-based anti intellectual nonsense that often surrounds it. They are no magic tools, the fundamental limits these giants of math/CS discovered still apply to them and we can save ourselves from a lot of pain if we don’t bother solving unsolvable problems.

The bitter lesson is that Messy AI is better able to cope with Messy World Problems than Neat AI (by light-years at this point), not that it can hack Neat Problems.

Why would it be extremely useful?

Being undecidable is a hard limit, not something that requires better algorithms. Another phrasing of it could be: No finite program can decide it. (And what even is an infinite program? Not something we can run on any current computer.)

If an AI is itself a finite-sized program, something we run on computers, it cannot possibly solve the halting problem.

And _any_ non-trivial property of programs is undecidable, so an AI "integrating with any .exe file" isn't really meaningful. It's just words.

The complexity of the program trying to decide is not really the issue. An "infinite program" could add an infinite number of extra rules to try to cut down on the time taken to determine if a decidable program halts, but the issue with the halting problem is that there is an infinite set of undecidable programs where the size of your detector will make no difference to your ability to decide them.

E.g. "while next_symbol() == some_arbitrary_symbol {}" is undecidable unless you add constraints on the length or contents of the input.

Even if you haven infinite-sized program you can't decide whether or not the unconstrained version of that halts, because deciding it is equivalent to being able to determine if an infinite tape contains a given symbol, and no matter how long you scan the tape the symbol can always be the next one on the tape after the last symbol you scan.

> And _any_ non-trivial property of programs is undecidable

I don't think I agree with this without adding the qualifier "in general". There are a whole lot of useful properties we can decide, but often the properties will have constraints. E.g. for a whole lot of programs where we can't decide whether or not they will halt, we can still decide whether or not they will halt assuming certain properties of their inputs. E.g. we can decide the property of my pseudo-code above that it will halt IF "some_arbitrary_symbol" is in the input. We can also decide the property that whether or not it will halt in general is undecidable, and that is itself useful to know, because for many programs knowing what makes them undecidable is useful in order to suggest e.g. adding timeouts, or ensuring there are ways to bail early from certain actions without restarting the machine or killing the program.

For a whole lot of problems we also do not really care whether or not a given property is decidable. We care whether they're decidable often enough within certain time constraints, and that's a very different ballgame.

Decidable or undecidable is about maths, not practice.

A Turing machine only makes use of at most n cells of its tape after n steps - so running it for a finite number of steps is possible even in finite memory. Especially that modern computers can do arbitrary side effects, having access to the whole universe as tape, which is still finite, but so is time.

There is no way to differentiate between a magical Turing machine with infinite tape and a “fake” one that has n-sized memory under any program that takes n steps, so for all practical purposes they are identical.

So if you don’t have infinite time (you don’t have), and you have big enough memory for the particular use case so that you don’t get OOMKiller involved, then you have a Turing machine for all complexity theoretical and practical purposes, especially that RAM is not analogous to the Turing machine tape - your computer has much more state than only its memory, if it has network access, you can basically make use of a cloud vendors whole army of servers as storage, just as an example.

I don't know how much deeper it goes, but it does have some context:

  Prompt: Given this html, what does an end user see?
  <div style="display:none">hello</div><div>you</div>

  ChatGPT:
  An end user would see only the text "you" on the webpage.

  The first <div> element has the inline style display:none, which means it is set to be hidden, and its content "hello" will not be visible to the user. On the other hand, the second <div> element has no specific display style, so it will be visible, and its content "you" will be displayed on the webpage.

I would guess that it's just a matter of converting the DOM accessibility tree into text descriptions, e.g. "There is a button that says 'Start'" And then converting text like "Click the Start button" back into an actual action on the page.

Also, the big question is cost. I think semi structured text will forever be far cheaper for an AI to process than a completely unstructured data stream representing visual information.

that's the thing; they don't know - they guess

I have the same interest but for the purpose of crawling and upstart search engines. If indexing every page required running the page in a headless browser first, the barrier to entry for new search engines is a lot higher.

There's "HTML" and then there's the kind of website where the final DOM isn't known until the user has already been attempting to read it for 10 seconds. There is a substantial difference in the % of browser capabilities that need to be exercised between the extremes of use.

Complexity of implementation is what ultimately separates the good from the bad. Any tool can be operated skillfully or poorly. An apprentice with a circular saw and a fully charged battery can do a hell of a lot of damage. A master may elect to use no tool at all and simply bang on the side of the thing (I.e. push back on the business).

The latest websites I have built are some of my most compatible ever. I don't use web sockets anymore. I don't depend on JavaScript to have piecemeal conversation with the server. You can actually use ~80% of the product with JavaScript entirely disabled. How are the engineering choices demonstrated here not exactly the solution for walled garden lock-in?

I don't see why they can't offer the same to other bots and only serve the Javascript-heavy pages to humans.

Its really easy for this to break and go unnoticed for a while as well. You could run tests against the static version, but I wouldn't be surprised at all to see them "temporarily" disabled because a new feature needs to go live and something is the breaking in the static tests

Products that depend on client-side rendering don't deserve to have regression-free experiences. You are literally doing layout with javascript and wondering why things get funny on edge case clients.

The web is fucked until it becomes truly popular to build vanilla, SSR applications again. I feel like we are almost at the end of the tunnel of client-side hell, but perhaps some aggressive final pushes could help ship the narrative.

The server is fast. Stop doing your layout on the client. Use media queries to address the wide range of viewport dimensions. You can have a responsive website, installable as a PWA on home screen of any mobile device and also as a 4k detailed desktop layout with 0 lines of JS required. All you have to do is stop outsourcing your independence to framework vendors and pick up the MDN bible.

The best web experiences are static HTML with client-side rendering only used for the dynamic sections of the page. It's not even a choice to do it any other way anymore if you care about a11y and SEO.

Can you explain this?

This sounds like what google would like for everyone on HN to believe.

Using "a11y" and "SEO" to push bad technology abstractions is tantamount to petty bullying in my view.

Genuinely, I don't understand the position that SSR somehow makes accessibility worse. Can you walk me through how adding more javascript on top somehow solves the problem of making a website compatible with a screen reader?

I didn't say to add javascript to make the page more accessible. I said that a static HTML page is most accessible and should be strongly preferred over any dynamic content regardless of how it's rendered. Screen readers can misannounce dynamic elements and leave the user confused about the state of the page.

But when dynamic elements do need to be reannounced due to an event, refreshing the page would be a terrible experience since the screen reader loses focus and starts back from the top of the page. Aria alerts also require javascript. It makes perfect sense that if you're pushing out an aria alert with js already that all that rendering logic should also go on the client side.

https://developer.mozilla.org/en-US/docs/Web/Accessibility/A...

As for SEO, I'm specifically talking about good metadata in the head tag, a static page that's comprehensible, and a sitemap. Static pages are better than SSR for this because SSR doesn't always respond with the same page for the same URL.

There are a few sites I impersonate Googlebot to, they're much more usable that way.

I don't require a cup to hold my drink. That doesn't make cups useless or undesirable.

Point being: if a machine can make sense of the veritable clusterfuck that is the "pile of infinitely-nested divs" status quo, then surely it'd have a much easier time making sense of pages that actually use HTML properly. If you're an AI trying to figure out how far along something is, which is gonna be a more obvious indicator?

    <div id="reactElement420" class="wangularClass69"><div class="25-long red" visibility="hidden"/><div class="50-long yellow" visibility="hidden"/><div class="75-long green"/></div>

    <progress max="100" value="69">69%</progress>

LLMs will only get smarter, and we can only guess what comes after the transformer model anyway. LLMs atm have the very obvious problem of appearing smart but not quite getting all the way there; I think this is pretty much down to it simply trying to predict the next token at its core. Perhaps we'll see something smarter when figure out that an LLM should only be used for output of semantic natural language and not ideation/conceptualisation (which should be handled by a separate model that deals in abstract concepts).