News for hackers. Stuff that matters.
Re:How to Get Started with Tree-Sitter by avodonosov on Sunday May 28, 2023 @05:58PM (#36108199)
Anyone knows, why github code navigation, which is based on tree-sitter [1], supports only subset of languages for which tree-sitter has parsers [2]? For example, Common Lisp is not supported. I asked github [3], but they are silent.

1 - the "search based" navigation is based on tree sitter https://docs.github.com/en/repositories/working-with-files/u...

2 - the list of parsers in the official docs: https://tree-sitter.github.io/tree-sitter/

3 - https://github.com/orgs/community/discussions/55704

Re:How to Get Started with Tree-Sitter by awayto on Sunday May 28, 2023 @06:25PM (#36108373)
I worked with tree sitter a bit for a small project. It seems to be the case that, while it covers a large set of languages, the actual implementation of each subset doesn't follow the exact same underlying API. And it follows that there is no universal AST parser as far as I can tell. Languages have many types of atomic parts to such an extent that unifying them is a tremendously complex task.
Re:How to Get Started with Tree-Sitter by avodonosov on Sunday May 28, 2023 @07:26PM (#36108792)
Have you used the `tree-sitter tags` command?
Re:How to Get Started with Tree-Sitter by awayto on Sunday May 28, 2023 @07:44PM (#36108923)
Unfortunately not. My stint of use was focused around dynamically loading in any given tree-sitter-langaugeName package via docker and then parsing any given file type in a runtime environment, in order to coalesce differing languages' parts into a unified metadata structure.

So, I wasn't trying to identify named things, like the tags command might seem to do, but more generic parts of the langauges like function, var, import/export statements, etc. I ultimately made my own walk functionality, but pulling out those atomic parts per language is still a very heavy task (basically need a mini lexer for each lang), even though TS seems to provide all the info to do the task.

Re:How to Get Started with Tree-Sitter by kfir on Sunday May 28, 2023 @06:19PM (#36108331)
I believe this paper has an answer to your question - https://queue.acm.org/detail.cfm?id=3487022
Re:How to Get Started with Tree-Sitter by avodonosov on Sunday May 28, 2023 @06:46PM (#36108525)
Thank you very much for the link.

I don't see an answer in there, though.

They use tagging (as described also here: https://tree-sitter.github.io/tree-sitter/code-navigation-sy...). From both docs I assume `tree-sitter tags` works out of box for any language that has a parser. (Since neither doc instructs to save a custom config for tag extraction query, I assume every language plugin provides tagging queries).

Re:How to Get Started with Tree-Sitter by euiq on Sunday May 28, 2023 @06:58PM (#36108614)
While we're talking about this, I've been quite disappointed by the new GitHub file browser: the new features look neat, but the core file browsing experience is significantly worse for me because the new UI is much more complicated and every action seems to do way more work.

I couldn't immediately find a way to opt out; I wouldn't mind the new file browser so much if I could drop into it on demand instead of having it forced down my throat.

Re:How to Get Started with Tree-Sitter by gurjeet on Sunday May 28, 2023 @07:38PM (#36108885)
Click on your profile in the top-right, and then in the resulting menu, click on 'Feature preview'. You might be able to disable it from there.
Re:How to Get Started with Tree-Sitter by euiq on Sunday May 28, 2023 @08:13PM (#36109111)
It’s not there anymore, presumably because it graduated from beta earlier this month: <https://github.blog/changelog/2023-05-08-the-new-code-search...>
Re:How to Get Started with Tree-Sitter by throwaway290 on Sunday May 28, 2023 @07:42PM (#36108905)
It was an optional opt-in labs feature that I tried for an hour then turned off, but now they force it.

The new UI is more busy for no good reason. Sidebar makes sense in a local IDE, but over web every time you expand a subtree it takes ages anyway. And because it merely exists it is distracting and makes pages load slower...

Re:How to Get Started with Tree-Sitter by eyelidlessness on Monday May 29, 2023 @02:01AM (#36111163)
I’ve had few (some, but minimal) problems with the new file tree browser, but the actual individual file view has become a minefield. It hijacks find in page, presumably to enhance it with semantic relations, but it breaks frequently when navigating. Previous searches (cmd-G) will fall back to the browser’s native search. If you’re lucky, you searched the same term in both, and if you’re luckier still the search term is in the top N lines of code (because the code itself is lazy loaded I guess?). Most of the time I’m not lucky and I have to start over each time I view a different file. Even though it feels more like a “single page app” than it previously did, I lose all of that state every time I do anything.

I get what they’re trying to do, and in some ways it’s clearly beta growing pains, but in other ways it’s clear they’re fighting the browser environment in ways that don’t and probably shouldn’t work.

Re:How to Get Started with Tree-Sitter by donio on Monday May 29, 2023 @02:27AM (#36111278)
I ended up blocking keyboard events from a userscript to deal with the keyboard hijacking. This combined with opting out of all the "feature previews" has made it more or less usable again for now. But the writing is on the wall, I am not using GH for any new projects.
Re:How to Get Started with Tree-Sitter by cstrahan on Sunday May 28, 2023 @07:22PM (#36108762)
I believe they use semantic (Haskell program that uses tree-sitter) for navigation: https://github.com/github/semantic

So the answer may be that semantic does not yet have support for the language in question.

Re:How to Get Started with Tree-Sitter by ghuntley on Sunday May 28, 2023 @10:43PM (#36109998)
correct. adding support for new language is a matter of opening up a pull-request with the appropriate gluecode. https://github.com/github/semantic/blob/793a876ae45d38a6bd17...
Re:How to Get Started with Tree-Sitter by avodonosov on Monday May 29, 2023 @08:34AM (#36113342)
If github allowed per-repo configs with custom parsing soripts, similar to gitLab LSIF support - https://docs.gitlab.com/ee/user/project/code_intelligence.ht... - it would be more realistic to think of pull requests. Otherwise, how can a contrioutor be sure that his pull request really produces convenient code navigation in github UI.
Re:How to Get Started with Tree-Sitter by avodonosov on Monday May 29, 2023 @05:48PM (#36118498)
The Semantic seems not used anymore:

> During this time, the team migrated from a Semantic-based tagging service to one that operated entirely with (newly developed) Tree-sitter queries. Though Semantic performed well, using the Tree-sitter query language allowed faster iteration and avoided the operational overhead of a program-analysis framework.

https://queue.acm.org/detail.cfm?id=3487022

Re:How to Get Started with Tree-Sitter by avodonosov on Sunday May 28, 2023 @09:02PM (#36109435)
Re:How to Get Started with Tree-Sitter by ghuntley on Sunday May 28, 2023 @10:42PM (#36109987)
ah, easy. it's because support has not been added into https://github.com/github/semantic which is the tech that powers the GitHub UI. Adding support is pretty easy/mainly glue code [1] that imports the tree sitter API.

[1] https://github.com/github/semantic/blob/793a876ae45d38a6bd17...