emeryberger

Since we are talking about Python type-checkers: we've built a (non-AI based) type assistant for Python called RightTyper (https://github.com/RightTyper/RightTyper). Below is a brief description; a technical paper describing RightTyper is here: https://arxiv.org/abs/2507.16051, "Getting Python Types Right with RightTyper"

RightTyper is a Python tool that automatically generates type annotations for your code. It monitors your program as it runs and records the types of function arguments, return values, local variables, and class fields — with only about 25% runtime overhead. This makes it easy to integrate into your existing tests and development workflow, and lets a type checker like mypy catch type mismatches in your code.

RightTyper is much better in addition to running orders of magnitude faster.

https://github.com/RightTyper/RightTyper

(full disclosure, I am one of its authors).

You can adjust the code on the page easily (it’s open source javascript) to determine the question you are after, which is a valid one: if an adversary starts today and adds x nodes to the existing network, what is their success rate?

BUT the author asked a different (but valid) question: assuming the adversary controls x out of N existing nodes, what is the success rate? I am unclear: is the assertion that everyone’s relay is honest today? From a privacy standpoint, that’s not a great assumption.

  The Tor Project web site makes a bold claim to its users:

  "Tor Browser prevents someone watching your connection from knowing what websites you visit. All anyone monitoring your browsing habits can see is that you're using Tor."

  Don't misinterpret this claim. It's not true that Tor protects you against "anyone monitoring your browsing".

  On this web page, I provide you with the critical information missing from the Tor Project's website: if you estimate your adversary's resources in dollars, I'll estimated the probability that Tor will fail to protect you.

(web site by Brian Levine, Professor at UMass Amherst <https://www.cics.umass.edu/about/directory/brian-levine> and director of Rescue Lab <https://www.rescue-lab.org/>)

Yes. It lets the LLM immediately obtain precise information rather than having to reason across the entire source code of the code base (which ChatDBG also enables). For example (from the paper, Section 4.6):

  The second command, `definition`, prints the location and source
  code for the definition corresponding to the first occurrence of a symbol on a
  given line of code. For example, `definition polymorph.c:118` target prints the
  location and source for the declaration of target corresponding to its use on
  that line. The definition implementation
  leverages the `clangd` language server, which supports source code queries via
  JSON-RPC and Microsoft’s Language Server Protocol.

You should check out ChatDBG project - which AFAICT goes much further than this work, though in a different direction, and which, among other things, lets the LLM drive the debugging process - has been out since early 2023. We initially did a WinDBG integration but have since focused on lldb/gdb and pdb (the Python debugger), especially for Python notebooks. In particular, for native code, it integrates a language server to let the LLM easily find declarations and references to variables, for example. We spent considerable time developing an API that enabled the LLM to make the best use of the debugger’s capabilities. (It also is not limited to post mortem debugging). ChatDBG’s been out since 2023, though it has of course evolved since that time. Code is here [1] with some videos; it’s been downloaded north of 80K times to date. Our technical paper [2] will be presented at FSE (top software engineering conference) in June. Our evaluation shows that ChatDBG is on its own able to resolve many issues, and that with some slight nudging from humans, it is even more effective.

[1] https://github.com/plasma-umass/ChatDBG (north of 75K downloads to date) [2] https://arxiv.org/abs/2403.16354

Nice UI. We started on a project that does this about 2 years ago. ChatDBG (https://github.com/plasma-umass/ChatDBG), downloaded about 70K times to date. It integrates into debuggers like `lldb`, `gdb`, and `pdb` (the Python debugger). For C/C++, it also leverages a language server, which makes a huge difference. You can also chat with it. We wrote a paper about it, should be published shortly in a major conference near you (https://arxiv.org/abs/2403.16354). One of the coolest things we found is that the LLM can leverage real-world knowledge to diagnose errors; for example, it successfully debugged a problem where the number of bootstrap samples was too low.

Correction: it’s been a feature for at least two years. To our knowledge, Scalene was the first profiler to incorporate LLMs.

  One major limitation the LLM has is that it can't run a profiler on the code, 
  but we can. (This would be a fun thing to do in the future - feed the output
  of perf to the LLM and say 'now optimize').

This has been a feature of the Scalene Python profiler (https://github.com/plasma-umass/scalene) for some time (at this point, about 1.5 years) - bring your own API key for OpenAI / Azure / Bedrock, also works with Ollama. Optimizing Python code to use NumPy or other similar native libraries can easily yield multiple order of magnitude improvements in real-world settings. We tried it on several of the success stories of Scalene (before the integration with LLMs); see https://github.com/plasma-umass/scalene/issues/58 - and found that it often automatically yielded the same or better optimizations - see https://github.com/plasma-umass/scalene/issues/554. (Full disclosure: I am one of the principal designers of Scalene.)

Debuggers are great, and even better with LLMs! https://github.com/plasma-umass/ChatDBG https://arxiv.org/abs/2403.16354

However, the resulting type annotations should work for Python 3.9+.

It relies on some features introduced in Python 3.12, specifically the `sys.monitoring` API.

Can’t honestly remember the details! But it’s very much an AWS-owned org.

In addition to being a faculty member at UMass Amherst, I am an Amazon Scholar, working at Amazon Web Services, where this work was conducted.

I agree that writing type hints can be painful, especially if you are starting with a large code base that is mostly untyped. You might consider using RightTyper (https://github.com/RightTyper/RightTyper) - basically run your Python 3.12+ program with it, and it will add type hints to your code. It’s fast, basically automatic, and by design RightTyper avoids overfitting to your types, letting a type checker like MyPy surface edge cases. In effect, the type checker becomes an anomaly detector (Full disclosure, I am one of the authors of RightTyper.)

From the GitHub page:

RightTyper is a Python tool that generates types for your function arguments and return values. RightTyper lets your code run at nearly full speed with almost no memory overhead. As a result, you won't experience slow downs in your code or large memory consumption while using it, allowing you to integrate it with your standard tests and development process. By virtue of its design, and in a significant departure from previous approaches, RightTyper only captures the most commonly used types, letting a type checker like mypy detect possibly incorrect type mismatches in your code.

Now posted: https://www.youtube.com/watch?v=_VF3pISRYRc&t=4862s

In his talk at REBASE, Fil said the overhead is now 1.5x to 5x, and he is just getting started optimizing.

From Fil Pizlo (http://www.filpizlo.com/), major contributor to WebKit's JavaScript engine, among other things:

> Fil-C is already powerful enough to run a memory-safe curl and a memory-safe OpenSSH (both client and server) on top of a memory-safe OpenSSL, memory-safe zlib, memory-safe pcre (which required no changes), memory-safe CPython (which required some changes and even found a bug), memory-safe SQLite, memory-safe libcxx and libcxxabi, and memory-safe musl (Fil-C's current libc).

Scalene author here - please file an issue!

(Scalene author here) Nope, but WSL2 (Windows Subsystem for Linux) is, and Scalene works great with it.

Those papers say "language" over and over again, in the titles, in the body of the text. That work confounds languages and their implementations, and make it sound like there is a one-to-one connection between the two (of course, there is not necessarily such a correspondence).

With respect to Ruby vs. JRuby: my student just checked and verified that some but not all of the benchmarks are implemented differently (k-nucleotide, mandelbrot, pidigits, spectral-norm).

We only pin to one core for one experiment described in Section 4.3. All the remaining experiments are run with full access to all cores.

Hi, co-author here! We're happy to answer questions. See some discussion here:

* Mastodon: https://mastodon.social/@ltratt/113282264909342842 * Lobsters: https://lobste.rs/s/y12hdo/it_s_not_easy_being_green_on_ener...

Would love to hear a technical comparison between this and King et al.'s classic paper on Time-Traveling VMs from USENIX ATC 2006: "Debugging operating systems with time-traveling virtual machines" (https://www.usenix.org/legacy/events/usenix05/tech/general/k..., 505 citations).

Here's the relevant quote that speaks to your statement:

  Mossad thus defies the conventional wisdom that defeating plagiarism detection
  is difficult or requires significant programming ability. The techniques that
  underlie Mossad could be implemented manually, relying on only the most basic
  understanding of programming language principles, letting them evade detection
  by both plagiarism detectors and some degree of manual inspection...

If you read the paper, you'll see that the attack is entirely feasible to implement by hand (we did this ourselves but do not report on it in the paper). It's a pretty mechanical process. A bit of trial and error will get the job done; it's a hell of a lot easier than most assignments.

FWIW the attack we describe in the paper works against MOSS, too (that was the original inspiration for the name, “Mossad”).